FDA & UA Analyzed Amazon Consumer Reviews for Homeopathy Adverse Events
By Alissa Gould, Boiron USA Director of Communication and Public Affairs
Over-the-counter homeopathic cough and cold products were the subject of a study to see if applying natural language processing (NLP) to Amazon consumer reviews showed comparable results to adverse events (AEs) reported to the U.S. Food and Drug Administration Adverse Event Reporting System (FAERS).
While the researchers successfully reached their objective (concluding that Amazon reviews may complement traditional data sources to identify AEs), this AAHP article focuses on what the homeopathic industry can learn from the study. To this point the study’s authors also concluded that, “while this information may be useful to regulators, it may also have utility for commercial enterprises, providing insight into consumer perceptions about products and identifying potential product-related safety and quality concerns.” AAHP agrees.
The Study
“Using Natural Language Processing to Characterize and Predict Homeopathic Product-Associated Adverse Events in Consumer Reviews: Comparison to Reports to FDA Adverse Event Reporting System (FAERS)” was published in the Journal of the American Medical Informatics Association in December 2023. https://academic.oup.com/jamia/article/31/1/70/7320056?login=false
This was one of 75 publications in 2023 to which CDER’s Office of Surveillance and Epidemiology (OSE) staff at FDA contributed, according to OSE’s 2023 annual report published July 2024. Two of the study’s authors were OSE staff. The other four authors were with the University of Alabama at Birmingham’s School of Health Professions in the Department of Health Services Administration, as well as University’s Informatics Institute, and the University’s Electrical & Computer Engineering in the Center for Integrated Systems.
Data was collected through FAERS and Amazon. Much of the nine-page study details in depth how the reviews, comments, etc. on Amazon were extracted, deduplicated, eliminated, analyzed for themes, etc. using NPL. Data collected through Amazon was reviewed by the physician author (one of the FDA staff members) to verify that the corresponding products were in fact homeopathic. Data collected through FAERS was limited to AE reports submitted by a consumer and described a single-suspect homeopathic product.
In the end, only 616 Amazon and 445 FAERS datapoints were analyzed from the millions extracted between May 1996 to July 2014. It was noted that the volume of reviews and reports was sparse until 2009 and that this was the year FDA issued a warning letter to a popular homeopathic product. The study’s authors state: “It is unclear if reports triggered the warning, the warning stimulated reporting, or both. A search of the Amazon review narratives for ‘FDA’ or ‘food and drug’ identified no commentary about the agency’s warning.”
Regarding this uptick in AEs, the authors fail to mention the Sept. 27, 2007 FDA Amendments Act, which required for the first time manufacturers to report adverse events for over-the-counter drugs. Prior to this, only prescription drugs were subject to mandatory reporting of adverse events. In the 21 months between the Act being signed into law and the 2009 warning letter, the industry was beginning to implement the new requirements aimed at improving drug safety monitoring.
Why Focus on Homeopathy?
Manufacturers and FDA have the same challenge in surveying AEs: no pharmacovigilance data source captures all AEs, sometimes simply because AEs are not reported. According to one study, underreporting of AEs may be particularly problematic for unapproved drugs1 — a category that includes homeopathic medicines.
Over the past decade or so, AAHP has seen homeopathic medicines jump from the niche natural channel to the mass market channel due to market demand; In doing so, the products became more visible to mainstream shoppers and usage rose.2 Similar to allopathic over-the-counter drugs, the largest therapeutic category for homeopathic medicines is the cough and cold segment. Furthermore, grouping the homeopathic products in the study by intended treatment showed the most common uses as for cough, cold, flu, sore throat, and/or chest congestion.3
From this rationale, we might assume the homeopathic cough and cold category gave researchers a large enough subject to be satisiticallly signifcant, without being overwhelming.
Sentiment Analysis
The study also conducted topic modeling and sentiment analysis (SA) of Amazon reviews, which provides information about consumers’ perceptions and opinions of homeopathic cough and cold products. The study lists terms used in processing natural language in reviews. These included negations (e.g., “can’t,” “don’t,” or “didn’t”); weighted intensifiers that are descriptive (e.g., “amazingly” at 1.90, “I’m not sure” at 1.0, or “rarely” at 0.10); and sentiment terms of emotion, including emojis (e.g., “awesome,” “adequate,” and “abysmal”).
The study’s authors observed that, “Overall, the sentiments expressed in Amazon reviews [for OTC homeopathic cough and cough products included in the analysis] were skewed toward being positive.”
Amazon ratings appear to be a good indicator of the presence or absence of AEs, however manufacturers should not assume five-star ratings are void of AEs. The study showed 12% of the 283 five-star reviews included an AE. Conversely, there were only 37 one-star reviews, of which about 78% included an AE. One of the most common complaints through both Amazon and FAERS data was about unpleasant taste.
New Study on AEs for Homeopathic Drugs in FAERS
While the NLP study included only a total of 445 AE reports from FAERS and did not provide much details in types of homeopathic drugs and seriousness of reported AEs, a new study proposed by AAHP will provide a more comprehensively exam of AEs of homeopathic drug products in FAERS. Jingjing Qian, PhD, associate professor, Harrison College of Pharmacy, Auburn University, Alabama, will examine the characteristics of these AEs (including patient demographics, location, type of reporter, type of homeopathic drugs, type of AEs, and seriousness of AEs), in addition to quantitatively assess reporting risk of serious AEs.
Findings of this project will provide an up-to-date and fundamental safety profile of marketed homeopathic drugs, which will further support a well-informed legislative and regulatory environment for homeopathic products in the United States. It will also establish a baseline of data that can be built upon in the future.
Future Considerations
For now, FDA encourages vigilance and monitoring of all available information, including user-generated content on third-party websites and social media. However, monitoring sites not owned by your business is a huge undertaking and requires more staff to manage the data discovered. A European document that has been accepted by FDA states that businesses “should regularly screen websites under their management or responsibility for potential adverse drug report cases. [These businesses] are not expected to screen external websites for adverse drug report information. However, if the [business] becomes aware of an adverse reaction on a website that it does not manage, it should review the case and determine whether it should be reported.”4
As ecommerce, consumer reviews, and social media influencers are increasingly important in the marketing of products, manufacturers should evaluate the benefit of monitoring third-party sites for possible adverse event reports within an overall robust monitoring system. Your business will gain a lot of information about your product in the process.
References
1. Ashley DD. Clarifying Misconceptions About U.S. Food and Drug Administration Unapproved Drugs Program. Anesth Analg. 2018;127(6):1292–1294. Accessed June 24, 2023. https://pubmed.ncbi.nlm.nih.gov/30433920/
2. “Chapter Two: History of AAHP,” AAHP 100th Anniversary Report. July 2004. https://theaahp.org/wp-content/uploads/2024/08/AAHP-100_Anniversary_Report-Ch-2.pdf
3. Konkel, K., Oner, N., Ahmed, A., Jones, S.C., Berner, E.S., Zengul, F.D. “Using Natural Language Processing to Characterize and Predict Homeopathic Product-Associated Adverse Events in Consumer Reviews: Comparison to Reports to FDA Adverse Event Reporting System (FAERS).” J Am Med Inform Assoc. 2023 Dec. 22; 31(1): 70–78. https://doi.org/10.1093/jamia/ocad197
4. “Harmonised Tripartite Guideline Post-Approval Safety Data Management: Definitions and Standards for Expedited Reporting E2D, 2003” International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH). Organized under Swiss Law. Implemented by FDA Sept. 1, 2003; Reference: Vol. 68, No. 178, p. 53983-4. https://www.ich.org/page/efficacy-guidelines