How Much Can Machine Learning Help Us?

By Mark Land, AAHP President
June 4, 2018

FDA docket 2017-D-6580 for comments on the draft guidance for FDA staff and industry entitled “Drug Products Labeled as Homeopathic” closed on Monday, May 21 after a 60-day extension. There were 4,792 comments submitted to the docket. That was about half the number of comments submitted to the docket following the Part 15 hearings in April 2015. At AAHP we were interested to know the type and nature of comments submitted but knew that analysis would be time consuming.

Last year I asked an intern to review the 2015 docket in detail to identify risks and opportunities we might not have considered. We established 25 categories of analysis and applied it to 9,467 comments. The process took two months and more than 300 intern hours. Copywriting is still reviewing the manuscript.

Recently I became acquainted with a firm that claims they can analyze a docket legislative or regulation and deliver sentiment map in seconds. They mentioned “machine learning” capability to achieve this goal so I was intrigued.

I asked them to apply their algorithm to our current docket. Sure enough I received a sentiment map in less than 30 seconds. This is what it said:

  • 43.0%   Were not analyzable
  • 47.5%   Opposed FDA’s guidance but were neither negative nor positive in sentiment
  • 3.5%     Opposed and were negative in sentiment
  • 1.6%     Opposed and were positive in sentiment
  • 1.3%     Supported the guidance but were neither negative nor positive in sentiment

A few thoughts came to mind looking at this data. First, 43 percent of these comments could not be analyzed. My reaction was that these comments must have been of such a bland flavor that the algorithm could not classify them. I wonder why a person would take the time to write such comments at all. Second, the majority of analyzable comments opposed the guidance but were not negative or positive. This means the staffer at FDA charged with the task of reading this volume of work will unlikely be emotionally assailed by our comments.

To answer my question in the headline, machine learning using this algorithm analyzed half of the docket and generated a rough map of sentiment. Assuming the ratios above hold up to analysis of the complete docket, machine learning can give a very quick sentiment analysis and point to commenters with strong opinions for further analysis.