Camo blog

Sharing our passion for industrial analytics

Insights, inspiration and reflections from Camosapiens.

Charles E (Chuck) Miller:

Detour: Reminiscing 2017

Although I had planned for my next blog to be a continuation of the series covering “PAT Value Proposition”, recent events have compelled me to do something different.

By now, it is painfully apparent that the current COVID-19 pandemic is taking us into uncharted territory, with the uncertain trajectory of the pandemic causing much anxiety about the future. As we all work to adapt to the “new normal” of our personal and occupational lives, it would seem natural for some of us to reminisce about prior personal and professional experiences that compare to the current one. I am not a psychiatrist, but such reminiscing could help one to cope with the uncertainty of the current crisis, while also helping to determine possible mitigating actions for the current crisis.

The past crises that one recalls likely depends on occupation, personal interests, and personal impact. Many have looked back towards the 9/11 terrorist attacks, the 2008 financial crisis, and past pandemics like the Influenza epidemic of 1918 or the Ebola epidemic of 2014-2016, to attempt to provide some useful reference points for the current crisis. For me, the COVID-19 crisis brings me back to two very different yet equally impactful events during my journey in PAT and MVA. Curiously, both of these events occurred in 2017: 1) Clinical supply and Commercialization of a “breakthrough” vaccine, and 2) The NotPetya Ransomware Incident. To avoid disclosing sensitive information, I will discuss these two events anecdotally, only disclosing details that have already been publicly released.

Breakthrough Vaccine

By my recollection, it was sometime in early 2017 when some of us on the Merck PAT team were asked to attend some meetings with a team working on a vaccine named “V920”, which had recently received FDA Breakthrough designation. This designation enables the expedition of development and review of medicines that demonstrated substantial improvement over existing therapies. Even under normal circumstances, product teams are rather stressed to efficiently develop and register their product, in an effort to be “first to market”- but with Breakthrough designation status such teams are under even more pressure to deliver, as public health is literally at risk.

When supporting active internal programs, I recall the general requirement that we refer to the product candidate by its internal code name only, and not by the indication (illness) that it is targeting. However, just like context is critical for multivariate datasets, context in this case pointed us towards “V920” being “the Ebola vaccine”, which had already been confirmed publicly. For those who do not recall, an Ebola outbreak had already occurred in West Africa, spanning from 2014-2016, with over 28,000 cases and 11,325 deaths. Unlike COVID-19, effective prevention programs and policy executions at the national and global level limited this outbreak to West Africa. However, just like COVID-19, the possibility of this virus resurfacing and spreading in the near future drove an urgency to develop a vaccine- an urgency which was clearly apparent on the V920 team. This was further motivated by the encouraging results of Phase II and Phase III studies of V920 during the West Africa outbreak.

Surely enough, Ebola indeed resurfaced in the Democratic Republic of Congo (DRC) in 2018, during which time the vaccine’s effectiveness was proven to be at least 97.5%, and FDA approval for the vaccine was received just a few months ago, in December 2019. Although these events are clear “wins”, I must admit that MVA likely played only marginal roles in these successes. When we first met with the program team in 2017, the on-line MVA process monitoring platform had already been installed and qualified at several supply sites, but unfortunately not yet at the specific Contract Manufacturing (CMO) site that had been chosen for this program. Due to the urgency of the program, and the limited resources available to deploy and qualify the platform at the CMO site in short order, the real-time process monitoring value of MVA could not be fully realized for this specific case. However, we were able to provide some off-line MVA support in the form of predictive modeling and multi-scale modeling, to support process transferability and optimization efforts. I should note that this was made possible by those folks on the early development team who had the foresight to save and contextualize a large amount of the data that had been collected in development.

Ransomware Attack

One morning in June 2017, my Merck colleagues and I (and many others!) were greeted to a rude awakening: PCs that had been left connected to the company network overnight had a ransomware note on the screen. Those of us (including myself) who had taken their PCs home that night but did not connect remotely to the company network via VPN were told to not connect to the company network until further notice. Just like the COVID-19 virus is to its human hosts, this ransomware worm was able to replicate and spread to other computer system “hosts”, including many company network systems. Unlike COVID-19, the time scale of this “virus’” spread was orders of magnitude faster: with systems being affected on the order of seconds. The attack was stopped within a few days of its discovery via an emergency patch – but the damage had already been done. Similar to a biological virus outbreak, the early days of this crisis were focused on containment, where the company directive was for all users to avoid connecting to the company network or any other system, as they were all presumed to be infected. For the PAT group, where most of the lab analyzer’s controller PCs had been required to stay connected to the network for regular patch updates and maintenance (and thus were likely infected), this directive effectively meant that we had no use of our personal PCs or the lab PCs for an indefinite period of time.

Like the current crisis, we all had to adjust to a “new normal” way of working for several weeks, which initially was limited to “old school” activities: like reviewing printed journal articles, organizing the paperwork in our workspaces, and cleaning and decluttering the labs. Thankfully, most of us had smartphones for personal use that were not affected by the hack- thus giving us a basic means of communication to obtain new corporate IT department directives, and to collaborate with PAT and project colleagues. During the ensuing weeks, we were updated by IT on progress of the containment work, and then progress on the next phase: remediation. I should note, however, that restoration was nothing like just installing a new app on your PC or iPhone: First of all, special care was needed to avoid unwittingly re-introducing the malware to systems during restoration activities. Furthermore, many of these systems were very complex, and many were also used for GxP purposes- thus requiring careful IQ (Installation) and OQ (Operation) testing. Finally, the sequence of system restoration was driven by the dependencies of the various systems, as well as their business-critical status: with the most business-critical systems being restored first.

Over the following several months, as systems were being restored based on criticality, work life slowly creeped back towards was considered normal before. During this time, we in the PAT group worked on updating and improving our asset management system: which included capturing all of the critical configuration elements of all PAT PCs and analyzers in the lab and in pilot plants around the site. At the same time, we collaborated with the manufacturing sites to capture the same information for their process analyzers and dedicated controllers. Through this crisis, I think that we gained a better appreciation of the key system attributes that need to be captured to help streamline remediation if such an event occurs in the future. It was also quite encouraging to see that the enterprise-level PAT management system was rather highly ranked on the list of critical IT systems on the restoration schedule. However, when all was said and done, the impact of this event to operations was significant, and it was clear that some permanent changes in our PAT workflows would need to be made to minimize the impact any future cyber events. Indeed, data security and reliable data backup are key parts of FDA’s 2016 guidance on Data Integrity [1].

Reflections, Lessons Learned

Even though these two stories are quite different in nature, they both provide some key learnings that can be leveraged towards developing more robust and effective PAT and MVA systems going forward:

  • On-line MVA analytics systems should not be viewed as software, but rather as assets that are part of the manufacturing infrastructure. Once they are established as part of the infrastructure and workflow at a GMP production site, they can be readily applied to any products that use that site, including those that might be on a “fast track”.
  • The traditional paradigm of “data silos” might be conducive to local data system administration, but they are a barrier to efficient and productive off-line MVA for product and process development.
  • The current industry trend of relying on CMOs for supply present challenges to both off-line and on-line MVA value propositions, due to intellectual property and data ownership issues. However, these challenges are not insurmountable.
  • Lifecycle management documents associated with PAT and associated IT systems certainly seemed like “overkill” at first, but they are invaluable for disaster recovery!
  • A strong data science culture has value: the extra work required to collect, aggregate and contextualize data at the time of measurement pays great dividends for any subsequent MVA analyses supporting development and commercialization.
  • Development and testing of backup systems for PAT and MVA data are critical for disaster preparedness. Any backup system must include capture of key configuration elements of the PAT analyzers and their PC controller systems.
  • Despite what anyone else in your organization might say about the value of PAT and MVA, their true importance is quickly realized when their systems are unexpectedly out of service.
  • Last but not least- there is another side to every crisis, and the journey provides opportunities for inspiration and improvement

That’s all for now – I hope you are all well!

[1] Data Integrity and Compliance With CGMP- Guidance for Industry, U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research (CDER) Center for Biologics Evaluation and Research (CBER), Center for Veterinary Medicine (CVM), April 2016. https://www.fda.gov/media/97005/download
Charles E (Chuck) Miller:

Career-long journey in Process Analytical Technology

Chuck Miller has over 30 years of experience in chemometrics, near-infrared spectroscopy, and Process Analytical Technologies, and applying these technologies to industrial challenges in R&D and manufacturing.

His career spans 13 years at DuPont, in the Process Analytical group and 10 years in the Process Analytical Technologies group within the Manufacturing Division of Merck Sharp and Dohme. In between he worked at Eigenvector Research in consulting, training and software development for 4 years.

Chuck obtained his Ph.D in Analytical Chemistry from the University of Washington, and did his post-doctoral research at the Max-Planck Institute for Polymer Research in Mainz, Germany and The Norwegian Food Research Institute in Ås, Norway.

Follow Chuck here:

Find the right solution for your analytics needs.

Get in touch if you have questions about our products, platform, how to get started or how best to address your analytics needs.

Contact form