Sharing our passion for industrial analytics
Insights, inspiration and reflections from Camosapiens.
Charles E (Chuck) Miller, Chief Data Scientist | David J Ryan, Principal Scientist, DuPont Nutrition & Biosciences | Nicole Stumpf, Research Associate, DuPont Nutrition & Biosciences
PAT Case Study 1: NIR Network for Soybean Products
Regarding the value proposition arguments made in the previous installments of this blog series, I don’t want you to just take my word for it: there are indeed many industrial applications of PAT that continuously demonstrate these, even though most such applications are not published in publicly available literature. In an attempt to illustrate many of these arguments, I will briefly review three publicly disclosed case studies involving a diverse set of long-standing PAT applications. The value of reviewing such enduring applications is two-fold: 1) they provide battle-tested insight into PAT best practices, and 2) they provide a glimpse into some of the challenges to be expected for newer applications.
This case involves the use of a network of off-line NIR diffuse reflectance analyzers to support quality assessment of a major agricultural product in the US . This is the oldest of the three case studies that I am sharing, with the first filter NIR analyzer put into service in 1974 (Dickey John GAC III), and the first set of filter instruments and a supporting monochromator instrument (Pacific Scientific 4250 and 6250) installed in the late 1980s. The first set of scanning spectrometers supported by PLS calibrations were installed at the field plants in 1992, with the full network put into service in 2000. In 2009, faced with obsolescence of the original spectrometer platform, the process of conversion of all multivariate methods to a new instrument platform was started, with about 50% of this conversion complete by 2014 and complete conversion achieved in late 2016. The overall scope of this application is enormous: covering five different soy products (one of which covers a global network), eight different product quality attributes predicted by NIR, with production histories of some of the products exceeding 25 years.
The longevity of this application, combined with the reliance on grown crops as feedstock, drive some unique challenges: including uncontrolled raw material variability from cultivar evolution, crop year and growing condition effects, changes in product specifications, changes in reference method protocols, instrument drift, and obsolescence of NIR instrument platforms. As a result, some of the key workflows for this application include instrument standardization, method transfer, method robustness testing, and calibration updating/maintenance. Before one even looks at any data, there are several key operational elements that needed to be in place, including strong domain knowledge of the reference methods, routine comparison of PAT vs reference results, product-specific sealed reference standards for monitoring relative responses of instruments in the network, and the generation of routine method quality diagnostics. Each NIR method is challenged at a regular interval, at a rate of approximately 10 test samples per month, to assess method robustness, and cumulative datasets are reviewed on a periodic basis to assess the need for model updates. Last but not least, as this is an off-line method that requires manual intervention by multiple operators, user training and certification, and regular internal audits for compliance to lab standard procedures (SOPs) were also key components of the application.
Discussion of this application here will focus only on the key workflow of calibration updating and maintenance, which is arguably the most difficult workflow, as it is impacted by all of the unique challenges listed above. More specifically, a recent study  exploited the vast sea of data from this application to assess the crop year effect on method performance. One of the most commonly-asked questions about PAT applications is:
When should I update my multivariate calibrations; and when I do, how should I update them?
This study addressed these questions head-on: by assessing the performances of the actual methods that were in service at the time of the study (2014) as a function of calibration set population, using the following calibration set perturbations:
- Sequentially eliminating the samples from the most recent crop years
- Sequentially eliminating the samples from the oldest crop years, and
- Selecting sample sets with different levels of NIR spectral diversity
Despite the unavoidable limitations of this design, including the non-deterministic nature of genetic cultivar behavior and growing conditions, and non-ideally distributed reference values and sample rates per year, the results of this experiment still provide some powerful results, summarized in Figures 1-4 below.
Figure 1: Effect of excluding older crop year samples from the calibration model, on the prediction error obtained when applying model to 2014 crop year samples, for 6 soybean product quality attributes (product in parentheses): IsoPAI = Protein (Isolate 90% Protein) , IsoDM = Moisture (Isolate 90% Protein) , WFPDB = Protein (Soy Flour), LecAI = Acetone Insoluble (Soy Lecithin), SMCF = Crude Fiber (Soymeal 48% protein), WFNSI = Nitrogen Soluble Index (Soy Flour). Best overall performance across methods was observed by retaining at least 5 – 8 or 9 of the most recient crop years in the calibration model and excluding samples older than that, unless special circumstances dictated otherwise.
Figure 1 shows the effect of excluding older crop year samples from the calibration set, expressed using the error of validation obtained from the same external test set (SEP)- where the X-axis labels indicate the year range of samples included in the calibration set. Consensus results across all 6 quality attributes indicate that retaining at least 5-8 or 9 of the most recent crop year calibration samples provided the best results. This result suggests that older crop year calibration samples do not contribute to the robustness of the updated 2014 methods. Of course, for any given application, special circumstances could exist that would drive different results that those seen above- but this case study still provides a good general example of how calibration models must be managed for such applications.
Figure 2: Effect of excluding newest crop year samples from the calibration model, on the prediction error obtained when applying model to 2014 crop year samples, for 8 soybean product quality attributes (product in parentheses): IsoPAI = Protein (Isolate 90% Protein) , IsoDM = Moisture (Isolate 90% Protein) , WFPDB = Protein (Soy Flour), WFNSI = Nitrogen Soluble Index (Soy Flour), LecAI = Acetone Insoluble (Soy Lecithin), SoyCF = Crude Fiber (Soy Flour), DAPDB = Protein by Diode Array (Soy Flour), DAFAT = Fat by Diode Array (Soy Flour). In most cases, performance decreases after removal of 2-3 most recent crop years.
Figure 2 shows the effect of excluding the newest crop year samples from the calibration set, expressed using the same SEP metric. Not surprisingly, the consensus best results (across the 6 quality attributes) are obtained when all of the most recently-cultivated samples (including 2014 samples) are used for calibration. However, the deterioration of method performance as the newest year samples are excluded shows some interesting patterns. For example, the method for moisture in the isolate 90% protein product is particularly impacted when the 2009-2011 crop year samples are excluded from the calibration set- suggesting that there is some special cause variation in these samples that is particularly important for the robustness of the 2014 method.
Figure 3: Effect of spectral diversity in the calibration model set to the prediction error obtained when applying model to 2014 crop year samples, for 8 soybean product quality attributes (product in parentheses): IsoPAI = Protein (Isolate 90% Protein) , IsoDM = Moisture (Isolate 90% Protein) , WFPDB = Protein (Soy Flour), WFNSI = Nitrogen Soluble Index (Soy Flour), LecAI = Acetone Insoluble (Soy Lecithin), SoyCF = Crude Fiber (Soy Flour), DAPDB = Protein by Diode Array (Soy Flour), DAFAT = Fat by Diode Array (Soy Flour). “2014” on the X-axis refers to all calibration samples up to and including those obtained in 2014. As “2014” can be considered a best case or “benchmark” value, general consensus across methods is that a minimum diversity of 300 to 500 samples is optimal.
Figure 3 shows the effect of spectral diversity in the calibration set, expressed using the same SEP metric, for 8 quality attributes. In this work, spectral diversity refers to the number of samples that are selected as a subset of the complete 2014 calibration dataset, using an algorithm like the Cluster Analysis Based method [2-3] or the Kennard-Stone method . These methods use sample distances in the PCA/PLS score space to select a representative subset of a pre-specified number of samples for calibration. Such sample selection is often employed in PAT calibration work, to both improve the efficiency of the modeling operation and to avoid over-representation of a few commonly-encountered sample states in the calibration set- which can adversely impact method performance . Results for this case show that, although there are a few cases where sample selection negatively impacts performance, most of the remaining cases do not see a significant deterioration of performance after a subset of 300 or 500 samples is used for calibration.
Figure 4 shows one case of using PCA scores to investigate the robustness of one of the methods- in this case the PAI (Protein) method for the isolate 90% protein product. From this plot, it is clear that the older crop year samples (1995-7) span a different spectral space than the most recent samples (2013). Furthermore, it shows that within the recent 2013 samples there are two distinct sub-populations of samples- later identifed as two new low-protein products. Such assessments can be quite helpful for verifying the robustness of PAT methods to cover multiple products or multiple product grades- as in this case.
There are many take-home messages from this long-standing application that readily translate to other PAT applications, many of which are listed below.
- Large multi-year calibrations tend to stay robust for several years, but routine model updates are still required, due to the ever-evolving landscape of factors that affect the product chemistry and measurement platform.
- In most cases, crop year effects and calibration data set variance are not randomly distributed, in that the instrument responses typically ”wander” in a direction away from the origin in the PC score space. As a result, the earliest crop years’ data (>8 Years old) generally do not seem to add value to any of the prediction models.
- Calibration models appear to perform best when at least 5 to 6 crop years are included.
- The Food & Ag datasets used to build the models require ≥ 300 samples to adequately describe the spectral and component space to ensure adequate prediction performance, or adequate transfer to a new platform.
- Some constituents’ calibrations are more robust to crop year effects than others, but this was difficult to assess a priori.
- Special cause effects, such as modifications in reference method chemistry, need to be proactively assessed in order to maintain good method performance.
- Model prediction error (SEP) is just one of many metrics that can be used to assess a method’s performance. Additional diagnostic metrics include PCA scores and prediction bias.
- Attaining an ideal distribution of constituent values (ex., by thinning the calibration database through sample selection) doesn’t contribute to improved performance in most cases.
- Calibration sample diversity needs to consider both reference values (Y) and spectral responses (X)
- Even after many years of operation, it is clear that there are still some unknown factors that continue to impact the methods.
To summarize, the need to monitor and maintain PAT methods over their lifecycle is more of an expectation rather than a possibility. Still, though, with the right ”method QA” tools in place, these workflows can be made very efficient and effective- thus ensuring that the business benefits of these methods can realized over several decades!
 D. Ryan, “Learning How to Break, So You Will Not Break, Your Calibrations: A Study of Crop Year Effects”, IDRC 2014 Conference, August 7th, 2014.
 C.E. Miller, ”Chemometrics in Process Analytical Chemistry”, in Process Analytical Technology, Blackwell Publishing, Oxford, 2005, pp. 226-324.
 Isaksson, T. and Næs, T., “Selection of Samples for Calibration in Near-Infrared Spectroscopy. Part II: Selection Based on Spectral Measurements”; Appl. Spectrosc. 1990, 44, 1152–1158.
 R. W. Kennard & L. A. Stone (1969): “Computer Aided Design of Experiments”, Technometrics, 11:1, 137-148.
About DuPont Nutrition & Biosciences
DuPont Nutrition & Biosciences is a world leader of innovative and sustainable solutions across food, health, pharma and biotech industries. With more than 10,000 employees around the world, they are focused on serving customers through scientific excellence and essential and sustainable innovations making people thrive. They are an innovation leader, drawing from strong consumer and market insights, deep knowledge, the best science and one of the broadest portfolios in the industry, to passionately solve any challenge.
Find the right solution for your analytics needs.
Get in touch if you have questions about our products, platform, how to get started or how best to address your analytics needs.