R.C. McDowall is the principle of McDowall Consulting and director of R.D. McDowall Limited, and "Questions of Quality" column editor for LCGC Europe, Spectroscopy's sister magazine.

R.D. McDowall

Universally available on desktop computers, easy to use, and universally loved by analytical chemists, spreadsheets are used to collate lists of instruments and perform a multitude of calculations in many regulated laboratories. But spreadsheets also present an auditing and inspecting opportunity to find abundant instances of noncompliance and can lead to problems and unnecessary risk. Why is the use of spreadsheets heavily discouraged in a regulated laboratory?

If you turn on any workstationin your laboratory, you will probably find a shortcut to a spreadsheet program. When presented with this icon of temptation and seduction, a willing analyst is happy to open Pandora’s spreadsheet box and navigate between Scylla and Charybdis. Charybdis is the whirlpool of quality assurance where you have the challenge of specifying, building, and validating each spreadsheet template. Alternatively, steer the course to Scylla, the six-headed monster of regulatory noncompliance. With Scylla, if the unvalidated spreadsheet you have written and used is found, you are dead.


Spreadsheets are often used to perform GMP–related calculations, trend data, plan calibration and qualification work, and manage laboratory assets. I have even seen spreadsheets used to document the test steps for validating software—and the least said about that the better. This article discusses why spreadsheets should be eliminated as much as possible from performing GXP work to increase business efficiency and reduce regulatory risk. Note that business efficiency came before reducing regulatory risk in the last sentence. This column is dedicated to all spreadsheet lovers, or in the words of that great analytical chemist, William Shakespeare, “I come to bury spreadsheets, not to praise them.”

One of many companies that have sailed into Scylla and been cited for noncompliant spreadsheets is Tismore Health, as seen in a Federal Drug Administration (FDA) warning letter in December 2020 (1). I hope this does not happen in your laboratory. Citation 3 is against 21 

211.160(b) (2), although I would have raised the citation against 211.68(b) for failing to verify formulas. However, that is the problem with the FDA’s last century GMP regulations:

Your firm failed to establish laboratory controls that include scientifically sound and appropriate specifications, standards, sampling plans and test procedures designed to assure that components, drug products conform to appropriate standards of identity, strength, quality, and purity (21 CFR 211.160(b)).

Your firm failed to validate the spreadsheet used to perform the assay calculation for your <redacted>. Your procedures lacked guidance on how to check and manually verify the calculation sheets. During the inspection, our investigator identified a calculation error within the spreadsheet. The incorrect formula for averaging the Internal Standard peak area was used (1).

If you really want to descend into the abyss of compliance hell, let an inspector find an error in a spreadsheet. You will receive a comment like this: “There is no assurance that the associated assay results recorded are reliable and accurate.”

It is not the best situation having an inspector that thinks that all of your data are garbage. Also, if there is one calculation problem in unvalidated spreadsheets, then there are bound to be more, because there is a total lack of control when using spreadsheets.

During the review, you identified another error within your spreadsheets. The assay test result for <redacted> batch <redacted> was incorrect due to a transcription entry error for active peak area. Your firm used a new spreadsheet and entered the correct active peak area. The result was recalculated, and the final result was reported. The product had already been released with test results using the incorrect calculation although the recalculated test result was still within specification. You have committed to manually check calculations until the spreadsheet has been validated (1).

In the last sentence, we see a way to resolve the problem: You should manually check all calculations until the spreadsheet is validated. How useful are the unvalidated spreadsheets now? Read the full warning letter to see the extent of the remediation required by FDA (1). However, the approach taken by the company to manually check calculations (which is error prone in itself) misses the point about spreadsheets: They are a hybrid system that has major business process efficiency issues as well as regulatory compliance problems, as we shall see later in this column.

Here goes a list of the spreadsheet screw-ups. That well-known pharmaceutical company, Enron Corporation, used spreadsheets for a number of activities, and after the company went bankrupt, more than 15,000 spreadsheets were analyzed from the archive of 65,000 e-mails by two academics, Felienne Hermans and Emerson Murphy-Hill (3). Their main conclusions were:

Out of all the spreadsheets used, 76% of them used the same 15 functions.

In regards to spreadsheets with at least one formula, 24% of them contained an error.

Spreadsheets were a frequent topic of e-mail conversation with 10% of emails either referring to or sending spreadsheets and frequently discussing errors in and updates to spreadsheets.

Just like in your laboratory! In fact, if spreadsheets are well embedded into your culture, a spreadsheet developed in the morning could be the department standard by the afternoon. If so, I’m assuming you are steering a course for Scylla rather than Charybdis.

In a study that had a major impact on governments, two eminent economists, Carmen Reinhart and Kenneth Rogoff, published an article “Growth in a Time of Debt

which concluded that counties with a high amount of debt to Gross Domestic Product (GDP) (>90%) have slow economic growth. This resulted in austere policies to reduce debt and simulate growth in some countries, including the United Kingdom. Unfortunately for the two intrepid economists, they published in a non-peer reviewed part of a journal and used a spreadsheet to perform their calculations. And guess what happened? There were a few problems with the spreadsheet and the major conclusion was reversed (see Wikipedia for more details).

If you really want to understand the scope of spreadsheet disasters, take a look at Ray Panko’s website (www.panko.com). Panko is a University of Hawaii academic who has studied spreadsheets and errors associated with them for about 25 years. On his website, there is a page on spreadsheet errors that states the following (4):

Humans are very accurate when we work. When we create a spreadsheet formula, we are correct 95 to 99% of the time.

Unfortunately, we are not as good at finding errors. When we look diligently for errors, we only find 40 to 90%, and finding 90% is rare.

We would like to know how effective we are at finding errors when we inspect a spreadsheet. This is not just an “academic” concern. If our spreadsheets have as many errors as they seem to have, then we must test them more. One way to do this is to inspect them cell by cell. Many people believe that if we are careful enough, and if we spend enough time, we can find all of the errors in a spreadsheet. However, this ability is not realistic. If detection rates are too low, then a team inspection is necessary to catch 80 to 90% of the errors (4).

The inability to find and resolve spreadsheet errors is a worrying problem. How should we resolve it? The obvious one is that more resources need to be put into specifying the spreadsheet and how the formulas are built and tested. As Panko’s website suggests, more effort should be placed in checking the formulas. However, there is a simpler way, which is explained later on in this article.

In a section guaranteed to ensure that many readers will want to stick large pins into my effigy, I am advocating that spreadsheets be exterminated from use in a regulatory laboratory. This is not because a spreadsheet application is bad per se, but in a regulated environment it is a recipe for potential disaster, for many reasons, as outlined in Table I.

I’m not interested in your personal life, but look at your business process in the laboratory when you are using a spreadsheet. Figure 1 shows the tasks of a general quantitative analytical process in the middle in green. If you use a spreadsheet, you’ll follow the left hand side of Figure 1: Prepare the sample, analyze it along with standards, interpret the data in the spectrometer data system, print the results, manually type data into your (hopefully) validated spreadsheet, print the results out, and then type the result into a laboratory information management system (LIMS) or similar informatics application. But just look at the mess of the process and the number of records that you have created. You are a laboratory masochist.

Job done? No, enter the poor sap who is the second person reviewer. Just look at the mess this person has to review: the number of transfers between computerized systems, and the need to check for transcription errors each time data are entered manually into a different computerized system with checks of both electronic records and paper printouts. Welcome to death by compliance: transcription error checking by the reviewer, which is also an error-prone process. If you are really unlucky, the second person review takes longer than the analysis (10)!

Guess what? Instead of the mess on the left-hand side of Figure 1, many calculations you have spent time incorporating into a spreadsheet can be done by the instrument data system—except few people bother to read the manual. You can avoid the need to print and manually enter data into a spreadsheet followed by the consequential transcription error checks. Take the time to specify and validate the calculations, and you can save much time and effort in performing the analysis and the subsequent second person review. Look at the right-hand side of Figure 1 and compare with the left-hand side and immediately you’ll see my point. You only have the sample preparation records, the manual input of data into the data system, and the electronic records in the data system. Add electronic signatures and your process is sped up immeasurably, and you only have a single location to check records. Why would you want to do anything differently?

As mentioned earlier, let us discuss two of the main problems with spreadsheets in more detail, which are hybrid systems and single accounts and passwords.

A spreadsheet macro or template is a hybrid system—an electronic record of the saved and completed file with a hand-signed paper printout. Poor design means that data are typically input manually, and there is a large transcription error checking burden placed on both the analyst and the second person reviewer. One typical omission in laboratories I have audited is the lack of record signature linking between the paper printout and the electronic record that generated it. It is not an option to state that the paper is the GXP record; the spreadsheet file is a key component of the record set of any analysis. Please read question 12 of the FDA data integrity guidance to understand what is required:

12. When does electronic data become a CGMP record?

When generated to satisfy a CGMP requirement, all data become a CGMP record. You must document, or save, the data at the time of performance to create a record in compliance with CGMP requirements, including, but not limited to, §§ 211.100(b) and 211.160(a) (11).

This is very easy to understand and follow: Save the spreadsheet file or book an appointment with Scylla.

One of the issues with spreadsheet security is that there is only a single account and password for a template. Once the developer has locked the spreadsheet (this can be individual worksheets as well as the whole template) with one or more passwords, what are you going to do? This is one case when you will write the password down and follow the steps below:

The developer needs to write the spreadsheet name and version along with the passwords, and place the paper into an opaque envelope, write the spreadsheet and version name on the front, seal the envelope, then sign and date over the seal. This is to indicate if the envelope has been opened.

The envelope should be placed in a secure location, typically a fireproof safe, until the spreadsheet needs to be updated.

The envelope is unsealed, the spreadsheet maintained and revalidated, and the new passwords start the process over again.

If you find this a bit of a pain, then get rid of the spreadsheet.

Having spent all this time saying that you should eliminate spreadsheets, I must acknowledge that there are some exceptions to this rule. There are situations where calculations across analyses cannot be performed by instrument data systems and, in the absence of a LIMS, a spreadsheet may be the only option for calculating results. However, do you want to input data manually, and what about tracking changes to data? Ideally, a validated spreadsheet should be used within an electronic lab oratory notebook (ELN) or equivalent environment. Here, data should be imported and loaded electronically with a validated process to eliminate transcription error checking, and any changes are monitored by the audit trail of the informatics application. Calculated results should be transferred electronically to the next part of the process for speed and to eliminate transcription error checking.

When it comes to persuading laboratories to use alternative ways of calculating other than spreadsheets, I feel like King Cnut, an 11th Century King of England and Denmark, who commanded the sea to retreat. This was an obvious exercise in abject failure. However, never mind regulatory risk. Look at your business processes and see what they look like; it may not be a pretty sight. Incorporate calculations into instrument data systems or use informatics applications such as ELN or LIMS to move away from spreadsheets into an electronic way of working. Slaughter spreadsheets, please. You know it makes sense.

Warning Letter Tismore Health and Wellness Pty Limited

, Part 211 (FDA, Silver Spring, Maryland, 2008).

F. Hermans and E. Murphy-Hill, “Enron’s Spreadsheets and Related Emails: A Dataset and Analysis,” paper presented at the IEEE/ACM 37th IEEE International Conference on Software Engineering, Florence, Italy, 2015.

Good Automated Manufacturing Practice (GAMP) Guide

 (International Society for Pharmaceutical Engineering, Tampa, Florida, 2008).

WHO Technical Report Series No.996 Annex 5 Guidance on Good Data and Records Management Practices (World Health Organisation: Geneva, Switzerland, 2016).

, Part 21.11 (Food and Drug Administration, Washington, DC, 1997).

European Commission Health and Consumers Directorate-General, 

The Rules Governing Medicinal Porudtcs in the European Union

Good Manufacturing Practice (GMP) Guidelines, Annex 11: Computerised Systems 

Guidance for Industry Data Integrity and Compliance With Drug CGMP Questions and Answers

 is the director of R.D. McDowall Limited and the editor of the “Questions of Quality” column for LCGC Europe, Spectroscopy’s sister magazine. Direct correspondence to: 

Articles

Spreadsheets are often used to perform GMP-related calculations, but can lead to serious problems and unnecessary risk. We explain why the use of spreadsheets is heavily discouraged in a regulated laboratory environment.

Spreadsheets: A Sound Foundation for a Lack of Data Integrity?

The U.S. Food and Drug Administration (FDA) recently issued an updated version of its Compliance Policy Guide for Pre-Approval Inspections (PAI). If you are not aware of the impact on analytical procedures and data included in regulatory submissions, then please read on.

	One of the U.S. FDA’s Compliance Program Guides (CPG), numbered 7346.832, is entitled Pre-Approval Inspections. This document is used to guide and inform inspectors conducting this type of inspection before granting a license to market an active pharmaceutical ingredient (API) or medicinal product for the U.S. market. The Able Laboratories fraud case in 2005 was a turning point for this document. The Agency had conducted seven PAIs of Able and not found any evidence of falsification. A whistle-blower in the quality-control (QC) laboratories at Able then informed the local FDA field office of the data integrity issues in the company. The following day, a posse of inspectors rode into town and camped at the Able facility looking at electronic data and comparing these records with the printouts that had been the focus of earlier inspections. Their findings are summarized in the 483 observations issued in July 2005 that resulted in the company’s going bankrupt. The Able 483 is still available on the FDA web site (1).

	The FDA realized that their inspection approach of paper review was inadequate, and that inspections had to focus on the electronic records in laboratory computerized systems. As a result, paper records became incidental to the inspection of computerized systems. This required rewriting the existing CPG 7346.832 (yes, there is a “Star Trek” fan working at the FDA!), and the new version was issued in May 2010 and became fully effective in May 2012 (2). This gap allowed time for industry to understand the new inspection approach, as well as time to train inspectors on ways for inspecting electronic records within the various data systems used in regulated GMP laboratories.

	The key changes in the rewritten version were the three inspection objectives (2):

		Objective 1: Readiness for Commercial Manufacturing

		Objective 2: Conformance to the Application

	Reading this list of objectives, one could be forgiven for  assuming that the only data integrity focus was objective 3. However, you would be mistaken; data integrity runs throughout all three objectives. This inspection approach, coupled with how to inspect electronic systems, has been used since 2012, and as a result there have been an increased number of 483 observations and warning letters issued to laboratories concerning data integrity.

Updating CPG 7346.832, Take 2: September 2019

	September 2019 saw the issue of an updated version of CPG 7346.832 that still contains the three objectives, but the reason for the revision is stated on the front of the document as (3):

	Program revised to add instructions for potential official action indicated (pOAI) reporting responsibilities and to align with the Center for Drug Evaluation and Research (CDER) and Office of Regulatory Affairs (ORA) agreement Integration of FDA Facility Evaluation and Inspection Program for Human Drugs: A Concept of Operations.

	On the front of the document, the implementation date is September 16, 2019, with completion by September 16, 2022. Thus, the document is effective now, but I think that inspectors will need training in the additional scope and content, and therefore it will be fully effective later rather than sooner. Before we discuss the new CPG in more detail, we need to understand what a pOAI is. Readers who know the classification of 483 observations can skip the next section.

Understanding a Form 483 Inspectional Observation

	To the uninitiated, let me take a short detour to explain how the Agency classifies non-compliances on an FDA Form 483–Inspectional Observations. The completed Form 483 (typically shortened to a 483) is left with senior management at the close out meeting of an FDA investigation, and it lists the non-compliances found during an inspection. The inspector does not classify them at the closing meeting; classification is undertaken after the inspection, and is subject to Agency review that takes into account each non-compliance, its seriousness, and compliance history of the inspected organization.

	There are three classifications that can be given by the FDA to each 483 observation; these can be found in the exotically titled Field Management Directive 86 (FMD 86) Establishment Inspection Report Conclusions and Decisions (4):

		NAI–No Action Indicated: No objectionable conditions or practices were found during the inspection, or the significance of the documented objectionable conditions found does not justify further FDA action.

		VAI–Voluntary Action Indicated: Objectionable conditions were found and documented, but the District and/or Center is not prepared to take or recommend any of the regulatory actions, because objectionable conditions do not meet the threshold for regulatory action. The district may request a written response from the establishment, but this is not necessary. Any corrective action is left to the establishment to take voluntarily.

		OAI–Official Action Indicated: Objectionable conditions were found and regulatory action should be recommended. Includes inspections resulting in recalls where the district has decided conditions warrant regulatory action.

	You can see the spectrum of potential classifications for each 483 citation. Typically, a classification of OAI results in a warning letter sent to the company and exhibited in all its glory on the FDA website. For a foreign company (outside of the USA) their products may also be placed on import alert; oh, joy of joys. You can now see that is better to be compliant than not. Furthermore, the cost of compliance (the cost of doing it right) is always cheaper than the cost of non-compliance (the cost of getting caught) (5).

	Returning to the reason for the new version of CPG 7346.832 above, you can see that one of the reasons for updating the document is to add instructions for potential official action indicated (pOAI) reporting responsibilities (3). These are instructions to inspectors on how to report serious non-compliances that could result in a warning letter to the organization.

	You may ask why we should discuss this document now, given that it doesn’t become fully effective until September 2022. A good point. However, consider that any analytical development work conducted that is part of a regulatory submission to the FDA comes under this new version of CPG 7346.832 (3), hence such work may be inspected against these more detailed requirements.

	The structure of CPG 7346.832 is shown in Figure 1, and discussed in the following sections of this column. Please note that while in Figure 1 there are areas of reformatting and enhancement, much of the 2010 wording has been retained in the new version. This column looks specifically at the high-level impact on a regulated laboratory, but to gain a full understanding of all of the changes, you will need to read and compare both versions of this document.

	Let us look in more detail at each of the objectives in the new CPG document. Remember that I mentioned that the old version has three objectives with data integrity running through them? Now is your opportunity to see and understand the impact of this new version of the document.

Objective 1: Readiness for Commercial Manufacturing

	Objective 1 in the new version simply states:

	Determine whether the establishment has a quality system that is designed to achieve sufficient control over the facility and commercial manufacturing operations (3).

	You may be thinking from the title that this objective only applies to the manufacturing side, and analytical scientists can relax and put their feet up. Sorry, the new version of CPG 7346.832 retains the five sub-objectives from the old version that focus on the overall Pharmaceutical Quality System (PQS). All of these sub-objectives have impacts on a regulated laboratory, from cleaning validation through sampling to release testing (3):

Manufacturing and laboratory changes, deviations, and trends relating to the development of drug substance and drug product manufacturing have been adequately evaluated.

	There is an inspection focus on the changes, deviations, and trends within the laboratory on the PAI menu. While this is a feature of all inspections now, it is the changes around a new product that will be subject to scrutiny.

A sound and appropriate program for sampling, testing, and evaluating components (including active pharmaceutical ingredients [APIs]), in-process materials, finished products, containers, and closures for purposes of releasing materials or products has been established, including a robust supplier qualification program.

	The laboratory does not escape from this sub-objective; you are in it up to your collective necks. If you outsource any analytical work, then how did you select the facility, how much due diligence was performed, and what do the quality and technical agreements cover? A good overview of the requirements for outsourcing is EU 

Sufficient facility and equipment controls are in place to prevent contamination of and by the application product (or API).

	If your laboratory conducts cleaning validation, guess what? This one’s for you.

Adequate procedures exist for batch release, change control and investigating failures, deviations, complaints, and adverse events, and for reporting this information to FDA.

	How are batches released, and what is the role of laboratory data in this process? Here, the inspector asks whether the procedures for various investigations are adequate rather than investigating deviations for the product itself, which is covered in Objective 1a. Obviously, there will not be any complaints or adverse events, as the product has not been licensed yet; the inspection will focus on determining if the procedures are adequate and meet applicable regulations.

The proposed commercial process and manufacturing batch record, including instructions, processing parameters, and process control measures, are feasible and scientifically and objectively justified. This objective is linked to the firm’s process validation program across the product lifecycle.

	Do your analytical procedures have sufficient detail? Do you collect all records in an analysis? How do you process your analytical data to generate the reportable result? Where are the analytical records stored? What data integrity and good documentation practices are there, and how are staff trained?

	Objective 1 in total is pervasive throughout both the organization and specifically any laboratories involved in method development, validation of analytical procedures, sampling, testing, product release, supplier qualification (contract analytical laboratories), change control, deviations and investigations. And by the way, everything the laboratory does must be scientifically sound, including any investigations for invalidating out of specification (OOS) results.

		Where is the documented evidence that you follow these procedures?

Objective 2: Conformanceto the Application

	Verify that the formulation, manufacturing or processing methods; analytical (or examination) methods; and batch records are consistent with descriptions contained in the CMC section of the application. This may include CMC information relevant to exhibit batches, biobatches, other pivotal clinical batches, and the proposed commercial-scale process (3).

	Unlike the first objective, with a neat and structured list of sub-objectives, the areas for inspection under this objective are presented like a shopping list. Table I lists the pertinent inspection areas for a regulated laboratory together with the inspection focus within each area. Please note that Table I is an edited list; please read the actual document for the full scope of Objective 2. Note that there is a recurring theme in this objective of checks from laboratory data on-site back to the application submitted by the organization.

	Some of you may be confused by the term 

 in Table I; this refers to the manufacturing batch used to make supplies for pivotal clinical studies. It can also be called a 

and is different from commercial batches as the amount synthesized is smaller, and typically there is no predetermined shelf life; rather, the product stability is determined as an ongoing process. As such, it is an inviting inspection target, especially for data integrity.

	Analytical procedures that will be high on a list of targets in a PAI will be those that are unique to the product under inspection rather than pharmacopoeial methods. In particular, have any procedures been modified since the application was submitted?

	One of interesting asides in the new CPG is the concept of factual and contextual data integrity discussed in a footnote at the end of Objective 2. In the context of a PAI, these terms are defined as:

		Information that has factual integrity is original and corresponds directly to that submitted to FDA (for example, a chromatogram showing a peak area that directly calculates to an assay value submitted in a data summary sheet in the application) (3).

		Information that has contextual integrity supports submitted information about the testing or manufacturing area and related products or processes (for example, a chromatographic sequence that shows all the assayed samples and that does not reveal failing assay values).

		Missing records (batch or testing) and unexplained losses of inventory of components used in production may call into question the contextual integrity of the information filed in an application (3).

	Figure 2 shows a hybrid spectrometer system at the bottom that has generated data and records that have been included in an application to the FDA at the top of the page. The journey from the instrument to the application takes two routes; one for factual integrity, and the other for contextual integrity. 

 ensures a match between data and information in the application and the electronic records and associated paper printouts (for hybrid systems) in the data systems of the organizations’s laboratory. In contrast, 

is an assessment that the organization has not been selective or fraudulent in including data in an application. This concept of factual and contextual integrity provides a bridge between Objectives 2 and 3, as data integrity issues identified in the former provide an input to the latter.

	Audit and verify raw data at the facility that are associated with the product. This information can, among other things, help to authenticate the data submitted in the CMC section of the application as relevant, accurate, complete, and reliable for CDER assessment (3).

	Similar to Objective 2 and dissimilar to Objective 1, there is a shopping list of items that have been extracted, edited, and ordered into topic areas below and as shown in Figure 1.

	Audit the accuracy and completeness of data reported by the facility for the product (3):

		Select key data sets from drug development or randomly select data from the application. Generally, data on finished product stability, dissolution, content uniformity, and API impurity are good candidates for this audit.

		Compare raw data-hardcopy or electronic-such as chromatograms, spectrograms, laboratory analyst notebooks, and additional information from the laboratory with summary data filed in application. Raw data files should support a conclusion that the data/information reported by the site are complete and accurate.

		When data discrepancies are observed, identify the firm personnel involved. Determine which actions or inactions contributed to the data integrity problem, and whether corrective actions were or are to be taken.

		Also determine whether data that should have been reported in the application were not reported.

	The new version of the CPG shows some indicators of possible laboratory data integrity problems and I have collated them in Table II. The aim of Objective 3 is to demonstrate the reliability of the data in a submission. However, if data are unreliable, the new CPG version suggests that an inspector can expanding the coverage of the inspection to marketed products under Compliance Program 7356.002 for Drug Manufacturing Inspections (7), or that the FDA invokes the Application Integrity Policy (8). If either of these occur, the laboratory has begun the long and very uncomfortable decent into inspection hell.

	Reading this, you can see that the majority of data generated for a drug submission will come from the research and development (R&D) departments. Not all R&D records may be subject to GMP, which can give some R&D departments an air of the wild west, which only tends to come under control when QA comes calling. However, the tendency of R&D to be outside of GMP needs a note of caution as any records submitted to the FDA in Drug Master Files and New Drug Applications are subject to 21 

	This part applies to records in electronic form that are created, modified, maintained, archived, retrieved, or transmitted, under any records requirements set forth in agency regulations. This part also applies to electronic records submitted to the agency under requirements of the Federal Food, Drug, and Cosmetic Act and the Public Health Service Act, even if such records are not specifically identified in agency regulations.

	R&D data included in FDA drug applications must comply with Part 11 requirements regardless, and GMP regulations where applicable. The validation of analytical procedures, any testing of components, APIs, and so forth must have the same regulatory approaches as QC.

	However, there are also key business reasons why R&D and QC should have the same approaches to instrument qualification, computerized system validation and validation of analytical procedures and testing:

The Food and Drug Administration (FDA) has issued an updated version of its Compliance Policy Guide for Pre-Approval Inspections (PAI). This article will guide you on this policy’s impact on analytical procedures and data included within regulatory submissions.

Do You Understand the FDA’s Updated Approach to Pre-Approval Inspections?

Data Quality and Data Integrity Are the Same, Right? Wrong!

Do the terms data integrity and data quality mean the same? As “maybe” and “yes” are not the correct answers, we will explore the differences between the two terms, and how data integrity is a key part of data quality. 

	
	Data integrity has been the subject of many “Focus on Quality” columns over the past few years (1–5). However, we have never mentioned, let alone discussed, data quality. In many people’s minds, 

 are terms that have equivalent meanings, hence the first part of the title of this month’s column. However, in reading the MHRA (Medicines and Healthcare Products Regulatory Agency, the UK regulator) GXP data integrity guidance, there is an indication from a regulator that the two terms are different (6). Newton and White have discussed the differences between data quality and data integrity in an ISPE blog in 2015 (7), and, in this column, we expand and explore the similarities and differences between the two terms.	

		The MHRA’s 2018 guidance has in the introduction section the following statement in clause 2.7:

		“This guidance primarily addresses data integrity and not data quality since the controls required for integrity do not necessarily guarantee the quality of the data generated (6).”

		Clearly, in the mind of the regulator, there is a difference between data quality and data integrity. But what exactly is it?

		Our data quality versus data integrity story starts toward the end of the last century with ALCOA-not the Aluminium Company of America, but an acronym standing for 

attributable, legible, contemporaneous, original,

. These criteria have has been discussed in several data integrity guidances from the Food and Drug Administration (FDA), MHRA, the World Health Organization (WHO), and the Pharmaceutical Inspection Convention and Pharmaceutical Inspection Co-operation Scheme (PIC/S) (6,8–10). The WHO guidance contains the most detailed discussion of ALCOA criteria in an appendix with a definition of each term, presentation of the expectations for paper and electronic records, plus special risk management factors (9).

		A European Medicines Agency (EMA) publication on electronic source data extended ALCOA to include an additional four terms of 

 (11). The extension of these four additional terms is known as ALCOA plus, or ALCOA+. A recent article in 

 discusses all nine ALCOA+ criteria in more detail (12).

		Reading further through the MHRA guidance, there is clause 3.10, a part of which is presented below, discussing ALCOA and ALCOA+ criteria:

		“….. ALCOA was historically regarded as defining the attributes of data quality that are suitable for regulatory purposes…. [6]”

	This is interesting; ALCOA criteria were originally defined as the attributes of data quality? How did this come about? The ALCOA acronym was invented by an FDA good laboratory practice (GLP) inspector named Stan Wollen, who developed it as an aide memoire to remember the key requirements of data quality. This is outlined in a publication entitled “Data Quality and the Origin of ALCOA” (13), available for download on the web. There is a lot of background information together with Wollen’s interpretation of ALCOA under the US GLP regulations 21 

 58 (14). For our discussion, ALCOA was originally associated with data quality, as noted by the MHRA in its GXP guidance document (6), long before it was stolen by various guidance documents for use as a cornerstone of data integrity.

	Before we go into more detail about the two terms, we need to define them. The definitions for both 

 are taken from the 2018 MHRA GXP Guidance for Data Integrity (6), and are presented in Table I. We will consider and discuss the data integrity definition first, and then spend time looking at data quality. I am not going to discuss the last paragraph in the right-hand column of Table I about sound science and risk management, although those are key inputs to data integrity, but not part of the data integrity and data quality debate in this column.

	What is interesting about the 2018 definition of data integrity is how far it has changed since the first MHRA guidance published in 2015. The earlier definition was:

	“The extent to which all data are complete, consistent and accurate throughout the data lifecycle (15).”

	This definition is a modification of a 1995 FDA definition of data integrity (16). The 2015 definition can be compared with that published in 2018, where the additions are highlighted in bold text:

 to which data are complete, consistent, accurate, 

trustworthy, reliable and that these characteristics of the data are maintained 

	There is the addition of two criteria (trustworthy and reliable), as well as a requirement to maintain all five data integrity characteristics throughout the data life cycle.

	What is not stated in this definition, but in the explanation for raw data, is that dynamic data must be maintained in this format throughout the data life cycle. We shall return to this regulatory expectation, and the total inability of industry and suppliers to meet it, in a future “Focus on Quality” column.

As discussed above and in Table I, we have additional criteria listed in the MHRA definition: complete, consistent, trustworthy, and reliable, in addition to the five original ALCOA criteria (6). Two are ALCOA+ criteria that will be discussed below, plus two new requirements. ALCOA+ on steroids perhaps? Let us look at these four criteria, which to make matters worse are not defined in the guidance document:

 This is one of the ALCOA+ criteria, and is also an FDA 

 regulatory requirement in 21 CFR 211.194(a) where 

 are mentioned (17). Put simply, every item of data and metadata collected during an analysis from sampling to generation of the reportable result is covered under “complete,” including (but not limited to) any instrument log entries, documentation of mistakes, investigations, deviations, and instrument and calibration failures. Of course, you won’t be allowing any laboratory user deletion privileges, will you?

 Another of the ALCOA+ criteria that requires that the data be consistent with the processes being followed, and that the time and date stamps reflect the creation, modification, and audit trail entries are as expected. The process (manual or automatic, hybrid or electronic) should ensure that the acquisition, transformation, or calculation of data is transparent and traceable.

This is a new data integrity requirement that, together with “reliable,” harks back to the last century and the requirements outlined in 21 

 11 for electronic records and electronic signatures (18). In essence, can you trust the data or records? In some respects, it is the reverse of the ALCOA criteria. For example, if a record is attributed to an individual (ALCOA requirement), did that person actually do the work? Was the person in the laboratory at the time the work was performed, or did somebody act on their behalf?

 Another new data integrity requirement that means that all records and data are suitable and can be depended upon. Where a true copy has been made of an original record, is the verification process acceptable? The record set is not the 

th iteration of testing to pass and that work has actually been performed as required under EU GMP Chapter 1.8 and 1.9 for in process and quality control testing, respectively (19)?

	The addition of trustworthy and reliable to the ALCOA+ criteria are good. So, what does this mean for data integrity as a whole?

	Data integrity provides assurance that the analytical work in the laboratory from the sampling to the calculation of the reportable result has been carried out correctly. In short, can you trust and rely on the analysis and the reportable results? If you cannot trust the analysis and the data, then you cannot make a quality decision based on poor, incomplete, or falsified data.

	It is the role of the analytical scientist and the second-person reviewer to ensure data integrity, but that also requires that the instruments, computer systems, analytical procedures, and laboratory environment are set up correctly, and that the organization within which they work ensures that they are not pressured to cut corners or falsify data.

	Therefore, ensuring the integrity of data is a major contributor to data quality. But, in focusing on data integrity, we must not lose sight of data quality.

in Table I that there are three main elements: Data produced is exactly what was intended to be produced       It is fit for its intended purpose

	Hmmmm, what do these mean in practice? Let us look at item 1, that data produced is exactly what was intended to be produced. The problem with this is the wording of the definition. Although a laboratory produces mountains of data, we abstract and use the information within them to make decisions. For a detailed discussion of this process together with the controlled and uncontrolled parameters, see the article by McDowall and Burgess (20). Therefore, the purpose of a laboratory is to generate information from the analysis of samples in order to make decisions. These decisions could be to aid product development or release a batch of product.

	To do this requires trained staff, qualified analytical instrumentation, validated software, and analytical procedures verified or validated under actual conditions of use, as required by 21 

 211.194(a)(2) (17). This is the same as the first three levels of the data integrity model discussed in earlier articles (21–23). Where the second item comes in is that the output of the analytical procedure must be fit for use. As a simple example, there is no point calculating results of an analysis to three significant figures if all that was required was the presence or absence of the analyte.

	Once you have ensured the integrity of data, we must consider data quality. In part, this focuses on the laboratory’s ability to deliver the results to the sample submitter and meet their requirements for the analysis such as:

		Precision and accuracy or measurement uncertainty of the result

		Supporting information for the analysis

		Speed of analysis (for example, for releasing a production batch or turnaround of analysis from a clinical trial)

	However, there is a compromise with data quality that is best illustrated by the data quality triangle.

There is a problem with the definitions of 

 that is not mentioned by the MHRA and the American Health Information Management Association (AHIMA); this is the data quality triangle, as shown in Figure 1. At each end of the triangle are the following attributes:

		Cost: cost in instrumental and human resources

		Quality: delivering data of acceptable quality to the end user of the information to make a decision.

	The problem is that you can only select two of the three at any one time. For example, if you want a fixed time and high quality, how much are you willing to pay or how much resource will you commit? Equally, if you want analysis performed quickly and at low cost, quality suffers. The quality triangle is applicable to any laboratory in any industry, but in a regulated laboratory you still must ensure data integrity as well. The influence of senior and laboratory management on managing the three criteria of the data quality triangle will directly impact both data quality and data integrity.

Integrity and Quality Use the Same Data Set

	It is imperative to realize, as mentioned earlier, that the data set for ensuring data quality and data integrity is the same, and not two different ones. Furthermore, data integrity is not ensured first and followed by a second pass to ensure data quality. To ensure data quality and data integrity, both must occur in parallel, simultaneously and on the same dataset.

	Because the two terms are different, as Newton and White (7) point out, you can have data integrity without data quality, and vice versa. A laboratory can have excellent data integrity, but if the analytical information to make a decision is delivered late, or not at all, then the data quality is useless. Alternatively, if each laboratory is permitted to create its local practices for data management, you can have integrity without quality (or, at least, very low levels of quality). All the data are there, but consolidating them requires a standardization project: good integrity, lousy quality. That is the one place where quality and integrity do differ, and where the regulations are weak (Newton, personal communication).

	Newton and White (7) present a case where an organization outsources analytical work to contract laboratories, but there is little oversight of the work, resulting in data integrity lapses. When the results are received by the organization, entry to the LIMS controlled and delivery of the results to the decision makers is rapid, and with good data quality.

	The similarities and differences between data quality and data integrity are shown in Figure 2. The similarities are:

		Application of ALCOA and ALCOA+ criteria to the record set

		The differences between the two terms are trusting the data versus making a decision using the results.

	The problem and potential conflict comes with the third element of the MHRA definition for data quality that states that data quality includes ALCOA. This is consistent with Wollen’s original view when devising the acronym ALCOA (13). However, when we see the MHRA definition of 

 in Table I, each ALCOA criterion is spelled out in the definition. Puzzled?

	Is ALCOA an integral part of data integrity, which in turn is a component of data quality, or is ALCOA applicable to both terms, as noted in the MHRA definition?

	Look at it from the perspective of the data and records being generated. Can you separate the data for data quality from data integrity? The answer is no, because the record set for both data integrity and data quality will be the same. Therefore, ALCOA criteria apply equally to data quality and data integrity. Think it through; otherwise you could have some data where all actions were attributable for data integrity, but not for data quality.

	Figure 2 shows the relationship diagrammatically; in the middle is the analysis workflow from sampling to a decision based on the reportable result from the analysis. Throughout this process, data and records are generated that comprise the complete data from the analysis. This record set is impacted by the ALCOA criteria (as well as the four additional requirements) for data integrity mentioned in the MHRA definition in Table I and the ALCOA criteria for data quality.

Data integrity has been the subject of many “Focus on Quality” columns over the past few years (1–5). However, we have never mentioned, let alone discussed, data quality. 

Analysis of 47 FDA Form 483 observations and warning letters for infrared spectrometers reveals a spectrum of data integrity problems and a lack of laboratory procedures for the technique. Is this your laboratory? 

	Since the Able Laboratories data falsification case in 2005 (1), there has been a regulatory focus on the lack of data integrity with chromatography data systems. This scrutiny has focused on inappropriate access privileges with conflicts of interest, deletion of data files, failure to configure the application, testing into compliance, and failure to review audit trail entries (2). More recently, inspection focus has moved to the use of peak integration to integrate into compliance and masking potential impurities (3). There have been many warning letters as a result of these noncompliances and poor data management practices; this is due to chromatography being a major technique that can comprise between 40–70% of a laboratory's workload. You may ask, "Why the focus on chromatography in a publication on spectroscopy?" Just think of chromatography as sample preparation for spectroscopy!

	In contrast, there are relatively few FDA warning letters involving spectroscopic techniques. An analysis of FDA warning letters citing infrared spectrometers was published in 2014 (4).

	Originally, we were going to write a column looking at some of the data integrity issues that could occur with Fourier transform infrared spectroscopy (FT-IR) analysis, with a few regulatory citations to illustrate each point. However, given the bonanza of Form 483 observations and warning letters that have been issued by the FDA, we realized that an analysis of these would provide a great overview of the current regulatory problems facing the technique, before discussing specific topics in the future.

	In this column, we present an analysis of 47 Form 483 observations and warning letters issued between 2012 and 2018 (5–51). It should be noted that many of these Form 483 observations are not generally available publicly. Therefore, these observations, the regulatory focus on infrared, and the conclusions we have drawn may be new to laboratories that only review FDA warning letters. We have taken each citation or observation and broken it down into one or more data integrity areas; therefore, a single observation can have one or more data integrity noncompliances. For example:

		The application does not have an audit trail. This is a single 483 observation and a single data integrity issue.

		In contrast, here is a more complex citation: System audit trails are not available on the two FT-IR instruments, and quality control (QC) operators have the option of not saving the IR data. This citation will be classified into two entries; one of no audit trail, and another of application software design because data are not automatically saved. This is a simple example of compound data integrity problems, but some citations can be far more detailed and complex.

	Therefore, there is not a one-to-one correlation between all non-compliances analyzed and the data classification presented here.

	The noncompliances have resulted in a total of 104 separate citations for IR analysis that have been classified and discussed below. Some of these are self-explanatory, and will not be discussed in part due to space considerations and because the solution is obvious (such as no software validation, for example). In the references, we have included the FDA Facility Establishment Identifier (FEI), which is the way the FDA identifies a specific site in a company. Where there are different FEI numbers for the same organization, this means that two sites or facilities were inspected.

	Our main challenge in this analysis is to interpret a regulatory citation, because we don't know all the circumstances at the time of the observation by an inspector. For example, we don't know the functionality of the application software, so the observation of no system audit trails available as shown above could be a true software design limitation, or, more likely, it could be that the users did not enable this function. We have used our experience and best judgement in classifying each citation in this analysis.

	We do not claim that this noncompliance review reflects the total number of 483 observations issued, but, based on the significant number, it is reasonably comprehensive to draw firm conclusions on regulatory gaps involving IR analysis. Enjoy reading about the calamities of others. If any noncompliance mentioned here applies to your laboratory, what are you going to do? We suggest that you remediate the problem before you are included in a future update of this article!

Citations Before the Operational Use of the Instrument 

	One of the surprises of our analysis is that 42% of the 483 citations occur before the instrument and the associated software application are supposed to become operational. You may doubt this statement, but look at both Table I and Figure 1 and you will see two main areas: inadequate software architecture, and design and failure to qualify, calibrate, or validate the system. The interpretation of these nonconformance data is based on the wording of the 483 observations.

Analysis of FDA Form 483 observations and warning letters for infrared spectrometers reveals a range of data integrity problems and a lack of laboratory procedures for the technique. Is your laboratory in the same situation?

Author | R.D. McDowall

Articles