John Carreyrou's bold investigative piece about Theranos that appeared in the Wall Street Journal has been widely quoted (see: Hot Startup Theranos Has Struggled With Its Blood-Test Technology). However, relatively unnoticed and deep in the article were his comments about possible proficiency testing (PT) errors by the company. If they prove to be correct, they could add another layer to the Theranos difficulties. Here's the relevant quote from the article:
In early 2014, Theranos split some of the proficiency-testing samples it got into two pieces, according to internal emails reviewed by the Journal. One was tested with Edison machines and the other with instruments from other companies. The two types of equipment gave different results when testing for vitamin D, two thyroid hormones and prostate cancer. The gap suggested to some employees that the [company analyzer] Edison results were off, according to the internal emails and people familiar with the findings....Former employees say [CEO Sunny Balwani]...ordered lab personnel to stop using Edison machines on any of the proficiency-testing samples and report only the results from instruments bought from other companies.The former employees say they did what they were told but were concerned that the instructions violated federal rules, which state that a lab must handle “proficiency testing samples…in the same manner as it tests patient specimens” and by “using the laboratory’s routine methods.”
In its everyday business at the time, Theranos routinely used Edison machines to test patients’ blood samples for vitamin D, the two thyroid hormones and prostate cancer, the former employees say. In March 2014, a Theranos employee using the alias Colin Ramirez alleged to New York state’s public-health lab that the company might have manipulated the proficiency-testing process. Stephanie Schulman director of the public-health lab’s clinical-lab evaluation program, responded that the practices described by the anonymous employee would be a “violation of the state and federal requirements,” according to a copy of her email. What the employee described sounded like “a form of PT cheating,” Ms. Shulman added, using an abbreviation for proficiency testing. She referred the Theranos employee to the public-health lab’s investigations unit. The New York State Department of Health confirms that it got a formal complaint in April 2014 “in regard to testing practices at Theranos” and forwarded it to the Centers for Medicare and Medicaid Services.
Here's a definition for proficiency testing (PT) according to CLIA (see: CLIA; Proficiency Testing; Do's and Don'ts):
Proficiency testing or PT is the testing of unknown samples sent to a laboratory by a CMS approved PT program. Most sets of PT samples are sent to participating laboratories three times per year. After testing the PT samples in the same manner as its patient specimens, the laboratory reports its sample results back to their PT program. The program grades the results using the CLIA grading criteria and sends the laboratory scores reflecting how accurately it performed the testing. CMS and accreditation organizations routinely monitor their laboratories’ performance.
So, to summarize, here's how I interpret Theranos' PT problems which are distinct from it's challenges relating to the validation and accuracy of its analyzer or its promise to perform multiple tests inexpensively using only a drop or blood:
- The company is reported to have split PT samples into two parts and tested one on its own analyzer, Edison, and the other on a standard analyzer. According to the story, this may have been an attempt to validate its own instrument. Aside from being against federal regulations, the approach also seems to make little sense because the true analyte values of the PT samples were unknown. Values between the two instruments, of course, could be compared.
- If the company was running vitamin D, two thyroid hormones and prostate cancer assays on the Edison analyzer but not using the same instrument for PT testing, the company would be out of compliance with federal regulations. Both this and the previous sample splitting could be the basis for lifting the lab's CLIA license or even worse under CLIA regulations.
- According to the article, all of these issues appear to have been the basis of a formal complaint to CMS via the New York State Department of Health. Some disciplinary action might be forthcoming if the complaints prove to be correct.