Since its inception in 1961, the mission of the Laboratory Accreditation Program (LAP) of the College of American Pathologists (CAP) has been to improve the quality of laboratory performance.1 Between biennial on-site inspections, the LAP monitors the performance of participating laboratories using the scores of proficiency testing (PT). Underlying the process is the assumption that PT will uncover problems that may affect the clinical care of the laboratory's patients.
Every PT program that is recognized by the CAP uses normative statistics to determine the acceptability of participants' results. Limits based on 95% confidence intervals, for example, will score 1 result in 20 as unacceptable on the basis of population variance alone. In the same way, for a PT event that has 5 challenges, subscribing laboratories will, on the average, report 2 or more results beyond calculated limits about 2% of the time, even without systematic bias or analytic problems.
The Clinical Laboratory Improvement Amendments of 1988 defined successful performance for most analytes as at least 80% of reported results within acceptable limits.2 Having 2 unacceptable responses out of 5 challenges (unsatisfactory performance: Table 1) leads to regulatory implications. As an accreditation provider under the Clinical Laboratory Improvement Amendments of 1988, the CAP must investigate unsuccessful PT performance.
The LAP uses the Proficiency Testing Exception Summary (PTES) algorithm to detect deviations from acceptable performance that may affect the accreditation of a laboratory. The overarching PTES process combines evaluation of performance data as scored by the laboratory's PT provider with explanatory data as reported by the participating laboratory to the LAP. The combination generates information useful in determining the need for further intervention. The CAP PTES program has the following characteristics:
1. Unsatisfactory performance for any gradable analyte can be tracked over 2 or more sequential testing events. Two such performances in a row, 2 unsatisfactory performances out of 3 testing events, or 3 out of 4 are flagged by the program.
2. The PTES algorithm may be applied to a graded challenge regardless of its regulatory status. The PTES is not a regulatory tool. It is an internal routine used exclusively by the LAP. Table 1 compares the scope of the PTES with that of the Clinical Laboratory Improvement Amendments of 1988.
3. Performance data from any approved PT provider may be included. The college recognizes PT programs other than its own. Among the requirements for recognition by CAP is that the provider submit data to the LAP in a format that can be read by the PTES algorithm.
4. Clusters of unacceptable results are extracted for review by the LAP. Repeated identification of an analyte by PTES prompts closer review.
Identification of a cluster of unacceptable results by PTES prompts a round of correspondence between the LAP and the laboratory's director. Any response from the laboratory that describes the results of the laboratory's investigation closes the initial dialogue.
Should the laboratory report unacceptable results in a subsequent testing event, the PTES program continues with a second round of correspondence. Because the expectation is that corrective action has been implemented, the LAP views repeated unsuccessful performance as a serious situation that requires definitive resolution.
At the end of each evaluation period, PTES also compares PT results with the menu of testing activities for that laboratory. Failure to have submitted results is considered an unsuccessful testing event that initiates PTES correspondence.
We reviewed the results from recent experience to determine whether the PTES program is achieving its stated goal: determine whether laboratories are correcting significant problems as identified by PTES.
MATERIALS AND METHODS
A 12-month survey year beginning in 2002 was studied. During that time, the CAP accredited approximately 6300 laboratories. CAP-accredited laboratories collectively tested about 1205000 analytes and reported on approximately 3 524 000 annual PT challenges.
The portion of the PTES algorithm that monitors PT scores was run 7 to 10 days after those scores were submitted by the PT provider. In most cases, the CAP was the PT provider, but the algorithm was applied to data submitted by any provider approved by the LAP. Each PTES report was mailed to the laboratory director with a request for a written reply.
In each round of correspondence, the laboratory director was asked to describe the apparent cause of the PTES event and the corrective actions taken. Instructions for categorizing the problem were provided. Cases that offered no useful data (eg, enrollment error, wrong laboratory identification number, or test no longer performed) were excluded from the analysis.
Three rounds of correspondence were compared. The first round followed a reported unsatisfactory PT event. Included with the LAP letter to the laboratory was a brochure entitled "How to Respond to a Proficiency Testing Exception Summary."3 The brochure includes a list of suggestions for investigating the event as adapted from an NCCLS guideline4 and helpful examples.5 Each response was scored only for completeness. Reminder letters were sent if the laboratory did not respond or if the response omitted required data.
Correspondence from the second round was reviewed by one of us (M.G.P.) for content. Items looked for at this stage were a description of how the problem was investigated, whether the cause of the repeated unsuccessful performance was identified, specific corrective action to prevent recurrence, and-ideally-evidence that the problem was corrected. Letters to gather additional documentation were sent as necessary.
For the third round, documentation was expected to indicate that the problem was corrected or that the laboratory had ceased testing for that analyte. All correspondence at this level was reviewed by 2 of us (M.G.P. and G.A.H.) for adequacy. By policy, inadequate responses to third-round correspondence are referred to the CAP Commission on Laboratory Accreditation for action.
Responses to the third round of correspondence were tallied as to category of apparent cause and whether the problem had been resolved on subsequent testing events.
RESULTS
There were 14085 PTES letters generated in the first round of correspondence, 1304 in the second round, and 119 in the third round. Table 2 details the totals. Regulated tests are those tests for which PT (3 challenges per year, 5 samples per challenge) is required by the Centers for Medicare and Medicaid Services for all laboratories performing tests on such analytes. The total for this category includes all results for such analytes, whether or not they were reported to the Centers for Medicare and Medicaid Services for regulatory purposes. Round 2 correspondence occurred only 7.6% as frequently as round 1 correspondence for regulated tests, and 99.0% of the PT performance problems were resolved before a third round of correspondence was triggered.
Second-round correspondence for nonregulated tests occurred 11.3% as frequently as first-round correspondence; 99.0% of the problems were resolved before reaching the third round.
Subspecialty testing (principally microbiology results, which are grouped and regulated by the Centers for Medicare and Medicaid Services according to subspecialty), toxicology, and gynecologic cytopathology results were counted separately. The number of PTES letters decremented similarly over time.
Of the 119 cases from all categories that reached a third round of correspondence, 115 were resolved. In 106 cases, subsequent PT performance was acceptable, and the laboratory chose to discontinue testing in 9 cases. Four cases remained unresolved; these were all international PT subscribers. Deterioration of the PT material in transit was presumed. Of the third-round cases, the results of 41 were attributed to methodologic problems, 42 to technologic problems, 28 to clerical issues, and 8 to problems with the survey material. No case lacked an explanation after investigation.
COMMENT
Proficiency testing is only one way to measure laboratory quality. As a metric for interlaboratory variation, it is a useful tool to ensure comparability of results between testing laboratories. Originally designed as an educational tool for self-evaluation,6 external PT took on new meaning after adoption by regulatory agencies. Accrediting programs such as the college's LAP must require their subscribers to participate in PT activities as defined by law.
The LAP endeavors to meet all regulatory requirements while maintaining laboratory improvement as its primary goal. Laessig et al7 and many others have commented on the difficulty in reconciling a quality-improvement goal with a regulatory mandate. Participants in the CAP LAP have elected external review beyond any regulatory mandate. Of the 423 graded analytes offered by LAP-approved PT providers, only 97 are required by Centers for Medicare and Medicaid Services regulation. Participation in PT for nonregulated analytes is a condition of accreditation, and laboratories that have chosen CAP as their accreditation provider have agreed to the expanded list.
The PTES program identifies potentially correctable problems. None of the performance difficulties flagged by a PTES report was unknown to the laboratory at the time of PTES correspondence. A PTES report presents data in a format that may focus a busy laboratory director's initial attention on an important problem. Educational materials are provided to assist laboratory workers' investigation and resolution of the problems. In addition, the scientific resource committees of the college stand ready to answer participants' questions about the CAP Surveys Program.
The LAP adds on-site inspection to the laboratory's tools for quality improvement. Data from PTES are available to the on-site inspector. Follow-up may include recommendations to the laboratory for further opportunities to improve care. When repeated problems as flagged by PTES are evident, the LAP has the information needed for intervention. A laboratory may, for example, be directed by the CAP Commission on Laboratory Accreditation to discontinue clinical testing until the problem has been fully resolved.
The limitations of PT as a stand-alone tool for laboratory improvement were summarized in a review by Shanangian,8 who concluded that 4 issues challenge any PT program: (1) the inability of PT to address the total testing process; (2) the inevitable differences between PT materials and fresh biological specimens; (3) inadequacies of PT evaluation criteria; and (4) the matrix effects unique to a particular PT system. Combining on-site inspection with the monitoring of PT addresses the first and second of these limitations. The LAP's criteria for approbation of a PT provider's program targets the third. All PT providers seek to minimize matrix effects as they are identified.
Klee and Forsman9 noted that external PT programs from each of 3 different providers introduced errors that were artifacts of the PT process itself. Such attributions decrease in frequency with subsequent rounds of correspondence with participating laboratories. Experience with the survey tool and improvements in the survey process both seem to contribute to this improvement. Even so, making errors in reconstituting PT specimens, reporting results in the wrong units, coding for the incorrect peer group, and similar problems persist even to the third round of correspondence. Such errors must be resolved for the value of interlaboratory comparison to be realized.
Parts of the improvement in external PT scores are attributable to better control of intralaboratory analytic processes. For this reason alone, PT is a legitimate component of the accreditation routine. Studies using CAP data have demonstrated that careful attention to analytic linearity (as evidenced by participation in the CAP Linearity Survey) is positively associated with fewer PT failures.10,11 Similarly, participation in the college's LAP was demonstrated by Lawson et al12 generally to correlate with fewer unacceptable PT scores. The present study illustrates how interpretation of these 2 measures of laboratory quality-proficiency testing plus inspection and accreditation-combine to target specific opportunities for analytic improvement.
The categorization of the causes of PT failures used in this study was proposed in 1987.13 With each PTES letter, the list reproduced in Table 3 is provided. The list of categories is provided to help a laboratory think through the problem. The category "no explanation after investigation" is unreasonable for repeatedly unsuccessful results (rounds 2 and 3).
In 41 cases (34% of the third round) a systematic problem in the method was detected. The cause was usually identified as a faulty instrument (20 cases). In 42 cases (35%), 1 or more technical errors were revealed. Twelve of these were repeated misidentifications of clinical microscopy or parasitology images. The need for remedial training in these and other technical cases was clear.
In a CAP Q-Probes study of 665 laboratories, Steindel et al14 reported that the rate at which the problems with the survey material contributed to PT failures varied from 4.3% to 29.7% among 8 PT providers, according to the participants in those programs. The rate of unexplained PT failures in that study ranged from 22.9% to 49.3%. Repeated unsuccessful performance demands an explanation. The present study provides at least indirect evidence that laboratories do identify contributory causes. Much of the improvement followed technologic or methodologic changes. A substantial number of laboratories decide to discontinue performing a problematic test when they realize that their difficulties are tied to the infrequency with which they perform the test.
The PTES program provides no novel information to the laboratory participant. By the time PTES correspondence is received, the laboratory should already have investigated and corrected the problem. The LAP expects each unacceptable response-including those that do not cluster to the level of unsatisfactory performance-to be investigated. Isolated PT failures are approximately 14 times more common than PTES clusters (unpublished observation). Accredited laboratories are investigating many more PT failures than are reflected in PTES statistics.
This study represents about 1% of analytic testing by accredited laboratories (14 085 first-round PTES reports among 1 205 000 analytes). The data show that laboratories are successful at correcting systematic problems within their LAP experience. Moreover, these are the problems that may have remained overlooked had they not been identified by the PTES program.
To document an investigation for external review like this requires firsthand involvement by the laboratory director. Spotlighting a problem in performance helps focus remedial resources on an issue that may have gone unobserved. Accredited laboratories perform consistently well on PT surveys. Their documentation of problem resolution in those exceptions that reflect systematic problems is evidence of the value of these CAP programs to laboratory improvement.
© 2005 College of American Pathologists Provided by ProQuest LLC. All Rights Reserved.
Copyright 2005 Archives of Pathology & Laboratory Medicine
