Revelations in In Vitro Diagnostic (IVD) Device Labeling - Proactive Prevention of Adverse Events by Informed Decision Making
MedSun: Newsletter #35, April 2009
By: Simon, K; Kondratovich, M
FDA Office of In Vitro Diagnostics
Picture this: a representative from a large, reputable company has just finished training your staff in the use of their best-selling IVD device. Each technician has passed their proficiency test and has been cleared by the company to run the assay. The instrument has been installed and calibrated and all validation runs are complete. You are ready to go, right? Wait! Don’t forget to carefully read the package insert; this document provides tips to help you ensure that any new IVD device will not be misused in your laboratory.
In clearing or approving a device, FDA must consider (among other factors) “the conditions of use prescribed, recommended, or suggested in the labeling of the device” (21 CFR 860.7). What does this mean? In short, an FDA cleared or approved device might not be safe and effective if it is not used in accordance with the device labeling. What is the pertinent information for you, the laboratory manager or the technician performing the test, to look for in each section of the labeling to help protect patients from adverse events?
This is the most important information in the device labeling, as FDA’s review of an IVD device is driven by the intended use of the device. FDA only evaluates, clears and approves a device for the intended use as stated in the labeling. This is the first item to look at in determining how to utilize the test appropriately in your laboratory. Important points to check in the intended use are the appropriate patient populations (target population), sample types, diseases or conditions (target condition), etc. For example, a test cleared for use in patients with signs and symptoms of a disease or disorder may have very different performance characteristics if it is used on asymptomatic patients. Off-label use in asymptomatic patients of IVD tests intended for symptomatic patients typically results in a relatively high proportion of false positive to true positive test results (in some cases, false positives can outnumber true positives 1 by several fold) – leading to inappropriate diagnosis and treatment of patients. Another common example would be a test where performance characteristics have been established in a pediatric or adult population, but not both. IVD test performance can vary greatly between adults and pediatrics, an example being IVD tests for influenza A/B. Pediatric patients seen by clinicians for influenza infections tend to present with significantly higher levels of virus in their nasal secretions than adults. Therefore, a test cleared only for use in a pediatric population may give a very high number of false negative results in an adult population, leading to a delay or absence of intervention, which could cause severe morbidity and mortality, particularly in elderly patients. It may be helpful to make certain the physicians that utilize your services understand the intended use of each test offered by your laboratory upfront, by having a comprehensive list of tests offered, which describes the intended use of each test. If a physician unknowingly orders a test for a specimen type which is not listed in the intended use, it may be important for you to inform the physician that the test has not been cleared for that specimen type (referencing your test menu with the intended use).
This section of the labeling contains key information about how the device may be limited in its ability to provide accurate results. What? FDA clears and approves IVD devices that may, at times, give false positive or false negatives? Yes. In fact, most do, to a certain extent (if you read the device labeling, it is extremely rare to find an IVD device that has perfect performance). Understanding the limitations of a device will help you to minimize the occurrence of false positive or false negative results and subsequently improve patient care. Typically, the limitations highlight key results from performance studies, or indicate where information on performance is lacking. For instance, a serology assay that has been shown to cross react with rheumatoid arthritis antibodies will likely have a limitation that the device should not be used in arthritic patients and consequently is probably not a good choice for a laboratory with a large proportion of specimens from an elderly population. Certain limitations may also arise from specific technological characteristics of a device. For instance, a device that has a qualitative output may have a limitation stating that the numeric value reported does not correlate with analyte levels present in the sample. The limitations section may indicate specific patient subpopulations for which device performance has not been established, or has been determined to be unsatisfactory, at a greater level of detail than can be found in the intended use. The limitations may also indicate variations on the targeted analyte that may render the analyte undetectable, such as a particular strain or mutant of an infectious organism. A common type of limitation is for patients with a disease or condition that affects (or may affect) the performance of the device, such as the arthritis example above. In cases where it is unlikely that the laboratory will be aware of such a disease or condition in patients, it may be beneficial for the laboratory to report the test limitation to clinicians along with each patient result.
Warnings and Precautions
Here is where you can determine the type of information and training needed to ensure that the test can be conducted safely in your laboratory. Check to ensure that your laboratory and personnel are equipped to handle the hazardous materials required to run the IVD device.
The detailed procedure in the device labeling gives key information about how to perform the test correctly. Be wary of abbreviated protocols when an IVD test is first being used; it is important to check to ensure consistency between the procedure in the device labeling and any abbreviated protocols that might be provided by the company. It is important to monitor equipment to ensure that the test is performed within the time and temperature ranges recommended in the procedure – control materials alone may not ensure that the assay was run properly.
Not all quality control materials are created equal. You should be able to extrapolate from the quality control section the information needed to determine if the quality control (QC) materials provided or sold for use with the IVD device are suitable to satisfy your laboratory’s local, state and federal requirements for QC materials. In many cases the controls provided are only intended to monitor for substantial reagent failure, and are not intended to monitor precision near the assay cutoff. This is why it is important to check control information, including analyte levels and sample matrix. Look to see if additional controls are recommended for you to supply yourself, or purchase from another vendor. Also check this section for QC procedures, frequency, and recommended corrective actions when controls fail. If recommendations sufficient for your laboratory are not found in the labeling, you will need to establish them according to your own laboratory policies and procedures.
Interpretation of Results
This section of the labeling will give recommendations on how test results should be interpreted and reported to physicians. Consider the following when reviewing the Interpretation of Results:
The Interpretation of Results section may include information on an equivocal zone.
Equivocal zones are put in place for many assays, due to less than perfect precision near the medical decision point(s). An additional factor that may come into play when adding an equivocal zone to an assay are the number of clinical specimens that fall close to the medical decision point(s), within the range of imprecision. Recommendations for retesting samples with equivocal results are typically made when such retesting improves the overall performance of the assay. Recommendations for testing a freshly collected specimen as a follow-up to an equivocal result are typically made when time is expected to affect levels of the measured analyte (as in serology assays), and such recommendations should be reported to physicians along with the equivocal assay result.
Additional Diagnostic Procedures
The Interpretation of Results section may include recommendations for additional testing based upon the initial test results, for example, confirmation of positive or negative results by additional diagnostic procedures. Recommendations for additional testing are typically put in place due to low clinical sensitivity and/or specificity, and as such should not be ignored, unless it is determined that a false positive or false negative result would have a negligible impact on the clinical management of the patient.
FDA’s Office of In Vitro Diagnostic Device Evaluation and Safety (OIVD) strongly encourages feedback from a laboratory (or physician) if they become aware of any general confusion on the part of physicians regarding the significance of IVD test results (as reported per the recommendations in the Interpretation of Results section of the labeling). This could increasingly become an issue as more IVD devices are developed that measure multiple analytes simultaneously. For example, the significance of multiple human genetic mutations may be difficult to summarize in the product labeling, such that a physician might have trouble interpreting the reported results of a multiplexed genetic test. Since physicians often contact laboratories first with these types of questions, the laboratory may wish to consider ensuring that its personnel are knowledgeable about the labeling recommendations on how to interpret test results. Also, the laboratory is in a good position to notify FDA of any recurrent difficulties in interpretation of assay results and should feel free to ask FDA/OIVD any questions that are difficult to answer. OIVD contact information and other general information can be found on our website 2
If a test has a relatively high invalid rate, this would likely be indicated in the labeling, typically in the interpretation of results, or in the limitations.
When trying to decide between different IVD devices for your laboratory, you will probably consider cost and convenience, but ideally, performance characteristics will weigh heavily in your decision. The following are some tips for interpreting performance characteristics as reported in IVD device labeling.
Sensitivity and Specificity vs. Percent Agreement
The performance of IVD devices are often measured against imperfect standards or comparators. Performance is reported as clinical “sensitivity” and “specificity” only in cases where (among other considerations) the comparator is considered a “gold-standard,” which can be defined as the best available method for determining the true clinical status of the patient. It is important to note that even a gold-standard is likely to be imperfect and can change with the development of new technologies. A gold standard is clearly the preferred method for reporting estimates of diagnostic performance, but in many cases this is not available as a comparator. In these cases, performance is reported against a non gold-standard comparator as “positive percent agreement” and “negative percent agreement” in lieu of “sensitivity” and “specificity” 3 . The problem with performance estimates reported as positive percent agreement and negative percent agreement is that the true clinical status of each patient is unknown. Take an example where a positive test result indicates the patient is positive for antibody to the hepatitis A virus. In such a case, if a new hepatitis A antibody IVD device reports 95% positive percent agreement to a comparator, the new device can not be interpreted as detecting 95% of all individuals with hepatitis A antibody. Why? The comparator assay is imperfect. The comparator may, for example, miss 7% of hepatitis A antibody positive individuals. Therefore, the new assay’s clinical sensitivity is essentially unknown because all the hepatitis A antibody positive individuals have not been correctly identified in this evaluation. This is why performance in this case is reported as positive percent agreement and not sensitivity.
Take note of the confidence intervals reported along with estimates of clinical sensitivity, specificity and percent agreement for an IVD device. Not all point estimates are created equal. Depending on the size of the study population and prevalence of the disease or condition of interest, the performance estimates reported can vary significantly in their level of confidence. Simply stated, if a test identifies 10 out of 10 true positive results correctly, your level of confidence that the test is 100% sensitive is much lower than if the test identified 100 out of 100 true positive results correctly. The true sensitivity may be as low as 72.3% (10/10) vs. 96.3% (100/100).
Be wary of discordant resolution
For specimens where a device and its comparator are discordant (e.g., one result is positive and the other is negative), the results of a third comparator method may be footnoted in performance tables. However, performance expectations are never changed based on the results of discordant resolution, because this type of analysis by default will always enhance performance estimates. Since only discordant specimens are undergoing additional testing, there is nothing to lose and everything to gain in estimates of performance. If a third comparator is performed on every specimen in the dataset, and if the performance characteristics of the third comparator are known, then adding a third comparator is truly informative.
Choosing the best IVD device to meet the needs of both patients and physicians is only possible if you read and carefully analyze the labeling of each test that your laboratory offers. Most importantly, good communication between the clinical laboratory and physicians regarding the appropriate use of tests offered by the laboratory can help make significant advances in the quality and cost-effectiveness of patient care. Periodically communicating with your ordering physicians regarding the importance of limiting tests ordered to the appropriate patient populations is one critical way of improving patient care. Relating key information, such as limitations, from IVD package inserts to ordering physicians is another. If you have not already done so, it may be beneficial to consider putting a program together for your laboratory that highlights key information for each test that you offer. Without it, the tests offered by your laboratory could easily be misused and misinterpreted. If you notice that such information is not being utilized by your customers, a periodic reminder of the availability of the information would be prudent.
1 False positive result is a positive test result for a subject in whom the target condition is absent (by clinical “gold-standard”).
False negative result is a negative result for a subject in whom the target condition is present.
True positive result is a positive result for a subject in whom the target condition is present.
True negative result is a negative test result for a subject in whom the target condition is absent.
Please see CLSI document EP12-A2 “User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline-Second Edition”, 2008 for more details
3For FDA’s detailed perspective on reporting diagnostic test results, see the FDA guidance document entitled “Guidance for Industry and FDA Staff: Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests” found at: http://www.fda.gov/cdrh/osb/guidance/1620.pdf
Revelations in In Vitro Diagnostic (IVD) Device Labeling – Proactive Prevention of Adverse Events by Informed Decision Making. By: Simon, K; Kondratovich, M. FDA Office of In Vitro Diagnostics.