Interobserver Variability of Radiographic Pulmonary Nodule Diameter Measurements in Dogs and Cats

Introduction

When encountered in clinical practice, incidental discovery of pulmonary nodules on thoracic radiographs presents a diagnostic challenge. Currently, the exact incidence of pulmonary nodule detection on routine radiographic studies in veterinary medicine is unknown. In a human study that evaluated 5,726 thoracic radiographs over 1 yr, 116 new pulmonary nodules (2.2%) were detected.¹ Of those, 51 nodules were benign, 19 were erroneously interpreted, and 22 lacked a final diagnosis. The remaining nodules were malignant. Those numbers suggest that benign pulmonary nodules are more common than malignant nodules, specifically in humans. In cases of either nonspecific or incidental pulmonary nodules, where benign versus malignant disease is the concern, repeat radiographic assessment can be used to monitor growth. An increase in nodule size and number suggests malignancy.^2–5 Correct assessment of the growth of pulmonary nodules is also important for monitoring response to chemotherapeutic treatment in cases of known neoplasia.

Pulmonary nodules are roughly spherical, and nodule size is measured using a two-dimensional radiographic image. The rate of growth of a nodule (i.e., the change in volume) is estimated based on measurement of the diameter. The diameter measurement is halved and cubed to calculate volume (4/3πr³, where r refers to the radius), which compounds potential measurement error. For example, a 1 mm increase in the diameter of a 5 mm nodule represents a 42% increase in volume. In humans, a 30% increase in diameter would equate to volume doubling, and a tumor doubling time of 4 wk is suggestive of malignancy.² Based on those calculations, only small changes in the diameter of small pulmonary nodules are required to quickly achieve volume doubling. Thus, it is possible that even small variability in diameter measurement as a result of either inter- or intraobserver effects could have a significant impact on determinations of malignancy of either a nonspecific pulmonary nodule or tumor response, particularly when considering nodule volume and the criterion of tumor volume doubling.^6,7

Radiologic features of pulmonary nodules that help identify a likelihood of malignancy in human medicine include size, morphology, and rate of growth.² Measurement of nodule growth using maximum linear lesion diameter on sequential radiographic studies is a common diagnostic test, particularly as those measurements are easily performed.^2,3,8 The method of measuring maximum cross-sectional lesion diameter is used as part of the Response Evaluation Criteria in Solid Tumors (RECIST) for standardizing independent image review in human oncologic studies.⁹ RECIST was designed to establish a set of criteria to evaluate the progression, stabilization, and response to treatment of solid tumors and utilizes sequential measurements of lesions obtained via radiography, computed tomography (CT), or MRI. The Veterinary Cooperative Oncology Group has adopted and revised RECIST guidelines for veterinary medicine.¹⁰ Based on RECIST guidelines, using the example of a 1 mm increase of a 5 mm diameter nodule, the 20% diameter increase suggests progression of disease.

In a study that evaluated the reasons for variability in response rate accuracy among clinicians in human oncology trials, the major cause of errors was variation in tumor measurement.⁷ Errors were attributed to the clinician’s difficulty identifying tumor margins and incorrect selection of the maximum tumor diameter. The variability among clinicians in tumor measurement in humans has been reported using radiographic assessment. Ten observers, including two radiologists, measured the maximum diameter of metastatic pulmonary nodules on 210 human chest radiographs, and an interobserver variability of 25% was reported.⁸ The lowest interobserver variability in tumor measurements obtained in a study of 19 radiographs from 12 patients diagnosed with nonsmall cell lung cancer was 10%.¹¹ Twenty-five experienced readers participated in the study. Due to the large variation in measurements (and ultimately response rate), a recommendation was made that all radiographs be reviewed by a single observer for any particular study.

Although the accuracy of radiographic techniques for detection of pulmonary nodules in companion animals has been studied, neither the inter- nor intraobserver variability of individual lesion measurements on survey radiography has been evaluated in veterinary medicine.^12–17 In clinical practice, inter- or intraobserver variability can lead to misclassification of tumor response under the RECIST criteria and can suggest malignancy in an otherwise benign nodule by producing measurement error.

The purpose of the current study was to evaluate the interobserver variability of radiographic pulmonary nodule measurements among multiple readers with varying levels of experience, as is typically found in general practice. The hypothesis was that statistically significant interobserver variability would occur in measured nodule diameters using thoracic radiography among a group of readers, contributing to the inaccurate estimation of nodule growth. An additional hypothesis was that interobserver variability in diameter measurement would exist based on experience level. Finally, a third hypothesis was that less interobserver variability would exist when measuring larger nodules due to improved conspicuity within the thorax.

Materials and Methods

Image Selection

Thoracic radiographs were chosen based on a keyword search of radiology reports with a conclusion of the presence of at least one solid soft-tissue pulmonary nodule. All patients were radiographed between April 2007 and July 2009 at Affiliated Veterinary Specialists, Maitland, Florida, and three-view radiographs consisting of a right lateral, left lateral, and ventrodorsal projection were available. One dog was imaged via a dorsoventral projection rather than a ventrodorsal view. Radiographs were obtained with either a digital radiography^a or computed radiography^b system.

Nodules were arbitrarily chosen by one of the study authors (J.W.) and marked by a circular region of interest drawn and saved as data appended to the original file (Figure 1). Some radiographs had other abnormalities that did not alter lesion visibility.

View Figure

• Download as Powerpoint
• Download Figure

Results

Twenty-nine sets of radiographs were selected representing 22 dogs and 7 cats. Canine patients ranged in age from 6 yr to 17 yr (mean, 10.5 yr). Mean feline patient age was 13.3 yr (range, 10–17 yr). A total of 47 nodules were identified for measurement. The number of nodules per patient varied from 1 nodule to 6 nodules (mean, 1.6 nodules), and a total of 376 nodule diameter measurements were recorded. Recorded nodule sizes ranged from 3.6 mm to 10.1 mm (mean, 6.4 mm). The individual contributions of reader, experience level, patient, and nodule to the overall variation in mean nodule diameter were evaluated, and a statistically significant variation in mean nodule diameter measurement based on the reader (interobserver variability) was identified (P < .0001). Thus, the hypothesis that significant interobserver variability would occur in measured nodule diameters among a group of readers was accepted. There was no statistically significant difference in mean nodule diameter among experience levels (P = .721). The hypothesis that interobserver variability would exist based on experience level was not verified. There was also no effect of the patient evaluated on the variability in mean nodule diameter (P = 1).

There was no significant contribution to the overall variability in mean nodule diameter based on size classification (P = .364). Thus, the hypothesis that lower interobserver variability would occur in the mean diameter of large nodules was not verified. Nine nodules were ≤ 5 mm (small), 26 nodules were 5.1–7.5 mm (medium), and 12 nodules were ≥ 7.6 mm (large). The estimated means and standard deviations of measured diameter for small, medium, and large nodules among all eight readers was 4.71 mm ± .022 mm, 6.10 mm ± .11 mm, and 8.32 mm ± .11 mm, respectively.

The estimated mean, standard deviation, and variation coefficient for 47 nodule diameters calculated for each experience level are shown in Table 1. A box-and-whisker plot indicating the median and range of pulmonary nodule diameters measured by each reader is shown in Figure 2. The interobserver variability in nodule diameter measurement for any given nodule was calculated as 16.2%.

TABLE 1 Mean, Standard Deviation, and Interobserver Variability of 47 Pulmonary Nodule Diameters Based on Level of Reader Experience

*

Interobserver variability within each group is calculated as the ratio of the standard deviation to the mean.

View Fullscreen

View Figure

• Download as Powerpoint
• Download Figure

Discussion

The interobserver variability in pulmonary nodule diameter measurements was statistically different among eight readers. Rate of growth has proven a more useful prognostic sign than lesion size, and tumor doubling time is a criterion in veterinary medicine to characterize tumor biologic behavior.^3,5 For pulmonary nodules, the method relies on radiographic measurements of a nodule at two points in time. A consistent quantification of nodule diameter is essential during follow-up radiographic assessments of either nonspecific pulmonary nodules or in monitoring response to therapy. According to RECIST criteria, complete response when evaluating tumor treatment is defined as a reduction in short-axis diameter of the nodule to < 10 mm (for nodules with a minimum original diameter of 20 mm).¹⁰ Partial response represents a 30% decrease in the diameters of target lesions. Classification of progressive disease requires a 20% increase in target lesion diameter as well as an absolute increase in the sum of target lesion diameters by 5 mm. The overall interobserver variability for any given nodule diameter measurement in this study was calculated as 16%. Although statistically significant, in the context of RECIST criteria, this finding is not clinically significant as the variability is below that which is required to classify progressive disease. In addition, based on the human derived criterion of a 30% diameter increase equating to tumor volume doubling, a 16% variability in diameter measurement is low enough to eliminate the possibility of misdiagnosis of malignancy based on nodule growth.²

Many factors likely contributed to the calculated variability, both behavioral and technical. Some readers recorded their measurements toward the end of the workday, when fatigue could contribute to diameter measurement variability. Perception of the largest diameter of each nodule is reader-dependent and difficult to standardize. The presence of concurrent lung pathology and the location of the nodules may have affected reader measurements, particularly due to superimposition over thoracic structures, such as the cardiac silhouette or diaphragm. The amount of variation may also depend on the shape and edge definition of individual nodules.

Experience level had no effect on nodule diameter measurement in this study, and the percent variability calculated for each experience level was relatively low (1–6% for each group of two readers compared with 16% for all eight readers). A recent study determined that board-certified radiologists had greater detection accuracy of pulmonary nodules compared with general practitioners.¹⁷ The results of the study reported herein indicate that once a pulmonary nodule is detected, the reader’s experience level is no longer a source of potential uncertainty in diameter measurement.

An unexpected result of the current study is the lack of statistically significant interobserver variability calculated for pulmonary nodules based on their size classification. The authors’ hypothesis that variability would be lowest for the largest nodules due to increased conspicuity was not verified. In developing this hypothesis, the authors considered that readers would be able to identify the edge of a larger lesion with more confidence compared with small nodules. In an effort to eliminate that potential contribution, radiographs containing pulmonary nodules of varying sizes were provided to the readers in random order so that all of the large nodules were not measured first or last at each session.

Despite RECIST guidelines that advocate the use of the longest diameter in the plane of measurement, cross-sectional diameter measurements obtained using either rulers or calipers have shown high interobserver variability. In a group of oncologists measuring simulated solid tumors (1.8–14.5 cm in diameter), an interobserver variability of 17% was reported.⁶ Linear measurements of diameter have been shown to deviate from the true diameter by an average of 12%, and overestimations for pulmonary nodule size range from 26% to 54%.¹⁸ A major argument against the use of one- or two-dimensional measurements is the fact that nodule growth is a three-dimensional phenomenon. Some malignant nodules exhibit asymmetric growth that cannot be adequately assessed using two-dimensional radiographs. Another consideration when using linear caliper measurements is the measurement bias that can occur due to approximation of lesion dimension to even integers. Despite the potential for measurement error using radiographic cross-sectional diameters, other approaches used to characterize pulmonary nodules in human medicine, including CT imaging and CT-guided percutaneous biopsy, may be either unavailable, invasive (in the case of biopsy), or cost prohibitive for companion animals. The current trend in human medicine is to follow nodule growth using CT because CT is superior to radiography for both detection of pulmonary nodules and response evaluation for chemotherapy treatment.^19–23 Unfortunately, CT is currently only available in referral veterinary hospitals.

One of the best predictors of the likelihood of nodule malignancy is the demonstration of growth, which can be quickly and noninvasively demonstrated with repeat radiographic assessment and linear diameter measurements.^3,24,25 For spherical masses, a 30% change in the diameter of a nonspecific pulmonary nodule corresponds to a doubling of the overall volume.² Because CT is not widely available, cross-sectional diameter measurements on radiographs (either hard copy film or digital) remain the most widely used test in veterinary medicine. Veterinary studies evaluating pulmonary nodule diameter and volume using CT are warranted based on the proven reliability of those methods in human medicine. In the interim, due to low cost, availability, and the veterinary clinician’s comfort level with interpretation, radiographs remain the most common diagnostic tool for monitoring pulmonary nodules. Evidence or absence of growth influences treatment decisions and a presumptive diagnosis of malignancy. RECIST criteria in veterinary oncology rely on lesion diameters to determine tumor regression (nodule diameter reduction to < 10 mm) and clinical response (30% reduction in nodule diameter).¹⁰ Ideally, the same clinician should perform follow-up radiographic measurements to eliminate interobserver variability, but this practice cannot be guaranteed in a multidoctor hospital.¹¹ The lack of contribution of experience level to interobserver variability in this study indicates that serial radiographic assessment of pulmonary nodule diameters is a valid diagnostic tool in both general and specialty practice.

Limitations of the current study include the fact that nodule diameter measurements were obtained from radiographs produced by both digital radiography and computed radiography systems. The contribution of the postprocessing algorithms to measurement variability was not evaluated and may be clinically significant. A determination of the amount of change in diameter that would be required to state with 95% confidence that a nodule has increased in size based on a two-dimensional radiograph is warranted. Another limitation is the lack of repeat measurements of the same nodules by the same individuals to determine the contribution of intraobserver variability. Intraobserver variability affects interobserver variability and is expected to be lower than the interobserver component.^8,11

Conclusion

The results of this study indicate that statistically significant interobserver variability in radiographic pulmonary nodule diameter measurement exists among any group of clinicians, but the magnitude of the variability is likely not clinically significant. Misdiagnoses of either nodule growth or subsequent malignancy or misclassification of tumor response to treatment are unlikely based on 16% interobserver variability when using multiple criteria to evaluate nodule growth. The experience level of the reader and the size of the nodule measured (≤ 5 mm, 5.1–7.5 mm, and ≥ 7.6 mm) did not contribute to the interobserver variability in this study. Based on the 16% interobserver variability reported in this study, the recommendation that the same clinician performs serial assessment of radiographic pulmonary nodules may no longer be valid. This is a significant finding, particularly as it is difficult to meet such a requirement in a busy general or specialty veterinary practice.