OpenLB
Open Library of Bioscience
Advanced Search
Radiology
09, 2022;304 (3): 683-691.

Artificial Intelligence Tool for Assessment of Indeterminate Pulmonary Nodules Detected with CT.

Roger Y Kim, Jason L Oke, Lyndsey C Pickup, Reginald F Munden, Travis L Dotson, Christina R Bellinger, Avi Cohen, Michael J Simoff, Pierre P Massion, Claire Filippini, Fergus V Gleeson, Anil Vachani
Author Information
  1. Roger Y Kim: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.). ORCID
  2. Jason L Oke: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.).
  3. Lyndsey C Pickup: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.).
  4. Reginald F Munden: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.).
  5. Travis L Dotson: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.).
  6. Christina R Bellinger: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.).
  7. Avi Cohen: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.).
  8. Michael J Simoff: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.). ORCID
  9. Pierre P Massion: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.).
  10. Claire Filippini: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.). ORCID
  11. Fergus V Gleeson: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.). ORCID
  12. Anil Vachani: From the Division of Pulmonary, Allergy, and Critical Care, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Suite 216, Stemmler Hall, 3450 Hamilton Walk, Philadelphia, PA 19104 (R.Y.K., A.V.); Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom (J.L.O.); Optellum, Oxford, United Kingdom (L.C.P.); Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC (R.F.M.); Department of Pulmonary, Critical Care, Allergy and Immunologic Diseases, Wake Forest School of Medicine, Winston-Salem, NC (T.L.D., C.R.B.); Division of Pulmonary and Critical Care Medicine, Department of Medicine, Henry Ford Health System, Detroit, Mich (A.C., M.J.S.); Division of Allergy, Pulmonary and Critical Care Medicine, Vanderbilt Ingram Cancer Center, Nashville, Tenn (P.P.M.); and Department of Oncology, Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom (C.F., F.V.G.). ORCID
PMID: 35608444 DOI: 10.1148/radiol.212182

Abstract

Background Limited data are available regarding whether computer-aided diagnosis (CAD) improves assessment of malignancy risk in indeterminate Pulmonary Nodules (IPNs). Purpose To evaluate the effect of an artificial intelligence-based CAD tool on clinician IPN diagnostic performance and agreement for both malignancy risk categories and management recommendations. Materials and Methods This was a retrospective multireader multicase study performed in June and July 2020 on chest CT studies of IPNs. Readers used only CT imaging data and provided an estimate of malignancy risk and a management recommendation for each case without and with CAD. The effect of CAD on average reader diagnostic performance was assessed using the Obuchowski-Rockette and Dorfman-Berbaum-Metz method to calculate estimates of area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. Multirater Fleiss κ statistics were used to measure interobserver agreement for malignancy risk and management recommendations. Results A total of 300 chest CT scans of IPNs with maximal diameters of 5-30 mm (50.0% malignant) were reviewed by 12 readers (six radiologists, six pulmonologists) (patient median age, 65 years; IQR, 59-71 years; 164 [55%] men). Readers' average AUC improved from 0.82 to 0.89 with CAD ( < .001). At malignancy risk thresholds of 5% and 65%, use of CAD improved average sensitivity from 94.1% to 97.9% ( = .01) and from 52.6% to 63.1% ( < .001), respectively. Average reader specificity improved from 37.4% to 42.3% ( = .03) and from 87.3% to 89.9% ( = .05), respectively. Reader interobserver agreement improved with CAD for both the less than 5% (Fleiss κ, 0.50 vs 0.71; < .001) and more than 65% (Fleiss κ, 0.54 vs 0.71; < .001) malignancy risk categories. Overall reader interobserver agreement for management recommendation categories (no action, CT surveillance, diagnostic procedure) also improved with CAD (Fleiss κ, 0.44 vs 0.52; = .001). Conclusion Use of computer-aided diagnosis improved estimation of indeterminate pulmonary nodule malignancy risk on chest CT scans and improved interobserver agreement for both risk stratification and management recommendations. © RSNA, 2022 . See also the editorial by Yanagawa in this issue.

References

  1. N Engl J Med. 2011 Aug 4;365(5):395-409 [PMID: 21714641]
  2. J Thorac Dis. 2020 Jun;12(6):3317-3330 [PMID: 32642255]
  3. Chest. 2013 May;143(5 Suppl):e93S-e120S [PMID: 23649456]
  4. Ann Am Thorac Soc. 2018 Oct;15(10):1117-1126 [PMID: 30272500]
  5. Arch Intern Med. 1997 Apr 28;157(8):849-55 [PMID: 9129544]
  6. Chest. 2022 Feb;161(2):562-571 [PMID: 34364866]
  7. N Engl J Med. 2020 Feb 6;382(6):503-513 [PMID: 31995683]
  8. Ann Am Thorac Soc. 2013 Dec;10(6):629-35 [PMID: 24063427]
  9. J Thorac Dis. 2020 Jun;12(6):3296-3302 [PMID: 32642253]
  10. Chest. 2015 Dec;148(6):1405-1414 [PMID: 26087071]
  11. Chest. 2019 Jul;156(1):112-119 [PMID: 30981723]
  12. Curr Opin Pulm Med. 2021 Jul 1;27(4):240-248 [PMID: 33973553]
  13. N Engl J Med. 2013 Sep 5;369(10):910-9 [PMID: 24004118]
  14. JAMA. 1996 Dec 4;276(21):1752-5 [PMID: 8940325]
  15. JAMA. 2021 Mar 09;325(10):962-970 [PMID: 33687470]
  16. Chest. 2017 Aug;152(2):263-270 [PMID: 28115167]
  17. Clin Radiol. 2020 Jan;75(1):13-19 [PMID: 31202567]
  18. JAMA. 2022 Jan 18;327(3):264-273 [PMID: 35040882]
  19. Am J Respir Crit Care Med. 2012 Feb 15;185(4):363-72 [PMID: 21980032]
  20. Sci Rep. 2017 Apr 19;7:46479 [PMID: 28422152]
  21. Thorax. 2020 Apr;75(4):306-312 [PMID: 32139611]
  22. Clin Lung Cancer. 2017 Jan;18(1):e27-e34 [PMID: 27530054]
  23. Acta Radiol. 2015 Oct;56(10):1180-6 [PMID: 25293951]
  24. Acad Radiol. 2008 May;15(5):647-61 [PMID: 18423323]
  25. JAMA Intern Med. 2014 Jun;174(6):871-80 [PMID: 24710850]
  26. Radiology. 2018 Jan;286(1):286-295 [PMID: 28872442]
  27. Proc SPIE Int Soc Opt Eng. 2020 Feb;11316: [PMID: 32351258]
  28. JAMA Intern Med. 2021 Apr 1;181(4):480-489 [PMID: 33464296]
  29. Radiology. 2015 Dec;277(3):863-71 [PMID: 26020438]
  30. Am J Respir Crit Care Med. 2015 Nov 15;192(10):1208-14 [PMID: 26214244]
  31. Acad Radiol. 2010 Mar;17(3):323-32 [PMID: 20152726]
  32. Am J Respir Crit Care Med. 2020 Jul 15;202(2):241-249 [PMID: 32326730]
  33. Biometrics. 1977 Mar;33(1):159-74 [PMID: 843571]

Grants

  1. P30 ES013508/NIEHS NIH HHS
  2. T32 HL007891/NHLBI NIH HHS

MeSH Term

Aged
Artificial Intelligence
Female
Humans
Lung Neoplasms
Male
Multiple Pulmonary Nodules
Retrospective Studies
Sensitivity and Specificity
Tomography, X-Ray Computed
Journal Article Research Support, Non-U.S. Gov't Research Support, N.I.H., Extramural
Article has an altmetric score of 410

Word Cloud

Created with Highcharts 10.0.0CADrisk0malignancyimprovedCTagreementmanagement001Fleissκinterobserver<=IPNsdiagnosticcategoriesrecommendationschestaveragereadervsdatacomputer-aideddiagnosisindeterminatepulmonaryeffectperformanceusedrecommendationAUCsensitivityspecificityscans50sixyears895%65%1%9%52respectively3%71alsoBackgroundLimitedavailableregardingwhetherimprovesassessmentnodulesPurposeevaluateartificialintelligence-basedtoolclinicianIPNMaterialsMethodsretrospectivemultireadermulticasestudyperformedJuneJuly2020studiesReadersimagingprovidedestimatecasewithoutassessedusingObuchowski-RocketteDorfman-Berbaum-MetzmethodcalculateestimatesareareceiveroperatingcharacteristiccurveMultiraterstatisticsmeasureResultstotal300maximaldiameters5-30mm0%malignantreviewed12readersradiologistspulmonologistspatientmedianage65IQR59-71164[55%]menReaders'82thresholdsuse9497016%63Average374%42038705Readerless54Overallactionsurveillanceprocedure44ConclusionUseestimationnodulestratification©RSNA2022SeeeditorialYanagawaissueArtificialIntelligenceToolAssessmentIndeterminatePulmonaryNodulesDetected

Similar Articles

Cited By (25)