Tissue contamination challenges the credibility of machine learning models in real world digital pathology.

Ismail Irmakci, Ramin Nateghi, Rujoi Zhou, Ashley E Ross, Ximing J Yang, Lee A D Cooper, Jeffery A Goldstein
Author Information

Abstract

Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. While human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole slide models. Three operate in placenta for 1) detection of decidual arteriopathy (DA), 2) estimation of gestational age (GA), and 3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in T-distributed Stochastic Neighbor Embedding (tSNE) feature space. Every model showed performance degradation in response to one or more tissue contaminants. DA detection balanced accuracy decreased from 0.74 to 0.69 +/- 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant raised the mean absolute error in estimating gestation age from 1.626 weeks to 2.371 +/ 0.003 weeks. Blood, incorporated into placental sections, induced false negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033mm, resulted in a 97% false positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.

References

  1. Am J Clin Pathol. 2022 Jul 1;158(1):4-7 [PMID: 35229867]
  2. Urol Clin North Am. 2021 Feb;48(1):25-33 [PMID: 33218591]
  3. Nat Med. 2019 Aug;25(8):1301-1309 [PMID: 31308507]
  4. Pediatr Dev Pathol. 2002 Mar-Apr;5(2):159-64 [PMID: 11910510]
  5. Ann Diagn Pathol. 2020 Aug;47:151561 [PMID: 32623312]
  6. Am J Reprod Immunol. 2013 Oct;70(4):285-98 [PMID: 23905710]
  7. Placenta. 2019 Jan 15;76:1-5 [PMID: 30803708]
  8. Acta Obstet Gynecol Scand. 2015 Sep;94(9):976-82 [PMID: 26054014]
  9. Arch Pathol Lab Med. 1996 Nov;120(11):1009-14 [PMID: 12049100]
  10. Science. 2019 Mar 22;363(6433):1287-1289 [PMID: 30898923]
  11. Mod Pathol. 2021 Dec;34(12):2098-2108 [PMID: 34168282]
  12. Nat Biomed Eng. 2021 Jun;5(6):555-570 [PMID: 33649564]
  13. Arch Pathol Lab Med. 2021 Mar 1;145(3):359-364 [PMID: 32886759]
  14. J Grad Med Educ. 2014 Mar;6(1 Suppl 1):180-1 [PMID: 24701281]
  15. Am J Obstet Gynecol. 2006 Dec;195(6):1674-9 [PMID: 16796983]
  16. Placenta. 2017 May;53:113-118 [PMID: 28487014]
  17. Int J Med Inform. 2022 Sep;165:104828 [PMID: 35780651]
  18. Am J Clin Pathol. 2022 Jul 1;158(1):96-104 [PMID: 35195717]
  19. Placenta. 2017 Oct;58:52-59 [PMID: 28962696]
  20. Comput Med Imaging Graph. 2024 Mar;112:102337 [PMID: 38228020]
  21. Pediatr Dev Pathol. 2016 Nov/Dec;19(6):502-505 [PMID: 26669929]
  22. Pediatr Dev Pathol. 2021 Jan-Feb;24(1):10-11 [PMID: 33023403]
  23. Med Image Anal. 2022 Jul;79:102474 [PMID: 35588568]
  24. Am J Clin Pathol. 2011 Nov;136(5):767-72 [PMID: 22031316]
  25. Placenta. 2016 Jul;43:61-8 [PMID: 27324101]
  26. J Pathol Inform. 2021 Dec 24;12:54 [PMID: 35070483]
  27. Surg Pathol Clin. 2013 Mar;6(1):101-14 [PMID: 26838705]
  28. Arch Pathol Lab Med. 2002 Jun;126(6):706-9 [PMID: 12033960]
  29. J Perinat Med. 2018 Aug 28;46(6):613-630 [PMID: 30044764]
  30. Acta Obstet Gynecol Scand. 2011 Jan;90(1):19-25 [PMID: 21275911]
  31. Br J Cancer. 2021 Feb;124(4):686-696 [PMID: 33204028]
  32. Int J Surg Pathol. 2023 Jun;31(4):387-397 [PMID: 35645148]
  33. Am J Obstet Gynecol. 2022 Jan;226(1):68-89.e3 [PMID: 34302772]
  34. Nature. 2021 Jun;594(7861):106-110 [PMID: 33953404]
  35. Nat Med. 2022 Mar;28(3):575-582 [PMID: 35314822]
  36. Am J Surg Pathol. 2012 Jan;36(1):e1-5 [PMID: 22173121]
  37. Am J Obstet Gynecol. 2011 Aug;205(2):124.e1-7 [PMID: 21722872]
  38. Lab Invest. 2021 Jul;101(7):942-951 [PMID: 33674784]
  39. World J Surg Oncol. 2019 Feb 13;17(1):31 [PMID: 30760274]
  40. Mod Pathol. 2022 Jan;35(1):23-32 [PMID: 34611303]
  41. Diagn Pathol. 2019 Dec 27;14(1):138 [PMID: 31881972]
  42. Pediatr Dev Pathol. 2004 Jan-Feb;7(1):26-34 [PMID: 15255032]
  43. Am J Pathol. 2020 Oct;190(10):2111-2122 [PMID: 32679230]
  44. Am J Clin Pathol. 2020 Jun 8;154(1):23-32 [PMID: 32441303]
  45. Arch Pathol Lab Med. 2019 Jul;143(7):859-868 [PMID: 30295070]
  46. Mod Pathol. 2020 Oct;33(10):2058-2066 [PMID: 32393768]
  47. J Med Imaging (Bellingham). 2019 Apr;6(2):027501 [PMID: 31037247]
  48. PLoS One. 2014 Feb 25;9(2):e89419 [PMID: 24586764]
  49. Cases J. 2009 Sep 09;2:7619 [PMID: 20181194]
  50. J Clin Pathol. 2008 Dec;61(12):1254-60 [PMID: 18641412]
  51. Arch Pathol Lab Med. 2022 Mar 1;146(3):372-378 [PMID: 34252177]
  52. JCO Clin Cancer Inform. 2020 Mar;4:221-233 [PMID: 32155093]
  53. J Perinat Med. 2020 Jun 25;48(5):516-518 [PMID: 32396141]
  54. Placenta. 2023 Apr;135:43-50 [PMID: 36958179]
  55. Sci Rep. 2021 Apr 19;11(1):8454 [PMID: 33875703]
  56. Arch Pathol Lab Med. 2016 Jul;140(7):698-713 [PMID: 27223167]
  57. IEEE J Biomed Health Inform. 2021 Feb;25(2):307-314 [PMID: 33347418]
  58. Biochim Biophys Acta Rev Cancer. 2021 Jan;1875(1):188452 [PMID: 33065195]
  59. Comput Med Imaging Graph. 2021 Mar;88:101820 [PMID: 33453648]
  60. Arch Pathol Lab Med. 2013 Dec;137(12):1723-32 [PMID: 23738764]

Grants

  1. K08 EB030120/NIBIB NIH HHS
  2. R01 LM013523/NLM NIH HHS
  3. U01 CA220401/NCI NIH HHS
  4. UL1 TR001422/NCATS NIH HHS

Word Cloud

Created with Highcharts 10.0.0tissuecontaminantsmodels0attentionpatchesMLslidesTissue1modelprostateneedlebiopsiescontaminantaddedfalselearningpathologyusewholetraineddetectexaminedimpactplacentadetectionDA2ageplacentalcancerpatientmeasuredperformancegivenpatchweeksinducedrateMachinepoisedtransformsurgicalpracticesuccessfulmechanismsexamineidentifyareasdiagnosticguidediagnosisfloatersrepresentunexpectedhumanpathologistsextensivelyconsider4slideThreeoperatedecidualarteriopathyestimationgestationalGA3classificationmacroscopiclesionsalsodevelopeddesignedexperimentswhereinrandomlysampledknowndigitallyproportionT-distributedStochasticNeighborEmbeddingtSNEfeaturespaceEveryshoweddegradationresponseonebalancedaccuracydecreased7469+/-01additionevery1001%Bladder10%raisedmeanabsoluteerrorestimatinggestation626371+/003BloodincorporatedsectionsnegativediagnosesintervillousthrombiAdditionbladderpositivesselectionhigh-attentionrepresenting033mmresulted97%positiveContaminantreceivedaverageinduceerrorsmodernhighlevelindicatesfailureencodebiologicalphenomenaPractitionersmovequantifyameliorateproblemcontaminationchallengescredibilitymachinerealworlddigital

Similar Articles

Cited By