Synthesis of diagnostic quality cancer pathology images by generative adversarial networks
Adrian B Levine
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
†These authors contributed equally to this work.
Search for more papers by this authorJason Peng
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
†These authors contributed equally to this work.
Search for more papers by this authorDavid Farnell
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorMitchell Nursey
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorYiping Wang
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorJulia R Naso
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorHezhen Ren
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorHossein Farahani
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorColin Chen
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorDerek Chiu
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorAline Talhouk
Department of Obstetrics and Gynecology, University of British Columbia, Vancouver, Canada
Search for more papers by this authorBrandon Sheffield
Department of Pathology, William Osler Health Centre-Brampton Civic Hospital, Brampton, Canada
Search for more papers by this authorMaziar Riazy
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorPhilip P Ip
Department of Pathology, University of Hong Kong, Hong Kong SAR, PR China
Search for more papers by this authorCarlos Parra-Herran
Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Canada
Search for more papers by this authorAnne Mills
Department of Pathology, University of Virginia, Charlottesville, VA, USA
Search for more papers by this authorNaveena Singh
Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
Search for more papers by this authorBasile Tessier-Cloutier
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorTaylor Salisbury
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorJonathan Lee
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorTim Salcudean
Electrical & Computer Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorSteven JM Jones
Canada's Michael Smith Genome Sciences Centre, Vancouver, Canada
Search for more papers by this authorDavid G Huntsman
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorCorresponding Author
C Blake Gilks
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Correspondence to:
A Bashashati, 2222 Health Sciences Mall, Vancouver, V6T 1Z3 BC, Canada. E-mail: [email protected]; or S Yip, E-mail: [email protected]; or CB Gilks,
E-mail: [email protected]
Search for more papers by this authorCorresponding Author
Stephen Yip
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Correspondence to:
A Bashashati, 2222 Health Sciences Mall, Vancouver, V6T 1Z3 BC, Canada. E-mail: [email protected]; or S Yip, E-mail: [email protected]; or CB Gilks,
E-mail: [email protected]
Search for more papers by this authorCorresponding Author
Ali Bashashati
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Electrical & Computer Engineering, University of British Columbia, Vancouver, Canada
Correspondence to:
A Bashashati, 2222 Health Sciences Mall, Vancouver, V6T 1Z3 BC, Canada. E-mail: [email protected]; or S Yip, E-mail: [email protected]; or CB Gilks,
E-mail: [email protected]
Search for more papers by this authorAdrian B Levine
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
†These authors contributed equally to this work.
Search for more papers by this authorJason Peng
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
†These authors contributed equally to this work.
Search for more papers by this authorDavid Farnell
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorMitchell Nursey
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorYiping Wang
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorJulia R Naso
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorHezhen Ren
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorHossein Farahani
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorColin Chen
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorDerek Chiu
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorAline Talhouk
Department of Obstetrics and Gynecology, University of British Columbia, Vancouver, Canada
Search for more papers by this authorBrandon Sheffield
Department of Pathology, William Osler Health Centre-Brampton Civic Hospital, Brampton, Canada
Search for more papers by this authorMaziar Riazy
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorPhilip P Ip
Department of Pathology, University of Hong Kong, Hong Kong SAR, PR China
Search for more papers by this authorCarlos Parra-Herran
Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Canada
Search for more papers by this authorAnne Mills
Department of Pathology, University of Virginia, Charlottesville, VA, USA
Search for more papers by this authorNaveena Singh
Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
Search for more papers by this authorBasile Tessier-Cloutier
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorTaylor Salisbury
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorJonathan Lee
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorTim Salcudean
Electrical & Computer Engineering, University of British Columbia, Vancouver, Canada
Search for more papers by this authorSteven JM Jones
Canada's Michael Smith Genome Sciences Centre, Vancouver, Canada
Search for more papers by this authorDavid G Huntsman
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Search for more papers by this authorCorresponding Author
C Blake Gilks
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Correspondence to:
A Bashashati, 2222 Health Sciences Mall, Vancouver, V6T 1Z3 BC, Canada. E-mail: [email protected]; or S Yip, E-mail: [email protected]; or CB Gilks,
E-mail: [email protected]
Search for more papers by this authorCorresponding Author
Stephen Yip
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
Correspondence to:
A Bashashati, 2222 Health Sciences Mall, Vancouver, V6T 1Z3 BC, Canada. E-mail: [email protected]; or S Yip, E-mail: [email protected]; or CB Gilks,
E-mail: [email protected]
Search for more papers by this authorCorresponding Author
Ali Bashashati
Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
Electrical & Computer Engineering, University of British Columbia, Vancouver, Canada
Correspondence to:
A Bashashati, 2222 Health Sciences Mall, Vancouver, V6T 1Z3 BC, Canada. E-mail: [email protected]; or S Yip, E-mail: [email protected]; or CB Gilks,
E-mail: [email protected]
Search for more papers by this authorThis paper has been previously posted as a preprint on bioRxiv (https://doi.org/10.1101/2020.02.24.963553).
DGH is an Associate Editor of The Journal of Pathology. No other conflicts of interest were declared.
Funding information: Canadian Institutes of Health Research, Grant/Award Number: 201903PJT-418734-MPI-CAAA-37916; Natural Sciences and Engineering Research Council of Canada, Grant/Award Number: RGPIN-2019-04896z; VGH & UBC Hospital Foundation: Carraresi OVCARE Fund
Abstract
Deep learning-based computer vision methods have recently made remarkable breakthroughs in the analysis and classification of cancer pathology images. However, there has been relatively little investigation of the utility of deep neural networks to synthesize medical images. In this study, we evaluated the efficacy of generative adversarial networks to synthesize high-resolution pathology images of 10 histological types of cancer, including five cancer types from The Cancer Genome Atlas and the five major histological subtypes of ovarian carcinoma. The quality of these images was assessed using a comprehensive survey of board-certified pathologists (n = 9) and pathology trainees (n = 6). Our results show that the real and synthetic images are classified by histotype with comparable accuracies and the synthetic images are visually indistinguishable from real images. Furthermore, we trained deep convolutional neural networks to diagnose the different cancer types and determined that the synthetic images perform as well as additional real images when used to supplement a small training set. These findings have important applications in proficiency testing of medical practitioners and quality assurance in clinical laboratories. Furthermore, training of computer-aided diagnostic systems can benefit from synthetic images where labeled datasets are limited (e.g. rare cancers). We have created a publicly available website where clinicians and researchers can attempt questions from the image survey (http://gan.aimlab.ca/). © 2020 Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.
Supporting Information
Filename | Description |
---|---|
path5509-sup-0001-Appendix_S1.docxWord 2007 document , 14.9 MB | Supplementary methods Figure S1. Additional examples of real (n = 2) and synthetic (n = 4) images from TCGA dataset Figure S2. Additional examples of real (n = 2) and synthetic (n = 4) images from the ovarian carcinoma dataset Figure S3. Examples of images generated by alternative generative methods Figure S4. Image information entropy of the synthetic images produced by different generative models (VAE, ESRGAN, DTS, progressive GAN) and real images Figure S5. Confusion matrices of survey responses Figure S6. Additional metrics for classifiers Figure S7. Examples of rare artifacts that can be present in synthetic images Table S1. Breakdown of GAN training images Table S2. Sample size calculation for image surveys Table S3. FID scores for images generated by different generative methods Table S4. Results for image entropy calculations between different generative methods Table S5. TCGA survey results for board-certified pathologists Table S6. Pathologist classification results by histological subtypes Table S7. Fleiss' kappa (representing inter-observer agreement) for all, synthetic, and real images Table S8. Ovarian carcinoma survey results for board-certified pathologists Table S9. Pathology trainee results on ovarian carcinoma survey Table S10. Performance measures statistics (mean, SD, median) associated with classifiers trained on baseline, baseline + real, and baseline + synthetic data for the OVCARE and TCGA datasets Table S11. Statistical comparison (P-values) of various performance measures associated with classifiers trained on baseline, baseline + real, and baseline + synthetic data |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
References
- 1Ehteshami Bejnordi B, Veta M, Johannes van Diest P, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 2017; 318: 2199–2210.
- 2Steiner DF, MacDonald R, Liu Y, et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am J Surg Pathol 2018; 42: 1636–1646.
- 3Chen P-HC, Gadepalli K, MacDonald R, et al. An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis. Nat Med 2019; 25: 1453–1457.
- 4Mobadersany P, Yousefi S, Amgad M, et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A 2018; 115: E2970–E2979.
- 5Courtiol P, Maussion C, Moarii M, et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat Med 2019; 25: 1519–1525.
- 6Nagpal K, Foote D, Liu Y, et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. NPJ Digit Med 2019; 2: 48.
- 7Coudray N, Ocampo PS, Sakellaropoulos T, et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med 2018; 24: 1559–1567.
- 8Kather JN, Pearson AT, Halama N, et al. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med 2019; 25: 1054–1056.
- 9Goodfellow IJ, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks. arXiv: 1406.2661 [stat.ML]; June 2014.
- 10Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv: 1511.06434 [cs.LG]; November 2015.
- 11Karras T, Aila T, Laine S, et al. Progressive growing of GANs for improved quality, stability, and variation. International Conference on Learning Representations. arXiv: 1710.10196 [cs.NE]; February 2018.
- 12Quiros AC, Murray-Smith R, Yuan K. Pathology GAN: learning deep representations of cancer tissue. arXiv:1907.02644 [eess.IV]; July 2019.
- 13Hou L, Agarwal A, Samaras D, et al. Unsupervised histopathology image synthesis. arXiv:1712.05021 [cs.CV]; December 2017.
- 14Xue Y, Zhou Q, Ye J, et al. Synthetic augmentation and feature-based filtering for improved cervical histopathology image classification. arXiv: 1907.10655 [eess.IV]; July 2019.
- 15Burlina PM, Joshi N, Pacheco KD, et al. Assessment of deep generative models for high-resolution synthetic retinal image generation of age-related macular degeneration. JAMA Ophthalmol 2019; 137: 258–264.
- 16Ghorbani A, Natarajan V, Coz D, et al. DermGAN: synthetic generation of clinical skin images with pathology. arXiv: 1911.08716 [cs.CV]; November 2019.
- 17Frid-Adar M, Diamant I, Klang E, et al. GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. arXiv: 1803.01229 [cs.CV]; March 2018.
- 18Han C, Murao K, Satoh S, et al. Learning more with less: GAN-based medical image augmentation. arXiv: 1904.00838 [cs.CV]; March 2019.
- 19Uzunova H, Ehrhardt J, Jacob F, et al. Multi-scale GANs for memory-efficient generation of high resolution medical images. arXiv: 1907.01376 [eess.IV]; July 2019.
- 20Yi X, Walia E, Babyn P. Generative adversarial network in medical imaging: a review. Med Image Anal 2019; 58: 101552.
- 21McGaghie WC. Medical education research as translational science. Sci Transl Med 2010; 2: 19cm8.
- 22Price WN 2nd, Cohen IG. Privacy in the age of medical big data. Nat Med 2019; 25: 37–43.
- 23Levine AB, Schlosser C, Grewal J, et al. Rise of the machines: advances in deep learning for cancer diagnosis. Trends Cancer 2019; 5: 157–169.
- 24Grossman RL, Heath AP, Ferretti V, et al. Toward a shared vision for cancer genomic data. N Engl J Med 2016; 375: 1109–1112.
- 25Abadi M, Agarwal A, Barham P, et al. TensorFlow: large-scale machine learning on heterogeneous systems. arXiv: 1603.04467 [cs.DC]; March 2016.
- 26Kingma DP, Ba J. Adam: a method for stochastic optimization. arXiv: 1412.6980 [cs.LG]; December 2014.
- 27Liu Y, Gadepalli K, Norouzi M, et al. Detecting cancer metastases on gigapixel pathology images. arXiv: 1703.02442 [cs.CV]; March 2017.
- 28Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv: 1409.1556 [cs.CV]; September 2014.
- 29Paszke A, Gross S, Massa F, et al. PyTorch: an imperative style, high-performance deep learning library. arXiv: 1912.01703 [cs.LG]; December 2019.
- 30Rotondi MA, Donner A. A confidence interval approach to sample size estimation for interobserver agreement studies with multiple raters and outcomes. J Clin Epidemiol 2012; 65: 778–784.
- 31Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960; 20: 37–46.
- 32McHugh ML. Interrater reliability: the kappa statistic. Biochem Med 2012; 22: 276–282.
- 33Doersch C. Tutorial on variational autoencoders. arXiv: 1606.05908 [stat.ML]; June 2016.
- 34Wang X, Yu K, Wu S, et al. ESRGAN: enhanced super-resolution generative adversarial networks. arXiv: 1809.00219 [cs.CV]; September 2018.
- 35Gatys LA, Ecker AS, Bethge M. Texture synthesis using convolutional neural networks. arXiv: 1505.07376 [cs.CV]; May 2015.
- 36Salimans T, Goodfellow I, Zaremba W, et al. Improved techniques for training GANs. In Advances in Neural Information Processing Systems. DD Lee, M Sugiyama, UV Luxburg, I Guyon, R Garnett (Eds), 2016; 2234–2242.
- 37Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull 1971; 76: 378–382.
- 38Evans AJ, Salama ME, Henricks WH, et al. Implementation of whole slide imaging for clinical purposes: issues to consider from the perspective of early adopters. Arch Pathol Lab Med 2017; 141: 944–959.
- 39Mahmood F, Borders D, Chen R, et al. Deep adversarial training for multi-organ nuclei segmentation in histopathology images. IEEE Trans Med Imaging 2019. https://doi.org/10.1109/TMI.2019.2927182.
- 40Shaban MT, Baur C, Navab N, et al. Staingan: stain style transfer for digital histological images. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019); 2019: 953–956.
- 41Xu Z, Moro CF, Bozóky B, et al. GAN-based virtual re-staining: a promising solution for whole slide image analysis. arXiv: 1901.04059 [cs.CV]; January 2019.
- 42Schlegl T, Seeböck P, Waldstein SM, et al. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Information Processing in Medical Imaging. Niethammer M, Styner M, Aylward S, Zhu H, Oguz I, Yap P-T, Shen D (Eds). New York: Springer, 2017; 146–157.
- 43Kommoss S, Gilks CB, du Bois A, et al. Ovarian carcinoma diagnosis: the clinical impact of 15 years of change. Br J Cancer 2016; 115: 993–999.
- 44Köbel M, Kalloger SE, Baker PM, et al. Diagnosis of ovarian carcinoma cell type is highly reproducible: a transcanadian study. Am J Surg Pathol 2010; 34: 984–993.
- 45Köbel M, Kalloger SE, Boyd N, et al. Ovarian carcinoma subtypes are different diseases: implications for biomarker studies. PLoS Med 2008; 5: e232.
- 46Köbel M, Kalloger SE, Lee S, et al. Biomarker-based ovarian carcinoma typing: a histologic investigation in the ovarian tumor tissue analysis consortium. Cancer Epidemiol Biomarkers Prev 2013; 22: 1677–1686.
- 47Arjovsky M, Bottou L. Towards principled methods for training generative adversarial networks. arXiv: 1701.04862 [stat.ML]; January 2017.
- 48He J, Baxter SL, Xu J, et al. The practical implementation of artificial intelligence technologies in medicine. Nat Med 2019; 25: 30–36.
- 49Dwork C. Differential privacy: a survey of results. In Theory and Applications of Models of Computation. Agrawal M, Du D-Z, Duan Z, Li A (Eds). Springer: Berlin, 2008; 1–19.
10.1007/978-3-540-79228-4_1 Google Scholar
- 50Campanella G, Hanna MG, Geneslaw L, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med 2019; 25: 1301–1309.
- 51Leape LL, Fromson JA. Problem doctors: is there a system-level solution? Ann Intern Med 2006; 144: 107–115.
- 52Barratt S, Sharma R. A note on the inception score. arXiv: 1801.01973 [stat.ML]; January 2018.
- 53Heusel M, Ramsauer H, Unterthiner T, et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. arXiv: 1706.08500 [cs.LG]; June 2017.