Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.
Nature Biomedical Engineering (2022 )Cite this article
Weakly supervised deep-learning models for the analysis of whole-slide images from tumour biopsies perform better at prognostic tasks if the models incorporate context from the local microenvironment.
Recent developments in deep learning and the common availability of computing resources and digitized tissue slides have enabled the computational analysis of gigapixel whole-slide images (WSIs)1. Digital pathology and computational pathology offer hopes for more standardized and objective diagnoses, prognoses and predictions of therapeutic responses, and for the discovery of new morphological phenotypes of clinical relevance. However, for these to become reality, technical challenges that are inherent to pathology images must be overcome. On the one hand, WSIs can be several gigapixels in size — and hence can be much larger than natural images — which limits the applicability (to computational pathology) of many algorithms that were developed for the analysis of conventional images. On the other hand, accurate representations of WSIs need to incorporate information at multiple resolution scales, from fine-level morphological descriptions (such as cellular patterns) to coarse-level contextual information (such as tissue architecture and features of the tumour microenvironment). Limited contextual information constrains the power of computational pathology for patient stratification and biomarker discovery. For example, the presence of immune cells may lead to different prognoses depending on whether the cells are in an inflammatory environment or an infiltrative one. Moreover, the complexity of acquiring dense pixel-level annotations calls for the use of weakly supervised machine-learning strategies, such as assigning the label for the case to every small patch in a gigapixel image2, or the use of multiple-instance learning (MIL) (that is, the use of labelled ‘bags of patches’)3,4.
Your institute does not have access to this article
Get full journal access for 1 year
All prices are NET prices. VAT will be added later in the checkout. Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Bera, K. et al. Nat. Rev. Clin. Oncol. 16, 703–715 (2019).
Coudray, N. et al. Nat. Med. 24, 1559–1567 (2018).
Campanella, G. et al. Nat. Med. 25, 1301–1309 (2019).
Lu, M. Y. et al. Nat. Biomed. Eng. 5, 555–570 (2021).
Chen, R. J. et al. Whole slide images are 2D point clouds: context-aware survival prediction using patch-based graph convolutional networks. In MICCAI 2021: Medical Image Computing and Computer Assisted Intervention (eds de Bruijne, M. et al.) 339–349 (Springer, 2021).
Lee, Y. et al. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-022-00923-0 (2022).
Jaume, G. et al. Quantifying explainers of graph neural networks in computational pathology. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (eds IEEE staff) 8102–8112 (IEEE, 2021).
Dosovitskiy, A. et al. An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations 2021 Paper 1909 (ICLR, 2021).
Shao, Z. C. et al. TransMIL: transformer based correlated multiple instance learning for whole slide image classification. In Advances in Neural Information Processing Systems 34 (eds Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. S. & Wortman Vaughan, J.) 2136–2147 (NeurIPS, 2021).
Chen, R. J. et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (eds IEEE staff) 16144–16155 (IEEE, 2022).
Department of Pathology, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA
Guillaume Jaume, Andrew H. Song & Faisal Mahmood
Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
Guillaume Jaume, Andrew H. Song & Faisal Mahmood
Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
Guillaume Jaume, Andrew H. Song & Faisal Mahmood
Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
Guillaume Jaume, Andrew H. Song & Faisal Mahmood
Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
The authors declare no competing interests.
Jaume, G., Song, A.H. & Mahmood, F. Integrating context for superior cancer prognosis. Nat. Biomed. Eng (2022). https://doi.org/10.1038/s41551-022-00924-z
DOI: https://doi.org/10.1038/s41551-022-00924-z
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
Nature Biomedical Engineering (Nat. Biomed. Eng) ISSN 2157-846X (online)
Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.