Deep learning-based annotation transfer between molecular imaging modalities: an automated workflow for multimodal data integration

Race, A. M. et al. (2021) Deep learning-based annotation transfer between molecular imaging modalities: an automated workflow for multimodal data integration. Analytical Chemistry, 93(6), pp. 3061-3071. (doi: 10.1021/acs.analchem.0c02726) (PMID:33534548)

Abstract

An ever-increasing array of imaging technologies are being used in the study of complex biological samples, each of which provides complementary, occasionally overlapping information at different length scales and spatial resolutions. It is important to understand the information provided by one technique in the context of the other to achieve a more holistic overview of such complex samples. One way to achieve this is to use annotations from one modality to investigate additional modalities. For microscopy-based techniques, these annotations could be manually generated using digital pathology software or automatically generated by machine learning (including deep learning) methods. Here, we present a generic method for using annotations from one microscopy modality to extract information from complementary modalities. We also present a fast, general, multimodal registration workflow [evaluated on multiple mass spectrometry imaging (MSI) modalities, matrix-assisted laser desorption/ionization, desorption electrospray ionization, and rapid evaporative ionization mass spectrometry] for automatic alignment of complex data sets, demonstrating an order of magnitude speed-up compared to previously published work. To demonstrate the power of the annotation transfer and multimodal registration workflows, we combine MSI, histological staining (such as hematoxylin and eosin), and deep learning (automatic annotation of histology images) to investigate a pancreatic cancer mouse model. Neoplastic pancreatic tissue regions, which were histologically indistinguishable from one another, were observed to be metabolically different. We demonstrate the use of the proposed methods to better understand tumor heterogeneity and the tumor microenvironment by transferring machine learning results freely between the two modalities.

• Item Type: Articles
• Additional Information: The authors wish to thank the wider Cancer Research UK Grand Challenge Rosetta Consortium members for supporting this research. They would also like to thank the Biological Services Unit at the CRUK Beatson Institute. The authors gratefully acknowledge funding from Cancer Research UK (A24034, A25142, A17196, A21139, and A25233) which supported this work.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Morton, Professor Jen and Goodwin, Dr Richard and Sansom, Professor Owen and Campbell, Dr Andrew
• Authors: Race, A. M., Sutton, D., Hamm, G., Maglennon, G., Morton, J. P., Strittmatter, N., Campbell, A., Sansom, O. J., Wang, Y., Barry, S. T., Takáts, Z., Goodwin, R. J.A., and Bunch, J.
• College/School: College of Medical Veterinary and Life Sciences > School of Cancer Sciences; College of Medical Veterinary and Life Sciences > School of Infection & Immunity
• Journal Name: Analytical Chemistry
• Publisher: American Chemical Society
• ISSN: 0003-2700
• ISSN (Online): 1520-6882
• Published Online: 03 February 2021