Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media

Andrew Schaumberg, PhD
Department of Pathology
Division of Computational Pathology
Poster Overview

Pathologists are medical doctors who use a microscope to look at potentially diseased tissue from patients. Pathologists diagnose a wide range of diseases this way, from infections to cancer, but some patient cases are more challenging than others. For challenging cases, pathologists may ask colleagues for their opinion, including colleagues around the world who are immediately accessible on social media.

Prior studies suggested 22% of social media posts from pathologists in developing countries were seeking opinions on diagnosis. Here, we asked how we could make social media work better for pathologists worldwide. We developed a new tool called “pathobot”, which is an artificial intelligence [AI] bot on Twitter. It responds to pathologists in real-time and finds similar patient cases either shared on social media or published in scientific articles.

To use pathobot, a pathologist shares a patient case in a social media post, along with the tag “@pathobot”. Pathobot responds with a list of the top eight most similar cases. It also notifies each pathologist who shared those cases, so they may discuss further. In this way, we expect our project to cultivate a more connected world of physicians and improve patient care worldwide.

Scientific Abstract

Pathologists rapidly provide a diagnosis on critical health issues. Challenging cases benefit from additional opinions of pathologist colleagues. There is an active worldwide community of pathologists on social media for complementary opinions. Access to pathologists worldwide stands to improve diagnostic accuracy and broaden consensus on next steps in patient care.

From Twitter we curate 13,626 images from 6,351 tweets by 25 pathologists in 13 countries. We supplement this with 113,161 images from 1,074,484 PubMed articles. Through machine learning we accurately (i) identify histopathology stains, (ii) discriminate between tissues, and (iii) differentiate disease states. We repurpose the disease classifier to search for similar disease states, given an image and clinical covariates.

Discriminative task performance is 0.805-0.996 AUROC. Search performance is precision@(k=1)=0.7618±0.0018. The classifiers find that texture and tissue are important clinico-visual features of disease. We implement a social media bot (@pathobot on Twitter) leveraging the classifiers to aid pathologists in obtaining real-time feedback on challenging cases. Pathobot lists similar cases across social media and PubMed.

Our project is a globally distributed expert system facilitating pathological diagnosis and bringing expertise to underserved regions. This is the first pan-tissue pan-disease (i.e. infection to malignancy) method for prediction and search on social media.

Clinical Implications
Artificial intelligence can do “grunt work” to aid pathologists. It finds pathologists worldwide with similar cases, brings them together to discuss next steps in patient care, adapts search results as more information becomes available, and learns as more pathologists collaborate.
Research Areas
Schaumberg AJ, Juarez W, Choudhury SJ, Pastrian LG, Pritt BS, Prieto Pozuelo M, Sotillo Sanchez R, Ho K, Zahra N, Sener BD, Yip S, Xu B, Rao Annavarapu S, Morini A, Jones KA, Rosado-Orozco K, Sirintrapun SJ, Aly M*, Fuchs TJ*
Principal Investigator
Mariam Aly and Thomas Fuchs

Explore Other Posters

Leave a Reply

Your email address will not be published. Required fields are marked *