Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media

Andrew Schaumberg, PhD
Department of Pathology
Division of Computational Pathology
Poster Overview

Pathologists are medical doctors who use a microscope to look at potentially diseased tissue from patients. Pathologists diagnose a wide range of diseases this way, from infections to cancer, but some patient cases are more challenging than others. For challenging cases, pathologists may ask colleagues for their opinion, including colleagues around the world who are immediately accessible on social media.

Prior studies suggested 22% of social media posts from pathologists in developing countries were seeking opinions on diagnosis. Here, we asked how we could make social media work better for pathologists worldwide. We developed a new tool called “pathobot”, which is an artificial intelligence [AI] bot on Twitter. It responds to pathologists in real-time and finds similar patient cases either shared on social media or published in scientific articles.

To use pathobot, a pathologist shares a patient case in a social media post, along with the tag “@pathobot”. Pathobot responds with a list of the top eight most similar cases. It also notifies each pathologist who shared those cases, so they may discuss further. In this way, we expect our project to cultivate a more connected world of physicians and improve patient care worldwide.

Scientific Abstract

Pathologists rapidly provide a diagnosis on critical health issues. Challenging cases benefit from additional opinions of pathologist colleagues. There is an active worldwide community of pathologists on social media for complementary opinions. Access to pathologists worldwide stands to improve diagnostic accuracy and broaden consensus on next steps in patient care.

From Twitter we curate 13,626 images from 6,351 tweets by 25 pathologists in 13 countries. We supplement this with 113,161 images from 1,074,484 PubMed articles. Through machine learning we accurately (i) identify histopathology stains, (ii) discriminate between tissues, and (iii) differentiate disease states. We repurpose the disease classifier to search for similar disease states, given an image and clinical covariates.

Discriminative task performance is 0.805-0.996 AUROC. Search performance is precision@(k=1)=0.7618±0.0018. The classifiers find that texture and tissue are important clinico-visual features of disease. We implement a social media bot (@pathobot on Twitter) leveraging the classifiers to aid pathologists in obtaining real-time feedback on challenging cases. Pathobot lists similar cases across social media and PubMed.

Our project is a globally distributed expert system facilitating pathological diagnosis and bringing expertise to underserved regions. This is the first pan-tissue pan-disease (i.e. infection to malignancy) method for prediction and search on social media.

Clinical Implications
Artificial intelligence can do “grunt work” to aid pathologists. It finds pathologists worldwide with similar cases, brings them together to discuss next steps in patient care, adapts search results as more information becomes available, and learns as more pathologists collaborate.
Research Areas
Schaumberg AJ, Juarez W, Choudhury SJ, Pastrian LG, Pritt BS, Prieto Pozuelo M, Sotillo Sanchez R, Ho K, Zahra N, Sener BD, Yip S, Xu B, Rao Annavarapu S, Morini A, Jones KA, Rosado-Orozco K, Sirintrapun SJ, Aly M*, Fuchs TJ*
Principal Investigator
Mariam Aly and Thomas Fuchs

Explore Other Posters

8 replies on “Andrew Schaumberg, PhD”

Great question! I agree Dr Wang, protection of PHI is an essential concern. We contacted our Institutional Review Board immediately when this project started, before downloading any data or proceeding. We also benefited from the sustained oversight of the National Institutes of Health, which funded our work, and the United States Department of State, which is involved with any international exchange of [de-identified] patient data.

Our earlier paper and its supplement mention briefly our IRB approval and the criteria we use in our manual data curation process to protect patients

Pathologists themselves have discussed ways to protect PHI on social media, and put these methods into practice. We cite a number of these articles in our work.

In summary, pathologists make rigorous efforts to protect PHI and we are supervised by a number of bodies to properly protect patients.

This reminds me of Dr. Lisa Sanders’ show on Netflix “Diagnosis,” where she crowd-sources to find diagnoses for rare diseases. It seems to help a lot of people who would otherwise be lost. Where along the stages of development is Pathobot? Do you see a role that it can play in the future for diagnosis of rare diseases?

I enjoyed watching episodes of that show, Ava! I’d agree, in some sense, we’re all in the crowd-sourced-expertise boat.

We share a few Pathobot stories in our recent piece in CAP TODAY, which is the news arm of the College of American Pathologists:

For instance, there was a lot of discussion about some rounded structures in a patient’s liver. Was it a parasite? Was it an artifact? Many pathologists worldwide weighed in. It’s discussed at CAP TODAY.

You may also enjoy reading a bit more about us at The Pathologist
There we discuss how artificial intelligence [AI] compliments pathologists, who are responsible for providing a diagnosis. In short, AI can do the ‘grunt work’ of sifting through many patient cases, and notify pathologists with potentially similar cases to invite discussion. Pathologists always make the call for diagnosis.

I liked your AI-related and search-related poster! Please contact me if you’d like to discuss, collaborate, etc.

Stay tuned for more stories, more to come in 2021!

Comments are closed.