Introduction
HotpotBio is the data lab and research group of Hotpot.ai dedicated to biomedicine.
Due to publishing restrictions on GenAI research, we established this arm to advance science in other ways. We draw inspiration from open source where ephemeral teams drive innovation by attracting talent across organizational boundaries.
We provide expert-verified datasets in clinical reasoning, general clinical AI, oncology, genomics, neurology, pediatrics, drug discovery and development, and other specialty areas. Datasets may contain board-level challenges and multimodal integration.
Data annotation is performed by a curated network of MDs, PhDs, and postdocs from Stanford, UCSF, and other top-tier institutions.
While some data errors are tolerable, perhaps even desirable, for general ML models, uncommon variants in biomedicine may drive pathology. Training on imprecise medical information may lead to misdiagnosis, clinical errors, misfolded proteins, or pharmaceutical drugs with increased MAEs (major adverse events).
Complicating matters, shifting medical facts may invalidate training data and model knowledge. What was true last year may be false today. For instance, in April 2024 the U.S. Preventive Services Task Force reversed its longstanding advice and now urges biennial mammograms starting at age 40 -- down from the previous benchmark of 50 -- for average-risk women, citing rising breast-cancer incidence in younger patients.
Accurate annotation of medical data is challenging and demands expert verification based on the latest guidelines. Even Google DeepMind's relabeled effort of MedQA from 2024 contains errors, which we uncovered.
This is why HotpotBio exists: to provide rigorously validated, expert-curated datasets and benchmarks in pursuit of advancing ML/AI in clinical and broader biomedical applications.
Problem
Cancer is the second leading cause of death worldwide, claiming the lives of roughly 10 million people per year and devastating the lives of millions more [1].
With 8 billion people and only 12.7 million doctors, personalized healthcare is impossible today. Human doctors alone cannot bridge this gap and provide the attentive care everyone deserves.
North Star
Our mission is to advance cancer research and AI doctors with open research and data annotation services for frontier labs [2].
While curing cancer and AI doctors remain fantasies today, many groups are working feverishly to close the gap between dream and reality.
We hope to play a tiny part.
Goal
Quality datasets and benchmarks can unlock rapid progress in machine learning (ML), but most technologists lack medical expertise while most doctors lack technical expertise.
Without accurate and comprehensive datasets and evaluations, it is hard to train models and improve AI -- not unlike teaching students with poor textbooks and exams.
Our objective is to package medical knowledge into a format suitable for ML engineers and researchers to further AI biomedicine, regardless of medical background.
Concretely, this work involves developing datasets and benchmarks in collaboration with medical professionals from Stanford, UCSF, and other leading institutions.
AI Doctor Research
The research is broadly organized into the categories below. Descriptions are tailored for non-technical audiences.
AI Vision
- Surgery: how to provide real-time anatomy detection for surgeons and self-paced video education for residents?
- Medical imaging: how to improve detection of disease, musculoskeletal injury, and anatomical abnormalities, both in the clinic and in telemedicine?
AI Hearing
- Biomedical transcription: how to transcribe audio, particularly conversations with heavy accents?
AI Reading
- Biomedical RAG factuality: how to accurately answer questions given a specific context?
- Biomedical text understanding: how to extract information and entities from both structured and unstructured text?
- Biomedical reliability: how to achieve consistency across identical conditions?
AI Privacy
- Biomedical privacy: how to preserve patient confidentiality while expanding datasets and facilitating multi-institute collaboration?
AI Reasoning
- Biomedical reasoning: how to ensure diagnoses and recommendations match expert clinical judgment?
Cancer Research
Our research investigates the association between Epstein-Barr virus (EBV) and cancer, concentrating on the topics below.
Viruses cause cervical cancer, Burkitt lymphoma, nasopharyngeal cancer (NPC), and several other cancer types, but the data is inconclusive for more common cancer types like breast cancer and lung cancer [3-8].
- 1K non-smoking lung cancer dataset: see below.
- 1K TNBC dataset: see below.
- Joint Omics Adaptive Nosological (JOAN) detection framework: systematic computational-experimental framework for detecting viruses in cancer samples, starting with adenocarcinomas.
- EBV association with breast cancer, starting with triple-negative breast cancer (TNBC).
- EBV association with lung cancer, starting with non-smoking lung cancer.
- EBV association with NPC, Burkitt lymphoma, and gastric cancer.
- EBV association with MYC.
- EBV sequence conservation.
How To Collaborate
- Review Collaboration Areas.
- To maintain the integrity of our network and the quality of data annotations, collaboration is by invitation only. We exclusively partner with MDs, PhDs, and postdocs from leading research institutions.
- Candidates must provide the details below and may reach out with questions. See below for contact information.
- Summary of experience and credentials in areas of interest
- Resume
- Publishing history
Collaboration Areas
We welcome contributors in the areas below.
- Bioinformatics
- Cardiology
- Clinical Investigations
- Endocrinology
- Gastroenterology
- Genomics
- Geriatrics
- Hematology
- Immunology
- Infectious Disease
- Medicinal Chemistry
- Neurology
- Oncology
- Pathology
- Pediatrics
- Pharmacogenomics
- Pharmacology
- Pharmaceutical Science
- Psychiatry
- Pulmonology
- Radiology
- Surgery
- Toxicology
- Translational Medicine
- Virology
Work can fit any schedule and take one of many forms:
- Creating 100-200 multiple choice questions per specialty
- Reviewing questions
- Defining key clinical tasks and requirements
- Conducting lit reviews
- Reviewing paper drafts
Machine Learning
- VLM
- Computer vision
- NLP
- LLM
Software Development
- Full-stack web development for simplifying how healthcare professionals create and review training data
Research Culture
HotpotBio focuses on science, deferring policy and ethics to other forums.
Although this position may not appeal to all, the benefit of clear values is cultivating an environment where everyone can concentrate on science. Organizational theory demonstrates that teams united by shared priorities and explicit expectations foster more productive collaborations.
I understand the anxiety around AI, but our culture is rooted in a deep study of technology history and societal progress. Throughout time, a consistent pattern has characterized the emergence of disruptive technology. This cycle was observed with books, computers, the web, and it's repeating again with AI. Fear dominates the discourse while concerned critics seek to curb capabilities and protect the masses.
With hindsight, we know those noble intentions were misguided and failed to account for the transformative benefits spawned by innovation. General technology, by definition, is wieldable for good or bad, but the good vastly outweighs the bad. This propels the world to greater heights of prosperity and accessibility.
On ethics, most people aspire to be moral and responsible, but the challenge is: whose values dictate tradeoffs and resolve disputes? Officials from California, Texas, China, India, France, Japan, the UK, Saudi Arabia, or where? Whose risk profile shines the way forward? For instance, GPT-2, GPT-3, and GPT-4 were all considered too dangerous for the average person, but those worries proved exaggerated at best and unfounded at worst. Moreover, it's presumptuous to assume one jurisdiction can bottle up software ingenuity or constrain global innovation. If America surrenders AI leadership, other nations will readily fill the void.
While healthy people can afford the luxury of endless deliberation, the sick cannot. With nearly 800K people passing away each month from cancer, discovering breakthroughs even one month sooner can save lives and spare immeasurable suffering.
Intelligent people may disagree. I respect different opinions and hope others can as well.
TNBC & Non-smoking Lung Cancer Datasets
Novel datasets for TNBC and non-smoking lung cancer could power tens to hundreds of studies and hopefully set a new precedent for tackling tumor subtypes. See here and here for details.
Contact Information
Clarence Hu
