Add MedFlamingo Multimodal VQA Pipeline

docs/api/datasets.rst

@@ -238,6 +238,7 @@ Available Datasets
     datasets/pyhealth.datasets.BMDHSDataset
     datasets/pyhealth.datasets.COVID19CXRDataset
     datasets/pyhealth.datasets.ChestXray14Dataset
+    datasets/pyhealth.datasets.VQARADDataset
     datasets/pyhealth.datasets.TUABDataset
     datasets/pyhealth.datasets.TUEVDataset
     datasets/pyhealth.datasets.ClinVarDataset

docs/api/datasets/pyhealth.datasets.VQARADDataset.rst

@@ -0,0 +1,11 @@
+pyhealth.datasets.VQARADDataset
+===================================
+
+The VQA-RAD dataset for medical visual question answering. The dataset loader
+converts the public JSON annotations into a flat metadata CSV that PyHealth can
+ingest, and its default task is :class:`~pyhealth.tasks.MedicalVQATask`.
+
+.. autoclass:: pyhealth.datasets.VQARADDataset
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/api/models.rst

@@ -48,6 +48,9 @@ routes each feature type automatically.
    * - :doc:`models/pyhealth.models.GraphCare`
      - You want to augment EHR codes with a medical knowledge graph
      - Combines code sequences with a :class:`~pyhealth.graph.KnowledgeGraph`
+   * - :doc:`models/pyhealth.models.MedFlamingo`
+     - You are solving multimodal medical tasks with images plus text prompts (for example, VQA-style radiology QA)
+     - Flamingo-style architecture with a frozen vision encoder + frozen language model connected by gated cross-attention layers
 
 How BaseModel Works
 --------------------
@@ -194,6 +197,7 @@ API Reference
     models/pyhealth.models.ConCare
     models/pyhealth.models.Agent
     models/pyhealth.models.GRASP
+    models/pyhealth.models.MedFlamingo
     models/pyhealth.models.MedLink
     models/pyhealth.models.TCN
     models/pyhealth.models.TFMTokenizer

docs/api/models/pyhealth.models.MedFlamingo.rst

@@ -0,0 +1,40 @@
+pyhealth.models.MedFlamingo
+===================================
+
+MedFlamingo: multimodal medical few-shot learner.
+
+This reference covers the visual resampler, the gated cross-attention
+building block, and the complete MedFlamingo model used in the VQA-RAD
+integration branch.
+
+**Paper:** Moor et al. "Med-Flamingo: a Multimodal Medical Few-shot Learner" ML4H 2023.
+
+.. note::
+
+   ``forward()`` follows the PyHealth training contract for dataset-backed
+   classification-style use, while ``generate()`` provides the multimodal
+   prompting path for direct medical VQA generation.
+
+PerceiverResampler
+------------------
+
+.. autoclass:: pyhealth.models.medflamingo.PerceiverResampler
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+MedFlamingoLayer
+----------------
+
+.. autoclass:: pyhealth.models.medflamingo.MedFlamingoLayer
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+MedFlamingo
+-----------
+
+.. autoclass:: pyhealth.models.MedFlamingo
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/api/tasks.rst

@@ -213,6 +213,7 @@ Available Tasks
     DKA Prediction (MIMIC-IV) <tasks/pyhealth.tasks.dka>
     Drug Recommendation <tasks/pyhealth.tasks.drug_recommendation>
     Length of Stay Prediction <tasks/pyhealth.tasks.length_of_stay_prediction>
+    Medical VQA <tasks/pyhealth.tasks.MedicalVQATask>
     Medical Transcriptions Classification <tasks/pyhealth.tasks.MedicalTranscriptionsClassification>
     Mortality Prediction (Next Visit) <tasks/pyhealth.tasks.mortality_prediction>
     Mortality Prediction (StageNet MIMIC-IV) <tasks/pyhealth.tasks.mortality_prediction_stagenet_mimic4>

docs/api/tasks/pyhealth.tasks.MedicalVQATask.rst

@@ -0,0 +1,12 @@
+pyhealth.tasks.MedicalVQATask
+===================================
+
+Medical visual question answering task for paired radiology images and
+questions. This task treats VQA-RAD answers as a multiclass prediction target
+so the resulting ``SampleDataset`` can be trained with the standard PyHealth
+trainer loop.
+
+.. autoclass:: pyhealth.tasks.MedicalVQATask
+    :members:
+    :undoc-members:
+    :show-inheritance: