OmniMed-FL: A Robust Multimodal Federated Learning Framework for Clinical Diagnosis
Clinical diagnosis rarely rests on one kind of evidence. A radiologist reads the scan alongside the referral note, the history and the labs, and a model that wants to do the same needs both imaging and text. Those records are also the most tightly guarded data a hospital holds, so pooling them across institutions to train one large model is usually off the table.
Federated learning offers a way around this: hospitals train a shared model collaboratively and only model updates ever leave a site, never a patient record. But most federated work in healthcare is single-modality, and real deployments are far from the tidy setting most methods assume — every site has a different patient mix, different scanners and different documentation habits, so the data across clients is strongly non-IID.
OmniMed-FL is a multimodal federated learning framework that combines medical imaging with clinical text for diagnosis while keeping every site's patient data decentralized, and that is built to be robust to exactly that heterogeneity. Under non-IID client splits it reaches a Macro-F1 of 0.956 — 99.1% of what a centralized model trained on the pooled data achieves — so the privacy constraint costs almost nothing in accuracy.
A diagnostic model is only useful if a clinician can check it. The framework therefore pairs each prediction with an explanation and retrieves the supporting evidence, so its output reads as a case for a diagnosis rather than a verdict — the form that clinical decision support actually needs.
- A federated training framework that fuses medical imaging and clinical text for diagnosis without centralizing any patient data.
- Robustness to non-IID data across participating sites — the realistic case where each hospital's population, equipment and documentation differ.
- Explainability and evidence retrieval built into the pipeline, so predictions come with a rationale a clinician can inspect.
- 0.956 Macro-F1 under non-IID settings.
- 99.1% of the performance of a centralized model trained on pooled data — a near-zero privacy penalty.
- Evaluated on combined imaging and clinical-text inputs, not a single modality.