Multimodal Medical Tool-Use Agent

MMedAgent-Plus

A vision–language planner that learns to orchestrate 18 state-of-the-art specialist models across 8 modalities — selecting the right tools, executing them in order, and grounding its answers in their outputs.

Ruoxin Wang1,2, Jinglong Xiong1, Yuanting Pan3, Xiaoquan Kong1,2, Binxu Li4, Ruiyang Ji5, Zhe Xu6, Yaqian Chen1, Jiayuan Ding7, Mingjie Li8, Yu Zhang2, Ehsan Adeli2, Kilian Pohl2, Zihao Lin9*, Haoyu Dong1*, Yixin Wang2*

1Duke University · 2Stanford University · 3Yale University · 4Princeton University · 5Virginia Tech · 6Columbia University · 7Hippocratic AI · 8Shanghai Jiao Tong University · 9University of California, Davis

* Corresponding authors

📄 arXiv soon Code ▶ Demo video ❝ BibTeX
18specialist models*
12medical tasks
8modalities
209Ktraining samples
98.1%tool-call accuracy†
90.6%end-to-end invocation

* each the state of the art for its task; exposed to the planner as 19 tools (Grounding DINO + MedSAM is a composed tool).
† multi-tool multi-round dialogues; 91.0% single-tool multi-round, 80.7% multi-tool single-round.

MMedAgent-Plus workflow orchestration pipeline: planning, tool execution, and aggregation
At each turn the planner receives the user query, images, dialogue history, and any tool outputs returned so far. It decides whether external tools are required, emits structured JSON tool calls (e.g. {"API_name": "MedSAM", "API_params": {"boxes": [[x1,y1,x2,y2]]}}), executes the selected specialists, and finally aggregates returned evidence — segmentation masks, bounding boxes, generated reports — into the answer. Later tool calls can depend on artifacts produced earlier, so compositional workflows like segment → report emerge naturally.

Overview

Why a medical tool-use agent?

Clinical questions are workflows, not single predictions — findings must be localized, characterized, and integrated step by step. Specialist models solve individual steps but don't coordinate; generalist VLMs cover many tasks but can't orchestrate them. MMedAgent-Plus closes this gap: a planner that selects specialist tools, executes them in sequence, and grounds its answer in their outputs.

Specialist models Generalist VLMs MMedAgent-Plus
Specialist-level accuracy ✓ on one task ✗ broad but shallow ✓ calls the right specialist
Broad task coverage ✗ single task each ✓ ✓ 12 tasks · 8 modalities
Workflow orchestration ✗ isolated components ✗ one-shot answers ✓ learned plan → execute → aggregate
Evidence-grounded answers ✗ raw outputs only ✗ prone to hallucination ✓ grounded in tool outputs
Swap in a stronger specialist without retraining ✗ ✗ ✓ when its interface stays compatible

Tasks

12 medical tasks, 8 modalities

The planner covers 12 medical tasks in four functional categories, across text, X-ray, CT, MRI, ultrasound, fundus, OCT, and pathology. Each task is served by one or more specialist tools from the library below.

Text Understanding & Generation

QA & Rationale GenerationPMC‑LLaMAText
Named Entity RecognitionRaTE‑NERText
Retrieval-Augmented GenerationChatCAD‑RText

Visual Understanding

Visual Question AnsweringLLaVA‑MedMRI · CT · X-ray · Histology · Gross
ClassificationBiomedCLIP · CONCH · DSMIL‑C16 · DSMIL‑TCGAMRI · CT · X-ray · Histology · Gross

Image Processing

GroundingGrounding DINOMRI · CT · X-ray · Histology
SegmentationMedSAM · UltraSAM · IterNet · CellViT · CellSAMMRI · CT · X-ray · US · Fundus · Histology · Gross
Grounding-SegmentationGrounding DINO + MedSAMMRI · CT · X-ray · Histology
RegistrationUniGradICONCT · MRI
ReconstructionHealthGPTX-ray · MRI · CT · US
Super-ResolutionHealthGPTX-ray · MRI · CT · US

Medical Report Generation

Report GenerationLLaVA‑Rad (radiology) · SpecialistVLMs (retinal OCT)X-ray · MRI · CT · US · OCT
Sankey diagram linking the 12 tasks to functional categories and the 8 modalities
Composition of the tool library across tasks, functional categories, and modalities. Band width indicates the number of instruction-tuning samples.

Tool Library

18 specialist models, one interface

Every tool is exposed to the planner through a standardized interface describing its inputs, parameters, and outputs — so heterogeneous models can be invoked, chained, and swapped without retraining the backbone. The library holds 18 specialist models, each the state of the art for its task, exposed as 19 tools: Grounding DINO + MedSAM is a composed grounding-segmentation tool, and the HealthGPT tool serves both reconstruction and super-resolution.

Text Understanding & Generation

PMC-LLaMAQA & rationale generationText RaTE-NERNamed entity recognitionText ChatCAD-RRetrieval-augmented generationText

Visual Understanding

LLaVA-MedVisual question answeringAll modalities BiomedCLIPImage classificationAll modalities CONCHZero-shot classificationPathology

Image Processing

Grounding DINOObject detection / groundingMRI · CT · X-ray · Histology MedSAMGeneral segmentationAll modalities Grounding DINO + MedSAMGrounding-segmentationAll modalities UltraSAMSegmentationUltrasound IterNetRetinal vessel segmentationFundus CellViTWSI cell segmentationPathology CellSAM2D cell segmentationPathology DSMIL-C16Tumor detectionPathology DSMIL-TCGACancer subtypingPathology UniGradICONImage registrationCT · MRI HealthGPTReconstructionAll modalities HealthGPTSuper-resolutionAll modalities

Medical Report Generation

LLaVA-RadRadiology report generationX-ray · CT · MRI · US SpecialistVLMsOCT report generationRetinal OCT

Examples

Three workflow patterns, end to end

Single-tool multi-round dialogue: CT registration with UniGradICON
Single-tool · multi-roundRegistration of paired scans: the planner recognizes the paired-image structure and invokes UniGradICON to spatially align them.
Multi-tool multi-round dialogue: fundus classification with BiomedCLIP then vessel segmentation with IterNet
Multi-tool · multi-roundProgressive fundus workflow: BiomedCLIP identifies the modality, then IterNet generates a vessel segmentation mask on request.
Multi-tool single-round dialogue: ultrasound lesion segmentation with UltraSAM and interpretation with LLaVA-Rad
Multi-tool · single-roundCompositional ultrasound review: UltraSAM localizes the lesion and LLaVA-Rad interprets the segmented region in one turn.

Results

Learned orchestration holds up under evaluation

Tool orchestration 1,500 held-out dialogues, 500 per setting

Dialogue settingTool-call acc.Whole-case acc.Micro F1Macro F1
Single-tool multi-round91.0%91.0%0.9130.903
Multi-tool single-round80.7%77.1%0.8400.820
Multi-tool multi-round98.1%95.8%0.9810.979

Final response quality LLM-as-a-judge: faithfulness + completeness + conciseness, 3–15

Radar plots of final response quality by modality and by task
MMedAgent-Plus forms the outermost profile on every modality and nearly every task axis, with the largest margins on fundus and retinal OCT. Averaged over modalities it reaches 96.1% of the maximum score vs. 78.2% for the strongest baseline.

Against four direct-answer baselines — Qwen3-VL-8B, GPT-5.4-mini, MedGemma-1.5-4B-IT, and LLaVA-Med — MMedAgent-Plus attains the highest total score in all three dialogue settings: 14.77 vs. 10.89 (multi-tool single-round), 14.52 vs. 13.27 (single-tool multi-round), and 14.36 vs. 12.86 (multi-tool multi-round). The gains concentrate in faithfulness and completeness — exactly where grounding answers in specialist tool outputs should help.

In end-to-end execution on 350 held-out cases — no oracle tool outputs provided — the planner invokes the required specialist in 317/350 (90.6%) cases and retains specialist-level task performance: it matches PMC-LLaMA's 0.98 QA accuracy and closely tracks specialist metrics for RaTE-NER, LLaVA-Rad, and BiomedCLIP.

More analysis: per-tool routing and error structure
Tool orchestration performance: aggregate accuracy table and per-tool precision, recall, and F1 for the three dialogue settings
Tool orchestration across dialogue settings, 500 held-out dialogues each. (a) Aggregate accuracy with micro- and macro-averaged precision, recall, and F1. (b–d) Per-tool precision, recall, and F1 for single-tool multi-round, multi-tool multi-round, and multi-tool single-round dialogues; lower precision reflects unnecessary tool calls, lower recall reflects missed ones.
Tool-level and task-level recall with residual incorrect assignments
Tool- and task-level routing, pooled over 1,500 held-out dialogues. All but three tools exceed 0.90 recall and nine reach 1.00; the low-recall tail is UniGradICON (0.36), CONCH (0.64), and MedSAM (0.76). Errors follow two patterns: swaps between related tools (CONCH → LLaVA-Med, 36%), and omitted calls where the model answers directly (64% of UniGradICON and 24% of MedSAM cases routed to no tool).
End-to-end evaluation across seven tasks: judge scores, specialist invocation accuracy, and task-specific metrics
End-to-end execution on 350 held-out cases across seven tasks, with no tool outputs provided. (a, c) LLM-as-a-judge scores on the five baseline-comparable tasks: MMedAgent-Plus averages 4.48 of 5 vs. 2.82 for the strongest baseline. (b) Specialist invocation accuracy, 317/350 (90.6%). (d) Task-specific metrics, with the raw specialist output as the upper bound.

Citation

BibTeX

@article{mmedagentplus2026,
  title   = {MMedAgent-Plus: Scaling Multimodal Medical Agents
             across Modalities, Tasks and Workflows},
  author  = {Ruoxin Wang and Jinglong Xiong and Yuanting Pan and
             Xiaoquan Kong and Binxu Li and Ruiyang Ji and Zhe Xu and
             Yaqian Chen and Jiayuan Ding and Mingjie Li and Yu Zhang
             and Ehsan Adeli and Kilian Pohl and Zihao Lin and Haoyu
             Dong and Yixin Wang},
  journal = {TODO},
  year    = {2026}
}