Anil Palepu and Mike Schaekermann on AMIE (Video): When Clinical AI Leaves Text Behind

The constraint that text could not solve

When a physician meets a patient, the consultation extends far beyond the words exchanged. The physician observes gait, registers visible signs of discomfort, notes breathing, and guides the patient through physical examination maneuvers. AI systems capable of clinical reasoning and dialogue have the potential to dramatically increase access to medical expertise—but text-based interfaces discard the visual and auditory dimensions of clinical practice.

That is the problem Anil Palepu, Senior Research Scientist, and Mike Schaekermann, Research Lead at Google Research, set out to address with AMIE (Video), announced on August 11, 2026.

Why a single agent was not enough

Conducting an effective clinical conversation over video requires balancing competing demands: the system must respond at natural conversational speed while simultaneously performing careful clinical reasoning and continuously processing visual and auditory streams.

Palepu and Schaekermann's team concluded that a single agent cannot satisfy all these requirements. Deep reasoning takes time, but conversational pauses erode patient trust and rapport.

Their solution is an asynchronous multi-agent architecture dividing labor across three specialized agents working continuously in parallel:

• Talker agent — drives responsive, low-latency spoken interaction while incorporating guidance from the other agents
• Planner agent — continuously refines clinical reasoning, updating differential diagnoses and management plans in the background
• Perception agent — reviews audio and visual streams, identifying clinically relevant non-verbal cues such as visible distress, physical findings, or auditory signals

This decoupled design allows AMIE (Video) to maintain natural conversational latency while performing diagnostic reasoning and audio-visual perception that would otherwise introduce unacceptable delays.

Built on Gemini and Project Astra

AMIE (Video) is built on Gemini and Project Astra. It conducts synchronous clinical video consultations, perceiving non-verbal clinical cues, guiding patient actors through virtual physical examinations, and reasoning diagnostically—all in real time.

The researchers note that their earlier work demonstrated expert-level performance in text-based diagnostic dialogue. AMIE (Video) extends those capabilities toward the perceptual richness of telehealth practice.

What the randomized study measured

To evaluate clinical competence in end-to-end audio-visual consultation, the team conducted a large-scale randomized Objective Structured Clinical Examination (OSCE) study with a synchronous video consultation interface.

The study spanned 100 clinical scenarios covering five body systems—cardiopulmonary, abdominal, head/eyes/ears/nose/throat (HEENT), neurological/psychiatric, and musculoskeletal conditions. Fifteen trained patient actors carried out 300 standardized consultations across three study arms:

• AMIE (Video) conducting real-time video consultations
• AMIE (Text) as a baseline to isolate audio-visual contributions
• Ten board-certified primary care physicians (PCPs) consulting via the same video interface

An independent panel of 20 experienced primary care physicians evaluated all consultations using established clinical rubrics, including general competency scales and detailed case-specific scoring criteria tailored to each scenario.

Key findings

Across core clinical competencies—history-taking thoroughness, diagnostic accuracy, management appropriateness, and communication quality—clinical evaluators rated AMIE (Video) on par with PCPs. AMIE (Video) also matched or exceeded AMIE (Text) on these dimensions.

AMIE (Video) was rated significantly higher, on average, than both PCPs and AMIE (Text) at eliciting physical signs and proactively guiding patient actors through virtual examination maneuvers.

Patient actors strongly preferred the synchronous video interface over text-based chat, rating it as significantly easier to use and more effective for communicating health concerns. They also rated AMIE (Video) favorably on empathy, rapport, and confidence in care compared to both PCPs and AMIE (Text).

Limitations they emphasize

Palepu and Schaekermann are explicit about what this study does not prove. The research was conducted entirely with professional patient actors in simulated clinical settings—not with real patients presenting with their own health conditions.

Patient actors, however skilled, cannot fully replicate the complexity and unpredictability of real clinical encounters. The scenarios were limited to conditions that can be authentically portrayed through acting, omitting important clinical presentations where audio-visual perception would be diagnostically consequential.

Targeted automated evaluations revealed occasional perceptual and reasoning errors, despite overall high-quality conversation and diagnostic accuracy. The system still exhibits intermittent technical issues that can disrupt conversational naturalness.

Assessing these findings in studies with real patients and real clinical conditions is an essential next step before any conclusions about real-world utility can be drawn.

The path toward real-world evidence

The researchers note they have already taken early steps toward responsible real-world evidence: a feasibility study with Beth Israel Deaconess Medical Center provided initial evidence for the safety and utility of text-based AMIE in clinical practice, and an ongoing nationwide randomized study with Included Health is further evaluating AI in real-world virtual care.

Together, these research experiences will help inform how audio-visual capabilities might be responsibly integrated into clinical practice.

While much remains to be done, the August 11 results mark an important milestone toward AI systems that could one day augment care by engaging with the sensory complexity of clinical practice.

Sources

• Google Research — Advancing AMIE towards expert-level audio-visual clinical consultations (August 11, 2026)