Speech Production on Real-time MRI Video

Video world models, diffusion synthesis, and multimodal encoders for real-time MRI of the vocal tract

MRI Voice Lab: articulator contours for lips, jaw, tongue, velum, and larynx traced on a real-time MRI frame
Click the image to open MRI Voice Lab, an interactive demo: drag the tongue, lips, jaw, velum, or larynx on a real-time MRI clip and hear a simplified vocal-tract model respond.

Sep 2023 – Present · Signal Analysis and Interpretation Laboratory (SAIL), USC

Real-time MRI (rtMRI) captures the full vocal tract in motion during speech: tongue, lips, velum, and larynx. This makes it a uniquely rich signal for studying how humans produce speech. It also matters for clinical speech applications. This project develops video models that learn the temporal dynamics of articulation directly from rtMRI.

Highlights

  • Arti-JEPA: a video world model adapted to rtMRI for speech-production analysis. It doubles cross-domain phoneme decoding over pretrained baselines and supports a range of downstream applications to clinical speech (Nguyen et al., 2026).
  • Speech2rtMRI: the first audio-guided diffusion model for real-time MRI synthesis of the vocal tract. Expert phoneticians misidentified up to 34% of the generated samples as authentic (Nguyen et al., 2025).
  • Interpretable articulatory modeling: a multimodal video encoder for rtMRI that fuses optical flow with 6-channel region-of-interest inputs, reducing phoneme recognition error rate by 5% (Park et al., 2026).
  • Articulation vs. acoustics: disentangling the contributions of articulatory and acoustic signals in multimodal phoneme recognition (Foley et al., 2025).

References

2026

  1. arXiv
    Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis
    Hong Nguyen, Sean Foley, Christina Hagedorn, and 4 more authors
    arXiv preprint arXiv:2609.09757, 2026
    arXiv PDF
    N/A Google Scholar citations
  2. ICASSP
    Interpretable Modeling of Articulatory Temporal Dynamics from Real-Time MRI for Phoneme Recognition
    Jay Park, Hong Nguyen, Sean Foley, and 4 more authors
    In ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026
    arXiv HTML
    N/A Google Scholar citations

2025

  1. ICASSP
    Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech
    Hong Nguyen, Sean Foley, Kevin Huang, and 3 more authors
    In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025
    arXiv HTML
    N/A Google Scholar citations
  2. arXiv
    Towards Disentangling the Contributions of Articulation and Acoustics in Multimodal Phoneme Recognition
    Sean Foley, Hong Nguyen, Jihwan Lee, and 4 more authors
    arXiv preprint arXiv:2505.24059, 2025
    arXiv
    N/A Google Scholar citations