Temporal Dynamics in Human Speech and Gestures
Motion-centric video understanding and rigorous evaluation of human behavior models
Oct 2024 – Present · Signal Analysis and Interpretation Laboratory (SAIL), USC
Much of what makes human behavior meaningful lives in how people move over time, not in any single frame. This project builds video models that pay explicit attention to motion. It also examines how such models are evaluated in human-behavior research.
Highlights
- MOOSE: a motion-centric video encoder that uses optical flow to focus on temporal dynamics for fine-grained human activity understanding (Nguyen et al., 2025).
- Autism motor behavior: improved detection of atypical motor behaviors in autism by 6% accuracy, with enhanced saliency-based explainability.
- Auditing co-speech gesture evaluation: showed that learned judges exploit speaker and session shortcuts, reaching 0.92 accuracy with no true speech–gesture coupling. Proposed 3 controls for rigorous evaluation.
- Real-time action detection: CAKE, real-time action detection via motion distillation and background-aware contrastive learning (Hoang et al., 2026).
