Haoyu Cao
曹浩宇Senior Researcher / Team Lead · Tencent Youtu Lab
Haoyu Cao is a Senior Researcher and Team Lead at Tencent Youtu Lab. He leads research on multimodal understanding and agents, with a focus on visual perception, long-horizon reasoning, and computer and office automation.
His research spans multimodal foundation models, document intelligence, and natural human–AI interaction. Representative projects include Youtu-VL, the Youtu-Parsing series, Omni-Diffusion, Video-MME-v2, and the VITA family.
He received his Ph.D. from the University of Science and Technology of China (USTC), advised by Prof. Linli Xu (徐林莉). He was part of the team led by Prof. Enhong Chen (陈恩红) at the State Key Laboratory of Cognitive Intelligence.
Openings: Research Scientists, Engineers & Research Interns
We welcome research scientists, engineers, and interns interested in multimodal understanding, document intelligence, and computer-use or office agents. Please email a brief introduction and your CV.
Research & Open-Source Projects
Click any project figure to inspect high-res detailsMultimodal Agents
We study agents that turn visual understanding into actions across desktop applications, office workflows, and embodied environments.
Screen-level visual grounding, coordinate prediction, and multi-step OS/web task execution powered by unified vision-language supervision.
Deep integration of structured omni-modal parsing (tables, formulas, charts, layouts) with autonomous document editing, analysis, and workflow automation.
Transferring rich multimodal priors into executable action policies via action-expert distillation and reinforcement learning.
Multimodal Foundation Models & Document Intelligence
Vision–language models, high-parallelism document & omni-modal parsing, fine-grained grounding, and authoritative multimodal evaluation — from Youtu-VL and Youtu-Parsing to Video-MME-v2.
Youtu-VL
Vision–language models built on Vision–Language Unified Autoregressive Supervision (VLUAS), shifting optimization from "vision-as-input" to "vision-as-target" for both vision-centric and general multimodal tasks.
- • Unified supervision for visual and text tokens.
- • Supports dense visual perception, visual grounding, and general multimodal understanding.
Youtu-Parsing
A document parsing model combining token and query parallelism for efficient recognition and structuring of text, tables, formulas, and reading order.
- • The technical report reports 93.2 overall on OmniDocBench v1.5.
- • Token-parallel decoding reports 5–11× speedup in the evaluated settings.
Youtu-Parsing-Omni
A 5B omni-modal parser with dual Vision & Audio stems and Youtu-Omni-Encoder, mapping documents, natural images, charts, chemical structures (SMILES), sheet music (ABC), geometry, speech, and video into unified OmniSchema JSON.
- • 8 specialized parsing domains unified under a single 5B model and OmniSchema output format.
- • Supports Markdown/LaTeX/OTSL tables, SMILES chemical formulas, ABC musical notation, and temporal audio-video grounding.
Omni-Diffusion
A mask-based discrete diffusion model unifying understanding and generation across text, images, and speech.
- • Models a joint distribution over discrete multimodal tokens.
- • Supports any-to-any tasks, including spoken visual dialogue and speech-to-image generation.
Video-MME-v2
A video understanding benchmark organized around retrieval, temporal understanding, and complex reasoning, with group-based evaluation.
- • Progressive tri-level taxonomy across 40+ fine-grained tasks with up to 5 rounds of quality assurance.
- • Group-based non-linear evaluation penalizes guessing and exposes the gap between frontier models (49.4%) and human experts (90.7%).
Multimodal Interaction & Embodied Action
Open-source models for real-time duplex vision–speech interaction, 1M-token long-context multimodal dialogue, fast speech generation, and vision–language–action (VLA) distillation — centered on the VITA family.
VITA & VITA-1.5
An open-source omni-modal model family for near real-time vision–speech dialogue, with end-to-end speech interaction in VITA-1.5.
- • Joint visual, speech, and text understanding.
- • VITA-1.5 enables speech-to-speech dialogue without separate ASR/TTS modules.
VITA-Audio
Efficient end-to-end speech modeling with Multiple Cross-modal Token Prediction (MCTP) for fast streaming audio–text generation.
- • Generates multiple audio tokens per forward pass with minimal parameter overhead.
- • Achieves 3–5× inference speedup on streaming audio generation with high speech naturalness.
VITA-QinYu
An expressive spoken language model for natural conversation, role-playing, and singing through hybrid speech–text modeling.
- • Interleaved speech–text modeling with multi-codebook audio tokens.
- • Open-source code and model checkpoints with a streaming full-duplex demo.
Long-VITA
Scales open-source multimodal large language models to 1 Million tokens (4,096+ frames) via context-parallel distributed training while retaining strong short-context accuracy.
- • Supports up to 1M context length across images, video frames, audio, and text.
- • Trained exclusively on open-source datasets with sequence parallelism.
VITA-E & VITA-VLA
Embodied humanoid interaction (VITA-E) and action-expert distillation for vision–language–action models (VITA-VLA), bridging multimodal perception with real-world robotic manipulation.
- • VITA-VLA distills specialized action experts into VLM backbones without degrading general visual-language understanding.
- • VITA-E enables natural interruptible speech-vision interaction on humanoid robots.
Recent News
-
2026.10Released Youtu-Parsing-Omni, a 5B omni-modal structured parser covering documents, images, charts, chemistry, music, audio, and video. [Hugging Face]
-
2026.08Released preprints on on-policy self-distillation (VISTA), video instruction-following evaluation (Video-IFBench), and checklist-driven video reward modeling (FIRM-Video). [VISTA] [Video-IFBench] [FIRM-Video]
-
2026.05
-
2026.04
-
20264 papers accepted to top conferences in 2026: Omni-Diffusion to ICML 2026, DiG to CVPR 2026, When Thinking Hurts to ECCV 2026, and Multimodal Table Understanding to AAAI 2026.
-
2026.01Released Youtu-Parsing and Youtu-VL technical reports, model weights, and code. [Youtu-Parsing] [Youtu-VL]
-
2025VITA-1.5 accepted to NeurIPS 2025 as a Spotlight; VITA-Audio to NeurIPS 2025, BASIC to ICCV 2025, and DREAM to ACM MM 2025.
Selected Publications
Google Scholar-
Multimodal Table Understanding with Difficulty-Aware Reinforcement Learning
AAAI 2026 -
BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
ICCV 2025 -
Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction
ACL 2024 -
Turning a CLIP Model into a Scene Text Spotter
TPAMI & CVPR 2023 -
Query-driven Generative Network for Document Information Extraction in the Wild
Experience & Education
Tencent Youtu Lab
Leading the technical directions of Multimodal Understanding and Multimodal Agents, covering Computer-Use & Office Agents, Multimodal Foundation Models & Document Intelligence (Youtu-VL, Youtu-Parsing, Youtu-Parsing-Omni, Video-MME-v2), and Multimodal Interaction (VITA series).
University of Science and Technology of China (USTC)
Research on multimodal learning, visual document understanding, and sequence generation.