Multimodal Perception
Representing the physical world
We study 3D geometry, multimodal fusion and scene representations, combining visual, geometric and other sensory information to understand objects and their surroundings.
Research & publicationsOMEGA LAB · HKUST(GZ)
OMEGA Lab
Omnimodal Multi-Embodiment
Generalist Agents
Our research connects multimodal perception, intent and motion modeling, and autonomous decision-making in physical systems and digital workflows.
Explore our researchTHE OMEGA PERSPECTIVE
Omnimodal
Modalities & representations
Multi-Embodiment
Land, water, air & robotic systems
Generalist Agents
World models, reasoning & decision-making
- Perceive
- Model
- Predict
- Reason
- Plan
- Act
Omnimodal Multi-Embodiment Generalist Agents
Select any number of modalities to highlight their matching orbits. Select again to deselect. Drag along the orbit to change its position; release to continue orbiting. Arrow keys also adjust position. Escape returns to the overview.
The three O, ME and GA sections fade, and their letters join to spell OMEGA, then condense into the lab’s Greek Ω logo. A nucleus and orbital paths then appear. After orbiting, the graphic unfolds back into the three sections and repeats. Each modality has its own tilted orbital plane around the central nucleus, with vision, language, geometry and tactile emphasized. The nine modalities and representations are: vision, language, geometry, tactile, audio, radio, proprioception, intent and trajectory. Inside the nucleus, ten kinds of embodiments form a ring around the central Ω: quadrupeds, UGVs, autonomous cars, USVs and UAVs; then dexterous hands, robot arms, bimanual systems, wheeled dual-arm robots and humanoids. Labels and tracks become smaller, softer and occluded as they pass behind the translucent nucleus. Multiple modalities can be selected independently; dragging pauses only the dragged modality until release. The Generalist Agents section forms a six-stage loop: Perceive, Model, Predict, Reason, Plan and Act. At the center, digital tasks and physical tasks connect through an agent and tool use. Software tools and control interfaces let agents act in digital workflows and physical environments, including autonomous navigation. Outward calls and returning results or observations form a shared feedback loop.
Wind-resilient UAV routing wins the IEEE RCAR 2026 Best Student Paper Award
Energy-Aware Wind-Resilient Routing for Truck-Assisted Multi-UAV Delivery under Wind Uncertainty, by Tianshun Li, Yanggang Sheng, Hongliang Lu, Zhongzhen Wang, Haoang Li and Xinhu Zheng.
HKUST(GZ)–Annu team places second in the Shenzhi Cup Embodied Intelligence and Robotics track
Led by Xinhu Zheng, the HKUST(GZ) team joined Annu to compete in the inaugural Shenzhi Cup AI Innovation Competition, also taking first place in material handling.
Synesthesia of Machines receives the 2026 IEEE ComSoc Best Survey Paper Award
The award recognizes Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines, co-authored by Xinhu Zheng and published in IEEE Communications Surveys & Tutorials.
OUR RESEARCH
Research directions
We study how agents perceive, model, reason and act, from robot policies and world models to tool use, multi-agent workflows and the safety of embodied systems.
Explore researchRobot Learning & Manipulation
Learning policies for action
We develop vision-language-action models that connect reasoning with manipulation, studying motion generation, policy improvement and efficient inference.
Research & publicationsWorld Models & Prediction
Modeling how the world changes
We investigate predictive world models and future motion, with a focus on how behavior, driving style and physical dynamics inform planning.
Research & publicationsVision-Language Navigation
Grounding language in space
We study how agents follow language instructions in physical environments, connecting spatial perception, motion prediction and planning for ground and aerial navigation.
Research & publicationsIntent & Decision-Making
Understanding behavior and decisions
We model intent and human driving behavior, and examine how visual evidence and context shape autonomous decisions under real-world constraints.
Research & publicationsMulti-Agent Systems
Coordinating agents and tools
We study how agents use specialized models and tools, share information and coordinate tasks, spanning digital workflows for analysis and generation as well as collaborative aerial and ground systems.
Research & publicationsEmbodied AI Safety
Stress-testing and hardening embodied agents
We probe how perception, planning and detection models fail under backdoor, adversarial and out-of-distribution threats, designing stealthy yet effective attacks to expose vulnerabilities and developing mitigation strategies for robust autonomous driving and multimodal systems.
Research & publicationsRESEARCH IN FOCUS
Selected research
Recent work by Xinhu Zheng and collaborators in 3D perception, multimodal learning and embodied intelligence.
View all publications
Multimodal Perception
UniT: Unified Geometry Learning with Group Autoregressive Transformer
UniT brings online perception and offline 3D reconstruction into one geometry model. Group autoregression accommodates different views and auxiliary inputs, while supporting metric-scale prediction and bounded memory over long sequences.
Robot Learning & Manipulation
Potential-Guided Flow Matching for Vision-Language-Action Policy Improvement
ForesightFlow learns from successful and failed robot experience. A single flow model generates action chunks and estimates their success potential, guiding action selection without a separate critic. The policy is evaluated on simulated and real-world bimanual tasks.
World Models & Prediction
PLAN-S: Bridging Planning with Latent Style Dynamics for Autonomous Driving World Models
PLAN-S decodes style-conditioned semantic cost maps from latent world models, making planning preferences inspectable before trajectory selection. The same bridge supports regression and anchor-score planners, evaluated on nuScenes and NAVSIM.
Vision-Language Navigation
P³Nav: End-to-End Perception, Prediction and Planning for Vision-and-Language Navigation
P³Nav brings perception, prediction and planning into one vision-and-language navigation model. Object and map cues describe the current scene; predicted waypoints and future scene features help the agent plan toward a language-specified goal.
Intent & Decision-Making
Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?
A structured visual-perturbation study examines how VLA driving decisions depend on camera input. Evaluations compare open-loop trajectory prediction with closed-loop driving behavior, revealing that visual grounding varies across perturbations and evaluation settings.
Multi-Agent Systems
Energy-Aware Wind-Resilient Routing for Truck-Assisted Multi-UAV Delivery under Wind Uncertainty
EWR plans truck-assisted UAV delivery routes with an energy graph updated by wind estimates and payload state. It continually checks return feasibility, aiming to preserve enough battery energy to reach the truck or depot as conditions change.
PEOPLE
Our team
Meet the principal investigator and explore our member directory.
View all people
Xinhu Zheng 郑心湖
Assistant Professor
The Hong Kong University of Science and Technology (Guangzhou)
Intelligent Transportation · Internet of Things
Xinhu Zheng studies multimodal perception, intent and trajectory prediction, and autonomous systems.
Hardware
Platforms, instruments, and experimental setups.
PROSPECTIVE STUDENTS & RESEARCHERS
Join OMEGA Lab
Bring your questions about multimodal intelligence and agents. Explore PhD, MPhil and postdoctoral pathways, university support, and how to get in touch.
Opportunities & applicationsGET IN TOUCH
Research &
collaboration
For research collaboration, prospective student enquiries and academic exchange, contact Xinhu Zheng.
xinhuzheng@hkust-gz.edu.cn
Office W2 L5 509
The Hong Kong University of Science and Technology (Guangzhou)