GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Manling Li
  • MLL-Lab

This view

  • Original page
  • New address

GeezerVison is showing limanling.github.io — Manling Li | Multimodal and Embodied Reasoning | Northwestern University

Manling Li, Assistant Professor of Computer Science at Northwestern University

Manling Li

Assistant Professor
Northwestern University

Computer Science

Center for Robotics and Biosystems (Affiliated)

Amazon Scholar

Amazon

manling.li AT northwestern.edu

Office Tech M259

Hi, there! I’m Manling. I am an assistant professor of Computer Science at Northwestern University, affiliated with Center for Robotics and Biosystems and Cognitive Science. I direct the Multimodal Learning with Language (MLL) Lab. I am also an Amazon Scholar working on conversational agents. Prior to this, I was a postdoc at Stanford University (2024), mainly working with the advisor Jiajun Wu, and Fei-Fei Li. I obtained my PhD at UIUC (2023), advised by Heng Ji, closely working with Shih-Fu Chang, Kyunghyun Cho and Jiawei Han. I am a recipient of MIT Tech Review Innovators Under 35 in 2025, ACL Inaugural Best Dissertation Award Honorable Mention in 2025, DARPA Riser in 2022, and a EECS Rising Star in 2022, Microsoft Research PhD Fellowship in 2021, etc. Our work on multimodal reasoning was recognized as ACL'24 Outstanding Paper Award, ACL'20 Best Demo Paper Award, NAACL'21 Best Demo Paper Award, etc. I served as organizing committee of ACL'25 (virtual chairs), NAACL'25 (publication chairs), EMNLP'24 (demo chairs), etc.

I aim to evolve AI models from passive observers into active, embodied agents. Today's foundation models, trained on static data, rich in semantic priors, yet lack grounded interactions with the dynamic and partially observable physical world. Reasoning ≠ Memorization. We work on Reasoning and Planning at the intersection of Language, Vision, and Robotics, to equip foundation models with an internal physics engine. Recently, we have been especially interested in agentic robotics, where foundation-model agents write and revise programs that act on robots (Embodied First Exam), and in continual learning for vision, where vision-language agents keep learning from their own interactions with the world (Planning with the Views, VAGEN).

    Reasoning: Multimodality (Language + Vision/Robotics).
    → show/hide work on moving from semantic-centric priors to spatial-geometric reasoning.     - Spatial Reasoning: MindCube, AdaptVis, Vector Graphics Reasoning, LayoutVLM.
    - World Modeling: VAGEN within RL, Self-Play Agent before RL, Embodied Agent Interface with BDDL transition models
    - Long-Horizon multimodal intelligence: T* for temporal search, HourVideo, LM4Video, VideoArgument, VideoEvent,
    - Multimodal alignment: CLIP-Event, VisualDecomposition, Knowledge-Driven Vision-Language Pretraining, VisualKnowledge, MuMuQA, M2E2, Image2Code.
    Planning: Exploration vs Exploitation, Foundation Agent.
    → show/hide work on self-evolving agent training with reinforcement learning, from exploitation to exploration via world modeling.     - Foundation Agent Training via Reinforcement Learning: LLM Agents (RAGEN, Self-Play Agent, FedAgent), VLM Agents (VAGEN with World Model Reinforcement Learning; RL for Vision)
    - Embodied Agents: Embodied Reasoning Agent for reinforcement learning training, Embodied Agent Interface for LLM agents, EmbodiedBench for VLM agents, ROSETTA with human feedback, IKEA Manual at Work for 4D grounded planning.
    - Planning and Compositionality: Planning Logic, Procedural Planning, Agriculture Task Planning, LLM Schema, Graph Schema, Generative Graph Schema, PathLM, Decomposition to Latent Text Prompt.
    Controlling and Safety: Mechanistic Interpretability, Foundation Model + Control Theory.
    → show/hide work on turning black boxes to safe, transparent, and trustworthy agents.     Safety - Open up models to understand inner workings: Why is Spatial Reasoning Hard for VLMs?, Reasoning/Perception Merging, Exploring Diffusion Transformer Designs via Grafting.
    - Physics of LLMs/VLMs: Knowledge Overshadow, Ripple Effect.
    - Control and intervene foundation models: LM-Steer, HallE-Control, Deep Concept Injection. Truthfulness - Factuality and hallucination: SmartBook, Explainable Fact Checking, Object Hallucination.
    - AI for Science, claim feasibility verification: COVID-19 Claim Radar, COVID Knowledge Graph. Fairness and Social Good - NLP + Human/Social (theory of mind, debiasing, propaganda, information extraction): Timeline Summarization, Meeting Summarization, GAIA IE, Info Propagation Pattern, DebiasPrompt, Multilingual KG.

Prospective students: I have several PhD positions in Fall 2027. » Show details

>> We have amazing new faculty at NU working on robotics and foundation models, please check out Ruohan! He also has several PhD positions.
>> If you are interested in PhD at Northwestern, please apply to NorthwesternCS. Due to the large amount of emails, I apologize that I will not be able to reply to individual emails (Please note that non-reply does NOT indicate non-interesting, largely means emails got missed or I unfortunately did not get time to check such emails before applications). Please choose me as a potential advisor in the application, and I will check every application carefully in late Dec and do interviews in Jan-Mar.
>> Proud of what our students have achieved in the very first year!

Invited Talks

Failure Modes in Agentic AI slides Failure Modes in Agentic AI RSI for Embodied Agents slides RSI for Embodied Agents World Model Reasoning for Open-World Agents slides World Model Reasoning for Open-World Agents Continual Learning for Vision slides Continual Learning for Vision Toward Controllable Foundation Agents slides Toward Controllable Foundation Agents Multimodal Reasoning Intelligence slides Multimodal Reasoning Intelligence Foundation Models Meet Embodied Agents slides Foundation Models Meet Embodied Agents From Large Language Models to Large Agent Models slides From Large Language Models to Large Agent Models
  • [2027/02] Talk at SIAM Conference on Computational Science and Engineering (CSE27) minisymposium on AI for Science.
  • [2026/12] Keynote at NeurIPS 2026 Workshop on Test-Time Continual Learning Agents.
  • [2026/12] Keynote at NeurIPS 2026 Workshop on Grounded and Embodied AI for Real-World Deployment (VLM4RWD).
  • [2026/10] Talk at IROS 2026 Workshop on Creativity in Assembly on "Foundation Models for Embodied Reasoning". [Slides]
  • [2026/10] Talk at EMNLP 2026 Workshop on Research on Agent Language Models (REALM).
  • [2026/10] Tutorial at EMNLP 2026 on Failure Modes in Agentic AI: From Reproducible Triggers and Diagnostics to Reliable Fixes [Website/Slides].
  • [2026/10] Keynote at Agentic AI for Earth Observation Workshop (BIFOLD / TU Berlin, with ESA) on "Failure Mode in Agentic Reasoning".
  • [2026/10] Talk at CSCW 2026 SIG on Rethinking Social Computing Research in the Age of AI.
  • [2026/10] Talk at Efficient Reasoning Workshop @ COLM 2026.
  • [2026/11] Talk at Northwestern University CogSci 110 on "Agent AI and Cognitive Science".
  • [2026/10] Panelist at SuperDebate Chicago Debate Showcase 2026 on AI agents and human oversight.
  • [2026/09] Talk at DoorDash on "Self-Improving Embodied Agents".
  • [2026/09] Talk at World Labs. [Slides]
  • [2026/09] Keynote at Amazon-Illinois Center on AI for Interactive Conversational Experiences Fall Research Symposium 2026.
  • [2026/08] Talk at Multi-Modal and World Model Symposium 2026 on "RSI for VLM Agents".
  • [2026/08] Talk at Ryan Institute on Complexity Summer Summit, Northwestern University, on "Failure Patterns in LLM Agents".
  • [2026/07] Talk at Institute of Foundation Models (IFM) on "World Model Reasoning for Open-World Agents".
  • [2026/07] Panelist at ACL 2026 Mentorship Program Panel on "How to Find NLP Problems That Survive 2030?".
  • [2026/06] Talk at Apple Workshop on Reasoning and Planning 2026 on "Failure Patterns of Agentic Reasoning".
  • [2026/06] Talk at Chicago AI Week Summit on "Foundation Agents: The Future of Multimodal and Embodied AI".
  • [2026/06] Talk at International Workshop on Symbolic-Neural Learning (SNL) 2026 on "Multimodal Reasoning Intelligence: How Foundation Models Learn (and Fail to Learn) Physical World".
  • [2026/06] Tutorial at CVPR 2026 on Towards Safe Multi-Modal Learning: Evolving Threats and Safety Solutions.
  • [2026/06] Talk at CVPR 2026 5th Workshop on Computer Vision in the Wild (CVinW), with panel discussion.
  • [2026/06] Talk at CVPR 2026 Workshop on Rediscovering Intelligence: Can AI Still Learn from Humans? (ReLearn), with panel discussion.
  • [2026/06] Talk at CVPR 2026 Workshop on Agentic AI for Visual Media (AAVM) on "Failure Modes of VLM Agents: A Reinforcement Learning Perspective".
  • [2026/06] Talk at CVPR 2026 Workshop on Test-Time Scaling for Computer Vision (ViSCALE) on "Spatial Mental Modeling from Limited Views".
  • [2026/06] Talk at CVPR 2026 Workshop on Any-to-Any Multimodal Learning (A2A-MML).
  • [2026/06] Talk at CVPR 2026 Workshop on Synthetic Data for Computer Vision (SynData4CV) on "Theory of Space: Evaluating Active Spatial Belief Construction in Foundation Models with Synthetic 3D Environments".
  • [2026/06] Talk at CVPR 2026 Workshop on Long-Form Video Understanding (LOVEU) on "World Models for Long-Form Video".
  • [2026/06] Panelist at CVPR 2026 Workshop on Embodied Reasoning in Action (ERA) on "Can Foundation Models Reason Their Way to Reliable Robots?", with Bernadette Bucher, Homanga Bharadhwaj, and Valts Blukis.
  • [2026/06] Panelist at ACSIC Silicon Valley Open Forum on Computing (SOFC) 2026 on "Collaboration between Industry and Academia".
  • [2026/05] Keynote at Jump Trading AI Research Symposium in New York, NY on "Foundation Agent Learning".
  • [2026/05] Talk at AgentSkills Workshop @ ACM CAIS 2026 in San Jose, CA.
  • [2026/05] Panelist at the Lambert ANN / SONIC / NICO Workshop, Northwestern University, closing panel "Charting a Path Forward".
  • [2026/04] Talk at UC San Diego Halıcıoğlu Data Science Institute (HDSI) on "Safe Agentic Reasoning".
  • [2026/04] Panelist at Midwest Speech and Language Days (MSLD) 2026, with Dan Roth, Ruhi Sarikaya, Karen Livescu, and Heng Ji.
  • [2026/04] Talk at Frontiers in Online Reinforcement Learning (IMSI) on "Failure Patterns of LLM Agentic Reinforcement Learning".
  • [2026/04] Highlighted Talk at Symposium on Machine Learning across Modalities at Yale University on "Multimodal Reasoning Intelligence".
  • [2026/04] Talk at Harvard University on "How Multimodal Models Learn (and Fail to Learn) the Physical World".
  • [2026/04] Talk at Northwestern Center for Human-Computer Interaction + Design (HCI+D) on "Towards Controllable Foundation Agents".
  • [2026/04] Guest lecture at Yale University.
  • [2026/03] Interview by MIT Tech Review China on Conversation with Manling Li: Rethinking Skill After OpenClaw [Article].
  • [2026/03] Talk at Carnegie Mellon University on "Toward Foundation Agents: How Multimodal Models Learn (and Fail to Learn) the Physical World".
  • [2026/03] Keynote at Texas A&M Connected Intelligence Initiative.
  • [2026/02] Talk at University of Illinois Urbana-Champaign on "Mechanistic Science of Multimodal Models".
  • [2026/02] Talk at Georgia Institute of Technology on "Toward Foundation Agents: How Multimodal Models Learn (and Fail to Learn) the Physical World".
  • [2026/02] Talk at University of Massachusetts Lowell Computer Science Colloquium on "From Large Language Models to Large Agent Models: Reasoning with the World".
  • [2026/01] Interview by MIT Tech Review China on Everyone a programmer? Experts on AI programming | 2026 Breakthrough Tech Top 10 [Article].
  • [2025/12] Talk at University of Washington on "Why does RL fail when interacting with OOD environments?".
  • [2025/12] Talk at Allen Institute for AI (Ai2) on "Towards Spatial Intelligence for Embodied Foundation Agents".
  • [2025/12] Talk at NeurIPS 2025 Workshop on Language, Agents, and World Models on "Spatial Mental Models from Limited Views", received Best Paper Honorable Mention.
  • [2025/12] Talk at Microsoft Research on "Bring Spatial Intelligence to Foundation Embodied Agents".
  • [2025/11] Talk at Northwestern University Cognitive Science Seminar on "Spatial Cognition in Foundation Models".
  • [2025/11] Talk at ICCV 2025 Best of ICCV Selected Talks on "Spatial Mental Models from Limited Views".
  • [2025/10] Keynote at ACM Multimedia 2025 Workshop on Multimodal Foundation Models for Spatial Intelligence (MFMFSI) on "Why is Spatial Understanding Hard for VLMs?".
  • [2025/10] Talk at Columbia University NLP Seminar on From Large Language Models to Large Agent Models: Reasoning Interface with World Modeling.
  • [2025/10] Talk at University of Pennsylvania NLP Seminar on "From Large Language Models to Large Agent Models: Reasoning Interface with World Modeling".
  • [2025/10] Talk at Capital One on "Conversational Agent Training".
  • [2025/10] Talk at Amazon on "Conversational Agent Training".
  • [2025/10] Interview by MIT Tech Review on Deepseek-OCR [Article].
  • [2025/10] Invited by Dreamforce 2025 as a panelist on AI Research Faculty Panel.
  • [2025/10] Tutorial at ICCV 2025 on Foundation Models Meet Embodied Agents.
  • [2025/10] Tutorial at ICCV 2025 on Safe Multi-Modal Learning.
  • [2025/10] Talk at ICCV 2025 Workshop on Multimodal Reasoning for Agentic Intelligence (MMRAgI).
  • [2025/10] Talk at ICCV 2025 Workshop on Multimodal Spatial Intelligence (MUSI).
  • [2025/10] Talk at ICCV 2025 Workshop on Memory and Vision (MemVis).
  • [2025/10] Talk at ICCV 2025 Workshop on Long Multi-Scene Video Foundations (LongVid-Foundations) on "Re-thinking Temporal Search for Long-Form Video Understanding".
  • [2025/10] Talk at ICCV 2025 Workshop on Structural Priors for Vision (SP4V) for the Best Paper Award on MindCube: Spatial Mental Modeling from Limited Views.
  • [2025/10] Panelist at ICCV 2025 KnowledgeMR Workshop on "Bottlenecks, Breakthroughs, and Bold Bets in Multimodal Reasoning".
  • [2025/10] Talk at Adobe Research on "Training Agents by Reinforcing Reasoning".
  • [2025/09] Talk at University of Edinburgh CHAI Seminar Series on "RAGEN: Training Agents by Reinforcing Reasoning".
  • [2025/09] Talk at Stanford University on "Theory of Space: How LLMs develop spatial internal beliefs".
  • [2025/09] Talk at Agentic AI Frontier Seminar on "RAGEN: Training Agents by Reinforcing Reasoning".
  • [2025/09] Guest lecture at Northwestern MSAI 431 on "LLM and Large Agents".
  • [2025/08] Talk at Agent AI Summit at UC Berkeley on "RAGEN: Training Agents by Reinforcing Reasoning".
  • [2025/08] Talk at Stanford University on "LLM Alignment with Control Barrier Functions".
  • [2025/07] Talk at Cross Future AI Summit on "See, Think, Act: Agent Training By Reinforcement Reasoning".
  • [2025/07] Talk at CVPR 2025 Workshop on Visual Concepts on "Why is Spatial Concept Learning Hard?".
  • [2025/07] Talk at Apple Workshop on Reasoning and Planning on "Training Agents with World Model Reasoning".
  • [2025/07] Talk at ACC 2025 Workshop on LLMs in Control Design and Decision Making on "LLMs for Embodied Decision Making".
  • [2025/06] Talk at Midwest Machine Learning Symposium 2025 on "RAGEN: Training Agents by Reinforcing Reasoning".
  • [2025/05] Talk at Google DeepMind on "RAGEN: Training Agents by Reinforcing Reasoning".
  • [2025/05] Talk at NAACL 2025 Workshop onKnowledge-Augmentation for Language Models and NLP Methods on "RAGEN: Training Agents by Reinforcing Reasoning".
  • [2025/05] Tutorial at NAACL 2025 on Foundation Models Meet Embodied Agents.
  • [2025/04] Talk at UIUC NLP Seminar on "RAGEN: Training Agents by Reinforcing Reasoning".
  • [2025/04] Guest Lecture at University of Michigan EECS 692 Advanced Artificial Intelligence on "Reasoning and Planning with Physical World".
  • [2025/02] Talk at AAAI 2025 New Faculty Highlights on "Agent Training Under a MDP Formulation".
  • [2025/02] Talk at AAAI 2025 Workshop on LM4Plan on "Agent Training Under a MDP Formulation".
  • [2025/02] Talk at AAAI 2025 Bridge on Foundation Models and Planning on "Agent Training Under a MDP Formulation".
  • [2025/02] Tutorial at AAAI 2025 on Foundation Models Meet Embodied Agents [Website/Slides].
  • [2025/02] Tutorial at AAAI 2025 on Lifecycle of Knowledge in LLMs: Memorization, Editing, and Beyond [Website/Slides].
  • [2024/12] Talk at SFU @ NeurIPS 2024 on "Embodied Agent Interface: LLMs and VLMs for Embodied Reasoning and Planning".
  • [2024/11] Talk at EMNLP 2024 Birds of a Feather on "LLMs for Embodied Agents".
  • [2024/11] Talk at EMNLP 2024 CustomNLP4U Workshop on "Customizing Large Language Models to Embodied Agents".
  • [2024/10] Talk at Adobe Research on "Chart Reasoning Agents".
  • [2024/10] Talk at University of Illinois Chicago Business School Seminar.
  • [2024/09] Keynote at Amazon-Illinois Center on AI for Interactive Conversational Experiences Fall Research Symposium 2024 on "From Large Language Models to Large Agent Models".
  • [2024/09] Talk at 2024 Allerton Conference on Communication, Control, and Computing on "Reasoning and Planning with Physical World Knowledge".
  • [2024/08] Talk at TTIC Multimodal AI Workshop 2024 on "Embodied Agent Interface: LLMs for Embodied Decision Making".
  • [2024/08] Talk at Summer Institute in Computational Social Science 2024 on "Multimodal Knowledge for Social Good".
  • [2024/08] Tutorial at IJCAI 2024 on Beyond Human Creativity: A Tutorial on Advancements in AI Generated Content (AIGC) [Website/Slides].
  • [2024/07] Talk at SpLU-RoboNLP 2024 Workshop on "Reasoning, Planning and Compositionality in Multimodality".
  • [2024/07] Talk at Adobe Research on "Visually Descriptive Language Modeling for Document Intelligence".
  • [2024/06] Talk at Apple NLU Workshop 2024 on "From Large Language Models to Large Agent Models".
  • [2024/06] Talk at UIUC NLP Seminar on "From Words to Worlds: A Close Look to Diffusion Models (through an NLP Lens)".
  • [2024/05] Talk at Midwest Machine Learning Symposium 2024 on "The Missing Knowledge in LLMs to Interact with the Physical World".
  • [2023/12] Keynote at NeurIPS 2023 Workshop on New Frontiers in Graph Learning on "Beyond the Beaten Path: Exploring the Role of Graphs in Multimodal Foundation Models".
  • [2023/11] Tutorial at ICAIF 2023 on Large Language Models for NLP in Finance [Slides].
  • [2023/10] Talk at Stanford Vision and Learning Seminar on "LLMs for robotics: Modeling the Knowledge of the Physical World".
  • [2023/10] Talk at Adobe Research on "Knowledge Foundation Models".
  • [2023/06] Talk at Stanford CogAI on "Modeling the Semantics of the Physical World".
  • [2023/06] Tutorial at CVPR 2023 on Knowledge-Driven Vision-Language Encoding [Website/Slides/Videos] [Reading List].
  • [2023/04] Talk "Towards Factuality in Information Access: Multimodal Knowledge Acquisition and Reasoning" at UCLA ECE.
  • [2023/03] Talk "Towards Factuality in Information Access: Multimodal Knowledge Acquisition and Reasoning" at University of Virginia CS; MBZUAI; Washington University in St. Louis CS; University of Toronto CS+ECE; UC Davis CS.
  • [2023/02] Talk "Towards Factuality in Information Access: Multimodal Knowledge Acquisition and Reasoning" at Carnegie Mellon University LTI; Northwestern University CS; Northeastern University ECE; Purdue University CS; Rice University CS; Virginia Tech CS; Max Planck Institute; UC San Diego ECE.
  • [2023/02] Tutorial at AAAI 2023 on Knowledge-Driven Vision-Language Pretraining [Website/Slides/Videos] [Reading List].
  • [2022/10] Talk "From Entity-Centric to Event-Centric Multimodal Event Knowledge Acquisition" at EECS Rising Stars, University of Texas at Austin.
  • [2022/10] Talk "Towards Accurate Intelligent Analysis: Event-Centric Multimedia Knowledge Extraction" at DARPA Forward (Invite-Only).
  • [2022/10] Talk "Event-Centric Multimedia Data Understanding" at Ohio State University; Singapore Management University; George Mason University; North Carolina State University.
  • [2022/09] Talk "Multimedia Event Extraction: From Object-Centric to Event-Centric" at Virginia Tech.
  • [2022/08] Talk "Event Knowledge Graph Construction" at LOGS Graph Reasoning Seminar.
  • [2022/07] Tutorial at NAACL 2022 on New Frontiers of Information Extraction [Website/Slides/Videos] [PDF].
  • [2022/06] Talk "Event Graph Structures in Vision-Language Understanding" at DataFun.
  • [2022/04] Talk "Connecting Vision and Text using Event Structures" at NewsBreak.
  • [2022/02] Talk "Memories as Repositories of Events: Structural Event Knowledge Acquisition" at University of Notre Dame.
  • [2021/12] Talk "Comprehensive Event Understanding in Multimedia Data" at USC ISI.
  • [2021/11] Talk "Structural Event Knowledge Acquisition from Multimedia Data" at UIUC NLP Seminar.
  • [2021/11] Talk "Event Extraction and Reasoning in Multimedia News Data" at Microsoft Research.
  • [2021/08] Talk "Improving Visual Event and Argument Role Understanding with Contrastive Image-Language Pretraining" at Microsoft Research.
  • [2021/08] Tutorial at ACL 2021 on Event-centric Natural Language Understanding [Website/Slides/Videos] [PDF].
  • [2021/02] Tutorial at AAAI 2021 on Event-centric Natural Language Processing [Website/Slides/Videos] [PDF].
  • [2020/10] Talk "Fine-Grained Knowledge Extraction System from Multimedia Data" at ai.science.
  • [2020/05] Talk "Event Understanding and Narration for Multimedia Data" at Intel MDI Research Lab.

We have a community effort on Foundation Models meet Embodied Agents

  • Tutorials: LLMs, VLMs, and VLAs, we give a comprehensive overview of the research based on Markov Decision Process (MDP) framework.
  • Challenges: Co-hosted with BEHAVIOR Challenge at NeurIPS 2025.
  • Workshops: we gather people from different communities including NLP, Vision, and Robotics to discuss the future of foundation models and embodied agents.

Publications

Selected All Agentic Reasoning Embodied Spatial Reasoning Safe Reasoning Tutorials

Selected Publications


* Equal contribution  ·  + Equal advising

Embodied First Exam teaser

Embodied First Exam [Website][Blog][Code][Video][tl;dr]
Kangrui Wang*, Qineng Wang*, Changhe Chen*, Pingyue Zhang*, Chi Gui*, Feishi Wang, Canyu Chen, Wenli Xiao, Xiaoyu Huang, Tonghe Zhang, Linjie Li, Zihan Wang, Minyeong Kim, Yiyou Sun, Mingkai Deng, Jinyu Hou, Eric P. Xing, Han Liu, Qi Zhu, Huaxiu Yao, Zhiwen Fan, Yuejiang Liu, Dawn Song, Pieter Abbeel, Li Fei-Fei, Jiajun Wu, Yuke Zhu, Guanya Shi+, Ruohan Zhang+, Manling Li+

Are Vision-Language Agents Latency Aware? teaser

LAGEN: Are Vision-Language Agents Latency Aware? [Website][PDF][Code][Models][Data]
Talha Chafekar*, Zihan Wang*, Zeju Li*, Xinyuan Li*, Qineng Wang, Jiajun Wu, Yuke Zhu, Yi Dong, Zhiding Yu, Ruohan Zhang, Manling Li

Planning with the Views teaser

Planning with the Views [Website][PDF][Blog][Code][Data][Models][tl;dr]
Kangrui Wang, Linjie Li, Zhengyuan Yang, Shiqi Chen, Zihan Wang, Li Fei-Fei, Jiajun Wu, Leonidas Guibas, Lijuan Wang, Manling Li

RAGEN-2: Reasoning Collapse in Agentic Reinforcement Learning teaser

RAGEN-2: Reasoning Collapse in Agentic Reinforcement Learning [Website][PDF][Blog][Code][Experimental Logs][tl;dr]
Zenus Wang*, Chi Gui*, Xing Jin*, Qineng Wang*, Licheng Liu*, Kangrui Wang, Shiqi Chen, Linjie Li, Zhengyuan Yang, Pingyue Zhang, Yiping Lu, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, Manling Li
ICML 2026 (Oral, Top 0.7%)
Best Paper Award at CVPR 2026 Workshop on Multimodal Reasoning for Agentic Intelligence (MMRAgI)
20 Papers That Matter at ICML 2026, selected by Nebius Science
Best Poster Award at MMLS 2025 (Midwest Machine Learning Symposium)
2.7k+ Github Stars, Featured by MIT Tech Review, Lambda Partner Spotlight, Medium, AI News, VentureBeat, MarkTechPost, Business Leaders Review, etc
Invited Talk at GenAI Summit 2025, AI Agent Summit at UC Berkeley 2025, Cross Future AI Summit 2025, UIUC NLP Seminar, Google Deepmind, etc.

VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents teaser

VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents [Website][PDF][Blog][Code][tl;dr]
Kangrui Wang*, Pingyue Zhang*, Zihan Wang*, Yaning Gao*, Linjie Li*, Qineng Wang, Chi Wan, Hanyang Chen, Yiping Lu, Zhengyuan Yang, Lijuan Wang, Ranjay Krishna, Jiajun Wu, Li Fei-Fei, Yejin Choi, Manling Li
NeurIPS 2025
Featured by MIT Tech Review, Stanford AI Blog

ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment teaser

ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment [PDF][Website]
Hongjue Zhao*, Haosen Sun*, Jiangtao Kong, Xiaochang Li, Qineng Wang, Liwei Jiang, Qi Zhu, Tarek F. Abdelzaher, Yejin Choi, Manling Li+, Huajie Shao+
ICLR 2026

Federated Agent Reinforcement Learning teaser

Federated Agent Reinforcement Learning [Website][PDF]
Canyu Chen, Kangyu Zhu, Zhaorun Chen, Zhanhui Zhou, Shizhe Diao, Yiping Lu, Tian Li, Manling Li+, Dawn Song+
NeurIPS 2026
Best Paper Award at AAAI 2026 Workshop on Trustworthy Agentic Systems
Outstanding Paper Award at AAAI 2026 Workshop on Personalization in the Era of Large Foundation Models

Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration? teaser

Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration? [Website][PDF][Data][Code][tl;dr]
Pingyue Zhang*, Zihan Huang*, Yue Wang *, Jieyu Zhang*, Letian Xue, Zihan Wang, Qineng Wang, Keshigeyan Chandrasegaran, Ruohan Zhang, Yejin Choi, Ranjay Krishna, Jiajun Wu, Li Fei-Fei, Manling Li
ICLR 2026
Featured by MIT Tech Review China, Stanford AI Blog

MindCube: Spatial Mental Modeling from Limited Views teaser

MindCube: Spatial Mental Modeling from Limited Views [Website][PDF][Data][Code][tl;dr]
Qineng Wang*, Baiqiao Yin*, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Jiajun Wu+, Li Fei-Fei+, Manling Li+
ICLR 2026
Best Paper Award at ICCV 2025 Workshop on Structural Priors for Vision
Best Paper Honorable Mention at NeurIPS 2025 Workshop on LAW 2025: Bridging Language, Agent, and World Models
The Best of ICCV 2025, featured by Voxel 51

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas teaser

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas [PDF][Code][Data][tl;dr]
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, Manling Li
ICML 2025

ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction teaser

ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction [Website][PDF][Code][Data][tl;dr]
Qineng Wang*, Wenlong Huang*, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, Manling Li
ICLR 2026
Outstanding Paper Award at ICLR 2026 Workshop on World Models
Outstanding Paper Award at ICLR 2026 Workshop on Lifelong Agents

Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs teaser

Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs [Website][PDF][Code][tl;dr]
Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Leonidas Guibas, Jiajun Wu, Yejin Choi
NeurIPS 2026
Best Paper Award at CVPR 2026 Workshop on Test-Time Scaling for Computer Vision (ViSCALE)

SkillCraft: Can LLM Agents Learn to Use Tools Skillfully? teaser

SkillCraft: Can LLM Agents Learn to Use Tools Skillfully? [Website][PDF][Code][tl;dr]
Shiqi Chen, Jingze Gai, Ruochen Zhou, Jinghan Zhang, Tongyao Zhu, Junlong Li, Kangrui Wang, Zihan Wang, Zhengyu Chen, Klara Kaleb, Ning Miao, Siyang Gao, Cong Lu, Manling Li, Junxian He, Yee Whye Teh

BAGEN: Are LLM Agents Budget-Aware? teaser

BAGEN: Are LLM Agents Budget-Aware? [Website][PDF][Code][Data][tl;dr]
Yuxiang Lin, Zihan Wang, Mengyang Liu, Yuxuan Shan, Longju Bai, Junyao Zhang, Xing Jin, Boshan Chen, Jinyan Su, Xingyao Wang, Jiaxin Pei, Manling Li

ROSETTA: Constructing Code-Based Reward from Unconstrained Language Preference teaser

ROSETTA: Constructing Code-Based Reward from Unconstrained Language Preference [Website][PDF][Data][Code][tl;dr]
Sanjana Srivastava*, Kangrui Wang*, Yung-Chieh Chan*, Tianyuan Dai, Manling Li, Ruohan Zhang, Mengdi Xu, Jiajun Wu, Li Fei-Fei
ICLR 2026
Best Paper Award at RSS 2025 on Continual Robot Learning from Humans

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents teaser

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents [Website][PDF][Code][tl;dr]
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang
ICML 2025
Oral (Top 1%)

Exploring Diffusion Transformer Designs via Grafting teaser

Exploring Diffusion Transformer Designs via Grafting [Website][PDF][Blog][Code][tl;dr]
Keshigeyan Chandrasegaran*, Michael Poli*, Daniel Y. Fu, Dongjun Kim, Lea M. Hadzic, Manling Li, Agrim Gupta, Stefano Massaroli, Azalia Mirhoseini, Juan Carlos Niebles, Stefano Ermon, Li Fei-Fei
NeurIPS 2025
Oral (Top 0.36%)

T*: Re-thinking Temporal Search for Long-Form Video Understanding teaser

T*: Re-thinking Temporal Search for Long-Form Video Understanding [Website][PDF][Data][Code][tl;dr]
Jinhui Ye*, Zihan Wang*, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, Manling Li
CVPR 2025
Oral at ICCV 2025 Workshop on Long Multi-Scene Video Foundations

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making teaser

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making [Website][PDF][Code][Data][Docker][PyPi][Doc][tl;dr]
Manling Li*, Shiyu Zhao*, Qineng Wang*, Kangrui Wang*, Yu Zhou*, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu
NeurIPS 2024 D&B Track
Oral (Top 0.6%)
Best Paper Award at SoCal NLP 2024, Top 0.4%

HourVideo: 1-Hour Video-Language Understanding teaser

HourVideo: 1-Hour Video-Language Understanding [Website][PDF][Data][Code][tl;dr]
Keshigeyan Chandrasegaran, Agrim Gupta, Taran Kota, Lea M. Hadzic, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, Li Fei-Fei
NeurIPS 2024 D&B Track

LM-Steer: Word Embeddings Are Steers for Language Models teaser

LM-Steer: Word Embeddings Are Steers for Language Models [Website][PDF][Code][Live Demo][Slides][Poster][tl;dr]
Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, Heng Ji
ACL 2024
(Outstanding Paper Award at ACL 2024)

Event-centric Multimodal Knowledge Acquisition teaser

Event-centric Multimodal Knowledge Acquisition [PDF]
Manling Li
Thesis Committee: Heng Ji, Jiawei Han, Chengxiang Zhai, Shih-Fu Chang, Kyunghyun Cho
PhD Thesis, 2023 (ACL Inaugural Best Dissertation Award Honorable Mention)

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners teaser

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners [PDF][Code]
Zhenhailong Wang†*,Manling Li*, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, Heng Ji
NeurIPS'22 (equal contribution)

CLIP-Event: Connecting Text and Images with Event Structures teaser

CLIP-Event: Connecting Text and Images with Event Structures [PDF][Data][Code]
Manling Li, Ruochen Xu, Shuohang Wang, Xudong Lin, Chenguang Zhu, Xuedong Huang, Heng Ji, Shih-Fu Chang
CVPR'22
(Oral, Top 4.1%)

COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation teaser

COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation [PDF][Code/Data]
Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, David Liem, Ahmed Elsayed, Martha Palmer, Jasmine Rah, Clare Voss, Cynthia Schneider, Boyan Onyshkevych
NAACL'21: System Demonstrations
(Best Demo Paper Award at NAACL2021)

GAIA: A Fine-grained Multimedia Knowledge Extraction System teaser

GAIA: A Fine-grained Multimedia Knowledge Extraction System [PDF][Code][Video]
Manling Li*, Alireza Zareian*, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare R. Voss, Dan Napierski, Marjorie Freedman
ACL'20
(Best Demo Paper Award at ACL2020)

GAIA at SM-KBP 2020 - A Dockerized Multi-media Multi-lingual Knowledge Extraction, Clustering, Temporal Tracking and Hypothesis Generation System teaser

GAIA at SM-KBP 2020 - A Dockerized Multi-media Multi-lingual Knowledge Extraction, Clustering, Temporal Tracking and Hypothesis Generation System [PDF][Project]
Manling Li, Ying Lin, Tuan Manh Lai, Xiaoman Pan, Haoyang Wen, Sha Li, Zhenhailong Wang, Pengfei Yu, Lifu Huang, Di Lu, Qingyun Wang, Haoran Zhang, Qi Zeng, Chi Han, Zixuan Zhang, Yujia Qin, Xiaodan Hu, Nikolaus Parulian, Daniel Campos, Heng Ji, Brian Chen, Xudong Lin, Alireza Zareian, Amith Ananthram, Emily Allaway, Shih-Fu Chang, Kathleen McKeown, Yixiang Yao, Michael Spector, Mitchell DeHaven, Daniel Napierski, Marjorie Freedman, Pedro Szekely, Haidong Zhu, Ram Nevatia, Yang Bai, Yifan Wang, Ali Sadeghian, Haodi Ma, Daisy Zhe Wang
TAC-KBP: Text Analysis Conference Knowledge Base Population Workshop 2020
Rank 1st in the National Institute of Standards and Technology (NIST) Streaming Multimedia Knowledge Base Population (SM-KBP) 2020

Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization teaser

Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization
[PDF]
Manling Li, Lingyu Zhang, Heng Ji, Rich Radke
ACL'19: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.2190–2196

Awards

Research Awards

  • ACL Inaugural Best Dissertation Award Honorable Mention, 2025
  • MIT Tech Review Innovators Under 35, Global List, 2025
  • Kuruganti Junior Faculty Fellowship, Northwestern University, 2026
  • Delta Institute Rising Professors, 2026
  • Best Paper Award at CVPR 2026 Workshop on Multimodal Reasoning for Agentic Intelligence (MMRAgI)
  • 20 Papers That Matter at ICML 2026, selected by Nebius Science
  • Best Paper Award at CVPR 2026 Workshop on Test-Time Scaling for Computer Vision (ViSCALE)
  • Best Paper Award at ACL 2026 Workshop on Trustworthy Natural Language Processing (TrustNLP)
  • Outstanding Paper Award at ICLR 2026 Workshop on World Models
  • Outstanding Paper Award at ICLR 2026 Workshop on Lifelong Agents
  • Best Paper Award at AAAI 2026 Workshop on Trustworthy Agentic Systems
  • Outstanding Paper Award at AAAI 2026 Workshop on Personalization in the Era of Large Foundation Models
  • Best Paper Honorable Mention at NeurIPS 2025 Workshop on Language Agents and World Models
  • Best Poster Award at Midwest Machine Learning Symposium 2025
  • Best Paper Award at RSS 2025 workshop on Continual Robot Learning from Humans
  • Best Paper Award at ICCV 2025 workshop on Structural Priors for Vision
  • Outstanding Paper Award at ACL 2024
  • Best Paper Award at SoCal NLP 2024
  • Best Demo Paper Award at NAACL 2021
  • Best Demo Paper Award at ACL 2020
  • AAAI 2025 New Faculty Highlights
  • Microsoft Research Postdoc Fellowship 2023
  • EECS Rising Star 2022
  • DARPA Riser 2022
  • Microsoft Research PhD Fellowship 2021
  • Mavis Future Faculty Fellow
  • C.L. and Jane Liu Award
  • National Scholarship

Student Awards

  • Workday AI PhD Fellowship (Zenus Zihan Wang)
  • Bloomberg Data Science Ph.D. Fellowship (Canyu Chen)
  • Northwestern Presidential Fellowship Finalist (Zenus Zihan Wang)

Academic and Scientific Competitions

  • Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2020
  • Ranked 1st in NIST TAC Streaming Multimedia Knowledge Base Population (SM-KBP) 2019, with more than 10% absolute gains compared the second ranked team

Academic Service

  • Organizing Committee
    • NAACL 2027 (Workshop Chairs)
    • ACL 2025 (Virtual Infrastructure Chairs) summary
    • NAACL 2025 (Publication Chairs) summary
    • EMNLP 2024 (Demonstration Chairs) summary
    • AI-Native Academia: Authorship, Peer Review, and Conference Governance under AI @ NeurIPS 2026
    • SCALE: Scalable Learning and Optimization for Efficient Multimodal AI Agents @ ICML 2026
    • Failure Modes in Agentic AI @ ICML 2026
    • 4th Workshop on Advances in Language and Vision (ALVR) @ ACL 2026
    • 2nd Workshop on 3D-LLM/VLA @ CVPR 2026
    • 1st Workshop on Efficient Spatial Reasoning @ ICLR 2026
    • Midwest Speech and Language Days (MSLD) 2026
    • Bridge Program on Reasoning and Planning in Foundation Models @ AAAI 2025, AAAI 2026
    • Responsible Foundation Models @ NeurIPS 2025
    • Space in Vision, Language, and Embodied AI @ NeurIPS 2025
    • Foundation Models Meet Embodied Agents @ CVPR 2025, CVPR 2026
    • Towards Knowledgeable Foundation Models @ ACL 2024, AAAI 2025, ACL 2025, ACL 2026
    • Language + Molecules @ ACL 2024
    • Knowledge Discovery from Unstructured Data in Financial Services (KDF) @ SIGIR 2023
  • Senior Area Chair: NAACL (from 2027), ACL (from 2026), EMNLP (from 2025)
  • Area Chair: ACL (from 2023), EMNLP (from 2023), NAACL (from 2024), COLM (from 2025), ICLR (from 2025), ICML (from 2026), NeurIPS (from 2026)
  • Program Committee: ARR (ACL Rolling Review, from 2021), ACL (from 2021), EMNLP (from 2021), NAACL-HLT (from 2021), AAAI (from 2021), WWW (from 2021), AKBC (2021), EACL (from 2021), KDD DI Workshop (2021), NLPCC (2021), COLING (from 2020), AACL (from 2020), CCL (from 2020)
  • Journal Reviewer: TACL, TIST, TKDE
  • Community Services
    • Mentor, ACL Mentorship Program, ACL 2026
    • Mentor, CVPR 2026 Doctoral Consortium
    • Mentor, Women in Machine Learning (WiML) @ NeurIPS 2025
    • Mentor, ICCV 2025 Doctoral Consortium
    • Guest Lecturer, Stanford AI4All (9th Grade Outreach Program)
    • Mentor, ACL Mentorship Program, EMNLP 2024
    • ACL Student Research Workshop (SRW) Mentor, ACL, 2024
    • ACM Mentor, ACM Mentorship Program at UIUC, 2022
    • CS Ambassador, UIUC CS Visit Day for Prospective Graduate Students, 2022
    • Advising Assistant, UIUC PhD Orientation Seminar, 2021
    • Graduate Student Representative, UIUC CS Visit Day, 2020

Teaching

Instructor:

  • COMP_SCI 396 Reasoning and Planning in the Foundation Model Era
    NU, Winter 2025
  • COMP_SCI 496 Agent AI
    NU, Spring 2025

Guest Lecturer:

  • Navigating Up and Down in the Job Searching
    COMP_SCI 496 Academic Job Search
    NU, Fall 2023
  • Event-Centric Multimedia Encoding
    CS 6604 Advanced Topics in Natural Language Processing
    Virginia Tech, Fall 2022
  • Knowledge-Driven Vision-Language Pretraining
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2022
  • Recent Advances in Multimedia Encoding
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2022
  • Timeline Summarization: Introducing Temporal Dimensions into Summarization
    CS 598 Knowledge Driven Natural Language Generation
    UIUC, Spring 2022
  • Multimedia Encoding via Vision-Language Pretraining
    CS 546 Advanced Topics in Natural Language Processing
    UIUC, Fall 2021

Students

It is a great pleasure to work with such talented young people. I am grateful for the trust that they have placed in me.
>> Proud of what our students have achieved in their very first year!

PhD Students

  • Zenus Zihan Wang (2024-)
  • Qineng (Aiden) Wang (2024-)
  • Kangrui (James) Wang (2024-)
  • Pingyue (William) Zhang (2024-)
  • Canyu Chen (2025-)

Visiting PhD Students

  • Shiqi Chen (2024-)
  • Lei (Max) Zhang (2025-)

Join Us!

MLL Lab is 1 year old now! We officially started in Oct 2024. We are growing and looking for more talented students and postdocs to join us!
MLL Lab first year highlights: RAGEN-2 2.7k+ GitHub Stars, ACL 2024 Outstanding Paper, NeurIPS 2025 Oral (top 0.36%), ICML 2025 Oral (top 1%), ICCV 2025 Workshop Best Paper, RSS 2025 Workshop Best Paper, MIT Tech Review Innovators Under 35, ACL Inaugural Best Dissertation Award Honorable Mention, AAAI 2025 New Faculty Highlights, Bloomberg PhD Fellowship; Multimodal Learning with Language (MLL) Lab at Northwestern University, directed by Manling Li

Why Northwestern?

Northwestern University - A Prestigious Institution

2026
#7 🏆
2025
#6 ⭐
2024
#9
2023
#10
2022
#9
2021
#9

US News National University Rankings

Good Research

Just in 2025:

We have 7 faculty members named as Sloan Research Fellows in 2025! Ranked as top 1 university having the most faculty in the 2025 cohort, together with MIT.

We have Joel Mokyr win Nobel Prize in economics in 2025! Also another winner Peter Howitt got his PhD from Northwestern.

Fun Facts

Mascot: Cat 🐱

School Color: Purple 💜

Campus Card: Wildcard 🎫

Money on it: Cat Cash 😸

Backyard: Two private beaches for students and faculty 🏖️

Lake-view: Floor-to-ceiling windows staring straight at Lake Michigan 🌊

Right downstairs: 6 yoga studios, 4 Pilates, Whole Foods, Trader Joe's, Paris Baguette, etc 🧘‍♀️

Wildcard Perks: Student discount pass to 300+ restaurants/stores in Evanston/Chicago (yes, really)

The personal touch: The only place that mailed me a customized gift after faculty interview: a mug with my name in 20+ fonts, all in purple 💜

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.