AgentVLN: Bringing Vision-Language Navigation Into the Agent Era
Vision-Language Models (VLMs) have become increasingly capable at understanding images, interpreting natural language, and performing semantic reasoning. However, transferring those capabilities from static visual tasks into the physical world remains a major challenge.
A robot navigating from a natural-language instruction such as:
“Walk down the hallway and find the chair next to the window.”
must do considerably more than understand the words. It needs to identify the hallway, determine the spatial relationship between the chair and window, reason about occlusions, select an appropriate direction, avoid obstacles, and continuously adjust its trajectory.
This is the core challenge of Vision-and-Language Navigation (VLN): translating language and visual observations into reliable, continuous physical actions.
AgentVLN addresses this problem by introducing a VLM-as-Brain architecture. Instead of forcing one large model to perform perception, planning, localization, and control simultaneously, the VLM handles high-level reasoning while specialized skills execute individual capabilities:
VLM Agent → Skill Calling → SLAM/Perception → Robot Navigation
The framework achieves state-of-the-art results on VLN benchmarks including R2R-CE and RxR-CE while using a relatively compact 3-billion-parameter backbone. It also supports real-time local execution on edge hardware such as NVIDIA Jetson platforms.
The work, titled “AgentVLN: Towards Agentic Vision-and-Language Navigation,” has been accepted to ECCV 2026.
🧠 VLM-as-Brain: Separating Reasoning From Execution #
The central idea behind AgentVLN is to avoid treating robot navigation as a monolithic prediction problem.
Instead, the VLM acts as the robot’s cognitive layer. It is responsible for interpreting the mission, understanding the current environment, deciding what information is needed, and selecting the appropriate skill.
Execution is delegated to specialized modules.
These skills can handle tasks such as:
- Environment perception
- SLAM and mapping
- Localization
- Obstacle detection
- Waypoint generation
- Path planning
- Locomotion and navigation
AgentVLN models the navigation process as a Partially Observable Semi-Markov Decision Process (POSMDP) and organizes its skills into perception-layer and planning-layer capabilities.
The resulting workflow resembles human navigation:
Observe → Plan → Act → Observe Again → Adjust
Rather than repeatedly relearning how to perform low-level operations, the agent can invoke existing capabilities when required.
A Plug-and-Play Skill Architecture #
This modularity also makes the system easier to adapt across different robotic platforms.
A quadruped robot, humanoid robot, or another mobile platform may have very different locomotion and sensing capabilities. Instead of retraining the entire VLM, developers can replace or extend the underlying skills to match the hardware.
This creates a useful separation:
Stable cognitive layer + replaceable execution layer
New environments and capabilities can therefore be incorporated by modifying the skill library rather than retraining a large vision-language model from scratch.
🌐 Turning 3D Navigation Into 2D Visual Decisions #
One of the fundamental problems in VLN is the mismatch between what the robot needs to know and what a VLM naturally understands.
The physical environment is three-dimensional. Navigation requires spatial information such as distances, coordinates, traversability, and geometry.
A VLM, however, primarily reasons over two-dimensional visual observations.
AgentVLN addresses this mismatch with Cross-space Representation Mapping.
The system first uses SLAM and other perceptual skills to understand the physical environment. It then projects relevant 3D navigation information back into the robot’s current 2D camera view.
This allows the VLM to make decisions in the visual representation it is naturally suited to understand.
From Coordinate Regression to Waypoint Selection #
The process begins with several perception operations:
- Construct a 3D spatial map.
- Extract the environment’s topological structure.
- Identify traversable regions.
- Generate candidate navigation waypoints.
- Project those waypoints into the camera’s 2D image plane.
The VLM can then select the waypoint that best matches the natural-language instruction.
This effectively transforms an open-ended problem:
“Predict the correct 3D coordinates.”
into a more constrained decision:
“Which visual waypoint best matches the instruction?”
That distinction is important because it allows the system to exploit the VLM’s strengths in visual-semantic reasoning without requiring the model itself to solve all of the underlying 3D geometry.
The selected visual waypoint can subsequently be mapped back into a physical navigation command.
🔄 Context-Driven Self-Correction #
Real-world navigation is inherently noisy.
Furniture can block the target, lighting conditions can change, camera viewpoints can become temporarily uninformative, and small motion errors can accumulate over long trajectories.
A navigation system that blindly follows its initial plan can therefore drift further from the intended route with every action.
AgentVLN introduces a Context-Driven Self-Correction Mechanism to address this problem.
When the current view no longer provides sufficient information or the observed environment conflicts with the expected trajectory, the agent can actively explore rather than continuing along the original path.
Possible corrective behaviors include:
- Turning to obtain a different viewpoint
- Adjusting the robot’s heading
- Searching for alternative paths
- Re-observing partially occluded regions
- Recovering from trajectory drift
This creates a feedback loop in which navigation is continuously adjusted according to newly observed context.
The mechanism is particularly useful for long-horizon navigation, where small errors can otherwise compound into complete task failure.
🔍 QD-PCoT Lets the Robot Ask for Missing Information #
AgentVLN extends the agent concept further with Query-Driven Perceptual Chain-of-Thought (QD-PCoT).
The key idea is that the robot does not always need to make a prediction from incomplete information. Instead, it can identify what information is missing and actively request it from an appropriate perception skill.
Suppose the robot can identify a chair but cannot reliably determine its distance.
Instead of guessing, the agent can issue a query such as:
“How far away is the chair in front of me?”
The system can then invoke a depth-perception module, retrieve the required spatial information, update its internal state, and make a more informed navigation decision.
The resulting process becomes:
Active Query → Retrieve Information → Update Cognitive State → Execute Decision
This gives the navigation agent a form of practical metacognition: it can reason not only about the environment, but also about what information it currently lacks.
Reducing Depth Ambiguity Without Increasing Model Size #
This approach is particularly relevant because a single 2D image often contains insufficient information to determine precise physical distances.
Rather than adding a large specialized 3D reasoning model to the VLM, QD-PCoT allows the agent to call an appropriate perception capability when necessary.
According to the reported experiments, this improves navigation performance without requiring additional model parameters and helps mitigate depth ambiguity in visual inputs.
⚡ Lightweight VLMs Enable Edge Deployment #
Large vision-language models can provide impressive reasoning capabilities, but their computational requirements can make direct deployment on robots difficult.
AgentVLN takes a different approach by using Qwen2.5-VL-3B as its core VLM.
The modular architecture reduces the need to embed every perception and navigation capability directly into the foundation model. Specialized skills handle tasks that are better suited to dedicated robotics modules, while the VLM focuses on high-level decision-making.
This makes local inference more practical on edge hardware.
The research team demonstrated deployment on robotic platforms including:
- Quadruped robot dogs
- Humanoid robots
- Jetson-based edge systems
The robots use onboard cameras to observe their surroundings while externalized perception skills provide mapping and navigation information to the VLM agent.
This architecture is particularly attractive for robotics because local execution reduces dependence on cloud inference and can provide lower-latency responses for physical interaction.
🏆 State-of-the-Art VLN Performance With a Compact Model #
Despite its relatively small 3-billion-parameter backbone, AgentVLN reports state-of-the-art performance across mainstream VLN benchmarks including R2R-CE and RxR-CE.
The results highlight an important design principle: increasing the intelligence of a robotic agent does not necessarily require continuously increasing the size of the foundation model.
Instead, performance can also come from improving how the model interacts with external capabilities.
AgentVLN effectively treats perception, planning, and execution as tools that the VLM can invoke when needed.
That shifts the optimization target from:
“Build a larger model that does everything.”
to:
“Build an agent that knows when and how to use the right capability.”
🧩 Toward Modular Embodied AI Agents #
AgentVLN represents a broader shift in how foundation models can be integrated into robotics.
Traditional end-to-end approaches attempt to map observations and instructions directly to actions. While elegant, these systems can become difficult to debug, adapt, and deploy as environments and hardware become more complex.
AgentVLN instead separates cognitive reasoning from physical execution.
The VLM interprets goals and selects actions at a high level, while specialized modules provide the physical capabilities required to execute those decisions.
This architecture has several potential advantages:
- Modularity: Skills can be added or replaced independently.
- Adaptability: The same cognitive model can work across different robot platforms.
- Efficiency: Specialized modules handle computationally intensive perception tasks.
- Interpretability: Skill calls provide clearer intermediate decision points.
- Edge deployment: A relatively small VLM can serve as the reasoning layer.
- Scalability: New capabilities can be incorporated without retraining the entire foundation model.
🔮 From Task Automation to Physical Agents #
The larger significance of AgentVLN extends beyond navigation.
As foundation models become better at reasoning, robotics systems can increasingly separate what needs to be achieved from how the robot physically achieves it.
In this paradigm, a robot is no longer limited to executing a fixed sequence of pre-programmed actions. Instead, it can interpret a goal, inspect its environment, determine what information it needs, select appropriate capabilities, execute actions, and correct itself when conditions change.
That creates a more agentic robotics loop:
Understand → Query → Plan → Act → Observe → Correct
AgentVLN demonstrates how this concept can be applied to vision-and-language navigation using a compact VLM and a modular skill architecture.
As perception, locomotion, manipulation, and foundation-model reasoning continue to improve, architectures built around this separation between cognitive intelligence and physical skills could become an important foundation for embodied AI.