WayTrace

Real world · Zurich

Four missions, 2.35 km, one wheeled-legged robot

Each run is shown with the third-person view, the robot camera with the predicted trace, and every command and reasoning of the orchestrator. The videos run at 5× speed, slow to 2× at highlighted moments and show the arrival in real time. The video starts when the pointer first moves over the player; click a chapter or a marker to jump to it.

manoeuvrecrossing arrivaloverride or stop

Chapters

Moments

Turns, crossings, construction sites and arrivals

Short clips from the four missions, each with the command the orchestrator issued. Hover to play, click to enlarge. Each view shows four clips drawn at random; click a category, or the same one again, for others.

Method

Map, language, trace

Overview of WayTrace with the logged inputs and outputs of one decision in the Snipes mission. The web map provides the route once, the off-the-shelf orchestrator VLM issues the next atomic command every 2 s, the trace predictor draws it in the current image, and an RL navigation policy follows the trace lifted to 3D.

Training data of the trace predictor. Reconstructed Mapillary walks provide real history, and intent-conditioned traces pair one image with several commands. In total 1.19M samples from 182 cities.

Results

What the experiments show

All numbers are from the paper. Hover a mark for its value, or open the table under each chart.

Try it

Trace predictor demo

Give the trace predictor a street-level image and an atomic command, and it returns the goal point and the ten-point trace it would send to the robot.

Upload your own image, or pick one of 50 example images from cities around the world.
Type a command such as "Take the left fork" or "Cross the road at the zebra crossing".
See the goal and the trace drawn in the image.

Hugging Face Space · coming soon

BibTeX

@article{patel2026waytrace,
  title   = {WayTrace: A Map-to-Language-to-Trace Hierarchy for Embodied Urban Navigation},
  author  = {Patel, Manthan and Frey, Jonas and Roth, Pascal and Hutter, Marco and Cadena, Cesar},
  journal = {arXiv preprint arXiv:2611.05129},
  year    = {2026}
}

Acknowledgments

The authors would like to thank Turcan Tuna for help with the state estimation and Claudio Bonaglia, Tim Tschanz and Cyril Püntener for help with the real-world deployments. This work was supported by the NVIDIA Academic Grant Program through the project Hierarchical Vision-Language Navigation Foundation Model. It was also supported as part of the Swiss AI Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID abs19 on Alps, by the Luxembourg National Research Fund (Ref. 18990533) and by the Swiss National Science Foundation (SNSF) through projects No. 200021E_229503 and No. 227617.

Abstract