---
title: "Can a Coding Agent Run a Robot? Stanford's VIA Experiment"
url: "https://goremy.ai/blog/articles/coding-agent-robot-brain-stanford-via-bitter-lesson"
author: Luis Chavez-Mattos
editor: Luis Chavez-Mattos
editor_title: Director, Product
published: "2026-11-19T22:00:15.224Z"
updated: "2026-11-19T22:00:15.224Z"
topic: AI Agents & Automation
description: "Stanford's VIA let a general coding agent finish 24 of 25 robot trials with no robot-specific training. Here's how it works, and why speed is the limit."
---

# Can a Coding Agent Run a Robot? Stanford's VIA Experiment

> Stanford's VIA let a general coding agent finish 24 of 25 robot trials with no robot-specific training. Here's how it works, and why speed is the limit.

Edited by Luis Chavez-Mattos, Director, Product · Published 2026-11-19

## Can a coding agent control a robot without special training?

Yes, in a small but striking test. Stanford researchers connected a general coding agent to a commercial Stretch 4 robot through a visual interface called VIA (Visual Interface Agent). The agent completed 24 of 25 trials across five physical tasks, with no extra robot-specific training. It looked at camera views, chose where the gripper should go, checked what happened, and adjusted. The limits are real, though. The study is small, and speed is a serious problem for any AI that has to think while the physical world keeps moving.

## TL;DR

- **A general coding agent drove a real robot** in Stanford's VIA experiment, finishing 24 of 25 trials across five tasks without robot-specific fine-tuning.
- **The agent picks targets, and a controller moves the arm.** The model does not compute every motor movement.
- **The Bitter Lesson explains why this matters.** Richard Sutton argued in 2019 that general methods that improve with more computation and learning tend to overtake carefully hand-built systems.
- **OpenAI has reorganized its robotics research** around a related bet: that better general frontier models help produce better robots.
- **Speed is the main obstacle.** In a separate October study on a G1 robot, roughly 90% of the time went to model deliberation, and successful manipulation runs took over an hour.
- **The surrounding software matters.** A November study improved a model's simulation score without changing the model, only the software around it.

## What is Stanford's VIA?

VIA stands for Visual Interface Agent. It is a way of giving an AI agent eyes and a pointer. Instead of training a special robotics model, the researchers handed an existing general-purpose agent an interface where it could look at a scene and choose where the robot's gripper should go in space and time.

In the physical experiment, they paired the agent with a commercial Stretch 4 robot. The agent was a coding agent, the same kind of tool people use to write software. Nobody retrained it on robot data for this test. The researchers describe this as taking capabilities the model already developed for other uses and getting useful behavior on the first attempt.

## How does the agent actually move the robot?

Picture a drawer you want the robot to open. The process works like this:

1. The agent looks at the camera view.
2. It places a virtual gripper where it thinks the real gripper needs to end up, on the drawer handle.
3. It checks that placement. Maybe the target is a touch high, or from another view a touch too far forward. It adjusts the target before the robot moves.
4. Once satisfied, it tells the controller to move the real gripper to that spot.
5. It looks again. Did the gripper catch the handle? Did it miss? Does it need to adjust before pulling?

The next decision comes after seeing what happened, not from a fixed script. The model is not working out every motor movement. A controller handles getting the arm to the target, and the agent's job is choosing the target. That is a real part of the work. You can move an arm perfectly and accomplish nothing if you aim at the wrong place.

It is similar to how you reach for a coffee cup. You pick the target and your body handles which muscles fire. And it looks a lot like what a coding agent already does on a computer: look at what's available, use a tool, check what changed, decide what to do next. Here the tool moves a physical machine, and the consequences are physical too. A virtual gripper in the right place does not prove you will get a good grip. You have to make the move and find out.

One task in the paper shows the behavior well. The robot had to wipe a mark off a table, and its own gripper blocked the camera's view. The agent moved the gripper away to check whether the table was clean. In one attempt, holding the wipe was not working well enough, so the agent left the wipe on the table and pushed it with the closed gripper. Try something, see whether it worked, change approach. Nobody trained it to do that for this robot.

## What is the Bitter Lesson, and why does it apply here?

Richard Sutton wrote "The Bitter Lesson" in 2019. His argument: AI researchers keep hitting the same pattern. We work out what makes a task hard, build that understanding into a system, and get something that works. Then a more general approach comes along that keeps improving with more computation, more learning and more search, and eventually it overtakes the system we spent years designing. Sutton traced this through chess, Go, speech recognition and computer vision. The bitter part is that the expertise you put in gets walked past.

For robotics, the possibility is this. Much of what a robot needs is what AI labs are already improving for other reasons:

- Better visual understanding
- Better tool use
- Better ability to work through a long task
- Better ability to notice when something went wrong

Those matter when an agent uses a computer. They can also matter when it uses a robot arm. So some of a robot's intelligence may be getting built while AI gets better at everything else.

This does not mean specialized robotics models are doomed. A model trained on huge amounts of physical experience can follow the Bitter Lesson too. The open question is how much you need to build specifically for a robot when a general model can do much of the same work.

## Why did OpenAI reorganize its robotics team?

OpenAI confirmed that its robotics researchers were moving into its main research organization, while applied robotics and hardware moved into its applied organization. The company says robotics is still part of the plan, and explains the change by saying general frontier models are becoming more useful for robotics work. Reporting also discusses financial pressures, so there may be several reasons behind the decision.

The technical logic is easy to see, though. Think about how much effort already goes into making agents better at using software: interpreting what they see, choosing an action, remembering what they tried, recognizing when they are stuck. If some of that carries over to a robot, a far larger group of people is improving robotics without working on robotics. A small robotics team no longer has to produce every useful advance itself.

## What limits a general AI agent on a robot?

Three things stand out.

**Speed.** A separate October study made this painfully clear. Its simulation could pause while the model thought, but the real world does not pause. In trials with a G1 robot, roughly 90% of the time went to model deliberation, and successful manipulation runs took over an hour. That is not the same setup as VIA, but it shows that being correct is not enough. On a quiet workbench you have time to stop and think. If an object starts sliding off the edge, or you are on a manufacturing line, you do not. A controller has to keep the arm moving correctly in real time.

**The harness.** A harness is the software around the model: what it sees, what tools it gets, how results are fed back. A November study kept the underlying model fixed and substantially improved its simulation score just by improving that surrounding software. A good interface determines what the model can see and do. Intelligence does not help if it lacks the information to make choices.

**Small evidence.** VIA is a small study. The 24 of 25 result is encouraging, and similar small results are appearing across the robotics industry, but it does not show that you can plug an agent into any robot and have it work.

Speed also decides whether a job is worth automating. A robot that eventually finishes while someone stands there supervising it may cost more attention than it saves. This is part of why household robots have not taken off. Dancing robots show that fast physical movement is possible, but coordinating motion is different from solving a complicated problem.

## What does this mean for you?

Most readers will not build robots. The takeaway still travels.

**If you follow AI for work:** when a new model ships, look past the obvious use. An improvement in computer use may also help a machine make sense of a camera view or keep track of a physical task. That transfer is not automatic, but it is testable.

**If you evaluate vendors or tools:** be skeptical of any pitch built on a specialized system that assumes the underlying model will stop improving. The history of AI is full of people who bet against the Bitter Lesson and lost. Ask what happens to the product when the general model gets better.

**If you work in operations, warehousing or manufacturing:** don't judge a robot demo by whether it completes the task. Ask three questions:

| Question | Why it matters |
|---|---|
| How long did the whole attempt take? | Slow success can still be a failure for the job |
| How often did a person intervene? | Supervision costs attention |
| What happened when it made a mistake? | Edge cases decide real-world value |

**If you build or tinker:** pair a general agent with a good harness and a controller that handles real-time motion, and test how fast it can respond.

## Could you track tests like these yourself?

If you run pilots of any automation, from a robot arm to an AI agent, you need a record of trials, outcomes, time per attempt and human interventions. Those are exactly the numbers the questions above call for, and most teams keep them in a messy spreadsheet. With [Remy](https://goremy.ai), you can describe a simple trial log in plain English (fields for task, result, duration, interventions and notes, plus a dashboard of success rates) and get a deployed web app your team can sign in to. It is a small internal tool you own, rather than another subscription.

## Frequently Asked Questions

### What did the Stanford VIA experiment show?

A general coding agent, with no extra robot-specific training, completed 24 of 25 trials across five physical tasks on a Stretch 4 robot. It worked by viewing the scene, choosing gripper targets through a visual interface, and checking results after each move.

### Does this mean general AI will replace specialized robotics models?

Not necessarily. A model trained on large amounts of physical experience can also follow the Bitter Lesson. The open question is how much robot-specific building is needed when a general model can do much of the same work.

### Why did OpenAI move its robotics researchers?

OpenAI says general frontier models are becoming more useful for robotics work. Its robotics researchers moved into main research, and applied robotics and hardware moved into its applied organization. The company says robotics remains part of the plan. Reporting also mentions financial pressures.

### What is the biggest obstacle to AI-driven robots?

Speed. In a separate October study with a G1 robot, about 90% of the time went to model deliberation, and successful runs took over an hour. Robots need to respond in real time, especially when something unexpected happens.

### Can I plug any agent into any robot and expect it to work?

No. The surrounding software matters a great deal. A November study improved a fixed model's simulation score substantially just by improving the software around it. Good interfaces and harnesses decide what the model can see and do.
