LAGEN

Are Vision-Language Agents Latency Aware?

Training agents for a world that keeps moving

* Equal contribution; author order is interchangeable.
† Work done during internship at Northwestern.

1 Northwestern2 UMass Amherst3 Stanford4 UT Austin5 NVIDIA

While a VLA agent calculates an action, the environment continues to move. When the action starts, the state is already different. On an RTX 3090 in real time, the median return of 36 VLA policies is only 6.6% of their return without latency. LAGEN measures the latency of a deployment and trains agents to act correctly with it.

A small delay makes the same actions fail

Both videos start from the same state and use the same actions. The right video applies each action later, as on a real GPU. Move the slider to change the delay.

Evaluation that stops the environment hides the effect of latency

This failure is not specific to one episode. On Flappy Bird and Demon Attack, we add a delay of only two frames. The return then falls to 1.2–7.1% of the return without delay.

OpenVLAπ₀.₅GR00T Single action Action chunk (horizon 8)

Flappy Bird

Eval latency (frames) Mean return 0 100 200 300 400 500 0 1 2 3 4 Zoom: latency > 0 0 10 20 1 2 3 4

Demon Attack

Eval latency (frames) 0 1000 2000 3000 4000 5000 6000 7000 0 2 4 Zoom: latency > 0 100 200 300 2 4

Deadly Corridor

Eval latency (frames) 0 100 200 300 400 500 600 700 0 1 2 3 4 5 Zoom: latency > 0 80 160 240 1 3 5

Intercept Grab (Fast)

Eval latency (frames) Success rate (%) 0 10 20 30 40 50 60 70 80 90 100 0 1 2 3 4

Focus a chart and use the left and right arrow keys to inspect latency columns. Press Escape to dismiss values. On touch screens, tap a column.

Each point shows one fixed action delay. For the games, the plots show the mean return. For Intercept Grab (Fast), the plot shows the success rate over 200 episodes. The small plots show the delays above zero at a larger scale.

Faster inference does not remove latency. Train with latency.

Faster inference makes the latency smaller, but the latency does not become zero. Thus, we keep the latency and train the agent to act with it.

Reduce latency

Make inference faster. Predict new actions while the agent does the current actions.

Train with latency (ours)

Use the deployment latency in simulation. Train agents to act with this latency.

Result: training with latency increases the real-time return for all 36 combinations of task and model.

Show results for all 12 tasks
OpenVLAπ₀.₅GR00T
Before training After training Zero latency (100%)
GameEmbodied 0% 25% 50% 75% 100% 125% 150% 175% 200% Relative performance (%) Demon Attack Asterix Atlantis AirRaid Deadly Corridor Flappy Bird Ant Hopper Inverted Pendulum Walker2D Balance Intercept Grab (Fast)
0% 25% 50% 75% 100% 125% 150% 175% 200% Relative performance (%) Demon Attack Asterix
0% 25% 50% 75% 100% 125% 150% 175% 200% Relative performance (%) Atlantis AirRaid
0% 25% 50% 75% 100% 125% 150% 175% 200% Relative performance (%) Deadly Corridor Flappy Bird
0% 25% 50% 75% 100% 125% 150% 175% 200% Relative performance (%) Ant Hopper
0% 25% 50% 75% 100% 125% 150% 175% 200% Relative performance (%) Inverted Pendulum Walker2D
0% 25% 50% 75% 100% 125% 150% 175% 200% Relative performance (%) Balance Intercept Grab (Fast)

Hover or tap to inspect values. Focus a chart and use the arrow keys to move between readings. Press Escape to close.

Hatched bar: return before latency training. Full bar: return after latency training. 100% is the return of the original policy without latency. GPU: RTX 3090.
Table 1 · Core results. Training with the measured RTX 3090 latency profile improves the real-time return in all 36 task–architecture pairs. Zero/Zero is the original policy without delay. Zero/Real runs that policy during live inference. Target/Real first trains with the matched profile and then evaluates in real time. Cells report mean episode return on each task's native scale.
GamesEmbodied Control
ModelCondition DemonAttackAsterixAtlantisAirRaidDeadlyCorridorFlappyBirdAntHopperInvertedPendulumWalker2DBalance(simple)InterceptGrabFast
OpenVLA Zero/Zero 2.36k 5.80k 39.56k 2.00k 2.10k 439.1 5.77k 2.67k 1.00k 2.26k 275.6 36.1
Zero/Real 584.4 374.5 18.59k 878.8 742.4 19.2 132.2 8.6 19.8 117.8 50.7 18.6
Target/Real 1.45k↑ 3.65k↑ 39.23k↑ 2.35k↑ 2.09k↑ 373.8↑ 2.25k↑ 3.40k↑ 586.9↑ 4.33k↑ 114.6↑ 25.8↑
π₀.₅ Zero/Zero 2.35k 5.76k 39.62k 1.94k 2.10k 408.7 5.78k 3.48k 1.00k 3.63k 328.6 36.8
Zero/Real 112.3 143.5 4.37k 1.39k 140.9 5.6 92.7 8.1 20.2 86.2 46.8 8.1
Target/Real 907.3↑ 1.02k↑ 26.45k↑ 3.23k↑ 1.45k↑ 344.5↑ 3.18k↑ 2.68k↑ 830.4↑ 3.80k↑ 72.3↑ 17.6↑
GR00T Zero/Zero 2.37k 5.71k 39.66k 2.10k 2.10k 428.5 5.94k 2.41k 1.00k 3.72k 267.5 33.5
Zero/Real 231.9 182.0 8.17k 733.8 341.8 6.6 134.2 8.6 15.6 94.0 45.3 14.3
Target/Real 1.20k↑ 1.44k↑ 28.49k↑ 1.59k↑ 2.09k↑ 414.5↑ 3.00k↑ 3.72k↑ 1.00k↑ 3.96k↑ 90.4↑ 25.2↑

Can general-purpose agents act with delay?

We test four agents on 12 tasks without task-specific training. They choose actions from observations while the environment keeps moving, using simulated OpenVLA delays.

Results vary by agent and task. On Asterix, GPT-6.1 Sol scores 5.85k; GPT-6 Luna and Claude Haiku 5.5 score zero.

Four zero-shot agents under simulated delay. Values are mean episode returns; k means thousands. Paper Table 2.

Games

AgentDemonAttackAsterixAtlantisAirRaidDeadlyCorridorFlappyBird
GPT-6.1 Sol668.55.85k18.34k567.5313.217.3
GPT-6 Luna90.30.00.068.8111.33.5
Claude Opus 5.5235.0560.01.30k451.3378.75.7
Claude Haiku 5.582.30.0100.00.0-44.25.1

Embodied control

AgentAntHopperInvertedPendulumWalker2DBalance(simple)InterceptGrabFast
GPT-6.1 Sol-14.5193.839.6322.446.51.2
GPT-6 Luna145.772.00.015.442.40.2
Claude Opus 5.50.3179.98.9150.044.70.6
Claude Haiku 5.5-203.198.60.033.344.70.3

Simulated delays: these scores are not directly comparable to the hardware-timed VLA results above.

How we evaluate the agents

GPT-6.1 Sol and GPT-6 Luna run through Codex; Claude Opus 5.5 and Haiku 5.5 run through Claude Code. All four use Low reasoning effort and select actions directly from observations.

Delays are sampled from measured OpenVLA latency profiles, rather than the agents’ own inference times. Each task uses its native reward scale.

We do not have a matched zero-delay baseline for each agent. Low scores could reflect task competence, the action interface, or latency. These results alone do not show that latency causes the low scores or that task-specific training is necessary. See paper §4.2 and Table 2.

How do we train with latency? First, we make a model of the deployment latency. Then, we train the agent with this model. See the method.

Simulate the deployment latency

First, we find the latency that the agent will have during deployment. We measure the latency on the target hardware and fit a latency profile to the measurements. Then, we use the profile to make new latency sequences in simulation.

Slow requests occur in groups

Two deployments can have the same mean latency but different sequences of latency. Only the Temporal profile shows that slow requests occur together in time.

Show the latency profiles

Action latency (ms)

5075100 010k20k Request index

Density (ms⁻¹)

0.25.5 5075100 Latency (ms)

Action latency (ms)

5075100 010k20k Request index

Density (ms⁻¹)

0.25.5 5075100 Latency (ms)

Action latency (ms)

5075100 010k20k Request index

Density (ms⁻¹)

0.25.5 5075100 Latency (ms)

Action latency (ms)

5075100 010k20k Request index

Density (ms⁻¹)

0.25.5 5075100 Latency (ms)

Action latency (ms)

5075100 010k20k Request index

Density (ms⁻¹)

0.25.5 5075100 Latency (ms)
● Worker 0■ Worker 1Measured distribution

Hover or focus to highlight. Click or press Enter to keep a selection; repeat to clear it. Use arrow keys to move between choices. Escape clears the selection.

The measured latency, and sequences from four fitted models. Select a latency range to highlight the matching requests in all five sequences. The plots show the range 45–100 ms. The distributions use the full sequences.

The latency simulation predicts the return on real hardware

Does the latency simulation predict the return on real hardware? With the Temporal profile, the prediction error decreases from 8.93 to 3.81 percentage points. The baseline is a fixed latency that is equal to the mean.

Show simulated and real returns
Flappy BirdInverted PendulumIntercept Grab (Fast)
RTX 3090RTX 4090RTX 5090L40SA100
OpenVLA
Fixed
0 0 25 25 50 50 75 75 100 100 Real mean return (%) Simulated mean return (%)
Normal
0 0 25 25 50 50 75 75 100 100 Real mean return (%) Simulated mean return (%)
IID
0 0 25 25 50 50 75 75 100 100 Real mean return (%) Simulated mean return (%)
Temporal
0 0 25 25 50 50 75 75 100 100 Real mean return (%) Simulated mean return (%)
GR00T
Fixed
0 0 25 25 50 50 75 75 100 100 Real mean return (%) Simulated mean return (%)
Normal
0 0 25 25 50 50 75 75 100 100 Real mean return (%) Simulated mean return (%)
IID
0 0 25 25 50 50 75 75 100 100 Real mean return (%) Simulated mean return (%)
Temporal
0 0 25 25 50 50 75 75 100 100 Real mean return (%) Simulated mean return (%)

Hover or tap to inspect values. Focus a chart and use the arrow keys to move between readings. Press Escape to close.

Both axes show the return relative to the same baseline without latency (100%). On the diagonal line, the simulated return is equal to the measured return. “Sim − real” is the difference in percentage points.
How we test the latency simulation

We evaluate OpenVLA and GR00T on three tasks: Flappy Bird, Inverted Pendulum, and Intercept Grab (Fast). We use five GPU types: RTX 3090, RTX 4090, RTX 5090, L40S, and A100.

For each combination, we use 25 runs to fit the latency model and 25 different runs to evaluate it. Each run has two episodes. The latency simulation includes both action delay and inference capacity.

Normalized return error for all combinations, in percentage points: Fixed 8.93, Normal 6.55, IID 4.58, Temporal 3.81. Refer to paper §3.4.

Train agents with the deployment latency

Then, we train the agent with the profile. First, an RL teacher learns to act with the simulated latency. Then, the VLA learns from the actions of the teacher.

  1. 1Measure the deployment

    Measure the latency for the target GPU, model, and task. Fit a profile that also shows groups of slow requests.

  2. 2Train a teacher with simulated latency

    In simulation, each action starts after a latency from the profile. An RL teacher learns to act with this latency.

  3. 3Teach the VLA

    Fine-tune the VLA on the actions of the teacher. DAgger is optional.

  4. 4Act with the same latency

    The VLA receives the task, the latency as text, and optional recent frames.

We use the same latency profile for training and for evaluation.
Full pipeline figure from the paper LAGEN pipeline from the paper: profile deployment timing using Fixed, Normal, IID or Temporal generators; train a latency teacher through fresh reinforcement-learning interactions; distill teacher demonstrations into a VLA with optional DAgger; deploy the VLA with delayed-action execution.

Select the figure to open it at full size. ActionReady shows the time when a predicted action is available. The curves show example latency profiles.

Three questions about latency training

What must we train with? How well does the training transfer? What information does the agent need?

1. Training with the full profile gives a higher return

Is the mean latency sufficient for training? No. Training with the full profile gives a higher return on three of four tasks. On Flappy Bird, the two returns are almost equal.

OpenVLA with one action for each request. The values are mean episode returns.
TrainingFlappy BirdDeadly CorridorAntIntercept Grab (Fast)
Fixed mean384.821620.801453.843.54
Full profile373.782091.231853.3318.20
Profile gain-2.9%+29.0%+27.5%+413.4%

2. Transfer from high to low latency is better than from low to high latency

Does a policy work at a latency that is different from its training latency? Transfer from high to low latency is better than transfer from low to high latency. On Flappy Bird, a policy works well only near its training latency.

Return relative to a policy trained at the evaluation latency

0%25%50%75%100%
Flappy Bird
Eval latency (raw frames)
4 1.1% 1.3% 1.1% 2.7% 100.0%
3 1.5% 1.4% 1.3% 100.0% 7.0%
2 1.5% 2.2% 100.0% 3.7% 1.8%
1 4.1% 100.0% 1.6% 1.6% 1.4%
0 100.0% 19.7% 1.8% 1.5% 1.4%
01234
Train latency (raw frames)
Demon Attack
Eval latency (raw frames)
32 4.8% 8.1% 37.8% 87.5% 100.0%
24 3.3% 6.0% 24.3% 100.0% 65.2%
16 4.6% 27.4% 100.0% 66.5% 51.8%
8 4.3% 100.0% 84.9% 64.6% 32.6%
0 100.0% 64.0% 36.5% 24.6% 23.0%
08162432
Train latency (raw frames)
VizDoom Deadly Corridor
Eval latency (raw frames)
32 36.2% -0.2% 65.4% 52.9% 100.0%
24 90.3% 76.0% 323.2% 100.0% 278.2%
16 65.2% 44.1% 100.0% 121.7% 122.9%
8 45.3% 100.0% 74.2% 79.8% 68.4%
0 100.0% 26.8% 50.0% 50.1% 25.9%
08162432
Train latency (raw frames)
Intercept Grab (Fast)
Eval latency (raw frames)
4 13.2% 19.3% 33.3% 80.7% 100.0%
3 33.8% 50.8% 84.6% 100.0% 89.2%
2 66.9% 91.9% 100.0% 91.9% 67.4%
1 60.1% 100.0% 100.0% 80.9% 64.9%
0 100.0% 96.4% 96.4% 77.9% 62.6%
01234
Train latency (raw frames)

Hover or tap to inspect values. Focus a chart and use the arrow keys to move between readings. Press Escape to close.

Each cell shows the return of a policy at one evaluation latency. 100% is the return of a policy trained at that evaluation latency. The arrows go from training latency to evaluation latency. High → low latency is the mean of the ten cells in the lower-right triangle. Low → high latency is the mean of the ten cells in the upper-left triangle.

3. Latency training can help on other tasks

Does the training help on other tasks? We train a policy on Demon Attack with mixed latencies. On Atlantis and AirRaid, this policy gets a higher return than a policy trained without latency. On Asterix, the result is mixed.

Zero-latency trainMixed-latency train
Demon Attack
0 2k 4k 6k 0 2 4 Mean return Eval latency (raw frames)
Asterix
0 100 200 300 0 2 4 Mean return Eval latency (raw frames)
Atlantis
0 1k 2k 3k 4k 5k 0 2 4 Mean return Eval latency (raw frames)
AirRaid
0 100 200 300 400 0 2 4 Mean return Eval latency (raw frames)

Hover or tap to inspect values. Focus a chart and use the arrow keys to move between readings. Press Escape to close.

We train both policies on Demon Attack. Each point shows the mean return over 20 episodes. Each task uses its own return scale.

4. The latency value in the prompt increases the return

At different latencies, the same frame can need different actions. When the prompt gives the latency value, the Flappy Bird return increases from less than 20 to 268–434.

Games: mean return. Intercept Grab: success rate (%). Each condition uses 20 episodes.
TaskDelay in prompt0 frames1 frame2 frames3 frames4 frames
Flappy BirdYes433.8434.0341.3290.3267.6
Flappy BirdNo18.814.39.46.54.5
Demon AttackYes7409.06558.05551.24038.03630.0
Demon AttackNo3814.84392.54478.84091.21995.2
Intercept GrabYes30.0%40.0%45.0%50.0%40.0%
Intercept GrabNo20.0%15.0%40.0%40.0%5.0%

In each column, the prompt gives the delay of that column.

5. Recent frames in the model input can increase performance

Recent frames show how the scene moved during the latency. The best format for these frames is different for each task.

Return (% of single-frame baseline)

Flappy Bird

Delay: 3 frames

50 75 100 125 150
Demon Attack

Delay: 6 frames

50 75 100 125 150
Deadly Corridor

Delay: 6 frames

50 75 100 125 150 N/A

Hover or focus to highlight. Click or press Enter to keep a selection; repeat to clear it. Use arrow keys to move between choices. Escape clears the selection.

100% is the return with a single-frame input at the same delay. The delay is three frames on Flappy Bird and six frames on Demon Attack and Deadly Corridor. Error bars show 95% intervals over episodes from one training run. OpenVLA uses 100 evaluation episodes for each condition. WanOFT uses 20. We did not evaluate the ghost trail on Deadly Corridor.

Citation

@misc{chafekar2026lagen,
  title={LAGEN: Are Vision-Language Agents Latency Aware?},
  author={Talha Chafekar and Zihan Wang and Zeju Li and Xinyuan Li and Qineng Wang and Jiajun Wu and Yuke Zhu and Yi Dong and Zhiding Yu and Ruohan Zhang and Manling Li},
  year={2026}
}