LAGEN

Are Vision-Language Agents Latency Aware?

Training agents for a world that keeps moving

Talha Chafekar^1,2,*,†Zihan "Zenus" Wang^1,*Zeju Li^1,*,†Xinyuan Li^1,*,†Qineng Wang¹Jiajun Wu³Yuke Zhu⁴Yi Dong⁵Zhiding Yu⁵Ruohan Zhang^1,3Manling Li¹

^* Equal contribution; author order is interchangeable.
^† Work done during internship at Northwestern.

¹ Northwestern University² University of Massachusetts Amherst ³ Stanford University⁴ The University of Texas at Austin⁵ NVIDIA

Paper Code

Models Data Profiles Citation

Timing changes the outcome

Both runs start from the same state and replay the same actions. Use the slider to delay the actions on the right.

No delay100 ms delay

Pausing the world hides the cost of latency

While an agent computes, the world changes. Its action then reaches a different state from the one it observed. On Flappy Bird and Demon Attack, a two-frame delay leaves agents with just 1.2–7.1% of their zero-latency return.

OpenVLAπ₀.₅GR00T Single action Action chunk (horizon 8)

Flappy Bird

Demon Attack

Deadly Corridor

InterceptGrabFast

Each point uses a fixed action delay. The game plots show mean return. InterceptGrabFast shows success rate over 200 episodes.

Reducing latency is not enough. Train with it

Reduce latency

Policy Faster inference

Environment

Action execution

The world keeps moving.

Accelerate inference. Predict new actions while executing the current ones.

Train with latency (ours)

Policy Faster inference

Environment

Action execution

Deployment
latency profile

The world keeps moving.

Reproduce deployment latency in simulation. Train agents to act under these delays.

Training with deployment latency improves real-time performance across all 36 combinations of tasks and architectures.

OpenVLAπ₀.₅GR00T

Before training After training Zero latency (100%)

RTX 3090 latency profiles · 12 tasks · 3 VLA architectures. The hatched region shows performance before latency training. The full bar shows performance after training. Both use the original policy’s zero-latency return as 100%.

All 36 results

The table reports mean episode returns. Zero/Zero uses a policy trained and evaluated without latency. Zero/Real evaluates the same policy in real time. Target/Real evaluates a policy trained with the deployment latency profile in real time.

Task	Architecture	Zero/Zero	Zero/Real	Target/Real
Demon Attack	OpenVLA	2362.2	584.4	1452.8
Demon Attack	π₀.₅	2350.9	112.3	907.3
Demon Attack	GR00T	2370.9	231.9	1203.1
Asterix	OpenVLA	5803.5	374.5	3653.5
Asterix	π₀.₅	5758.5	143.5	1015.0
Asterix	GR00T	5714.0	182.0	1440.0
Atlantis	OpenVLA	39560.0	18588.0	39229.0
Atlantis	π₀.₅	39615.0	4369.0	26447.0
Atlantis	GR00T	39662.0	8167.0	28488.0
AirRaid	OpenVLA	2000.0	878.8	2353.8
AirRaid	π₀.₅	1943.8	1386.3	3227.5
AirRaid	GR00T	2101.3	733.8	1586.3
Deadly Corridor	OpenVLA	2098.7	742.4	2091.2
Deadly Corridor	π₀.₅	2104.1	140.9	1450.8
Deadly Corridor	GR00T	2100.5	341.8	2090.8
Flappy Bird	OpenVLA	439.1	19.2	373.8
Flappy Bird	π₀.₅	408.7	5.6	344.5
Flappy Bird	GR00T	428.5	6.6	414.5
Ant	OpenVLA	5565.3	194.3	1853.3
Ant	π₀.₅	5782.5	131.6	1774.7
Ant	GR00T	6222.6	165.4	1956.0
Hopper	OpenVLA	1419.9	67.9	424.6
Hopper	π₀.₅	3482.5	8.4	235.2
Hopper	GR00T	872.8	8.6	242.6
Inverted Pendulum	OpenVLA	1000.0	19.8	586.9
Inverted Pendulum	π₀.₅	1000.0	20.1	830.4
Inverted Pendulum	GR00T	1000.0	15.6	1000.0
Walker2D	OpenVLA	983.9	103.7	857.4
Walker2D	π₀.₅	3631.4	81.0	371.1
Walker2D	GR00T	3528.0	63.2	852.9
Balance	OpenVLA	275.6	50.7	114.6
Balance	π₀.₅	328.6	46.8	72.3
Balance	GR00T	267.5	45.3	90.4
Intercept Grab Fast	OpenVLA	36.1	18.6	25.8
Intercept Grab Fast	π₀.₅	36.8	8.1	17.6
Intercept Grab Fast	GR00T	33.5	14.3	25.2

Reproducing deployment latency in simulation

Simulation predicts deployment performance

We reproduce deployment latency in simulation to predict how agents perform. Temporal replay reduces normalized return error from 8.93 to 3.81 percentage points compared with a fixed mean delay. This comparison covers five GPU types, three tasks, and two models.

Flappy BirdInverted PendulumInterceptGrabFast

RTX 3090RTX 4090RTX 5090L40SA100

OpenVLA

Fixed

Normal

IID

Temporal

GR00T

Fixed

Normal

IID

Temporal

Both axes show return relative to the same zero-latency baseline (100%). The diagonal marks equal simulated and measured returns. Sim − real shows their difference in percentage points.

How we test the simulation

We evaluate OpenVLA and GR00T on three tasks: Flappy Bird, Inverted Pendulum, and InterceptGrabFast. We use five GPU types: RTX 3090, RTX 4090, RTX 5090, L40S, and A100.

Each combination uses 25 runs to fit the latency model and 25 separate runs to evaluate it. Each run contains two episodes. The simulation models both action delays and inference capacity.

Overall normalized return error, in percentage points: Fixed 8.93, Normal 6.55, IID 4.58, and Temporal 3.81. See paper §3.4.

Slow requests occur in bursts

Two deployments can have the same average latency but different sequences of delays. Temporal profiles capture how often different delays occur and how slow requests cluster over time.

Action latency (ms)

Density (ms⁻¹)

Action latency (ms)

Density (ms⁻¹)

Action latency (ms)

Density (ms⁻¹)

Action latency (ms)

Density (ms⁻¹)

Action latency (ms)

Density (ms⁻¹)

● Worker 0■ Worker 1Measured distribution

Measured latency and sequences generated by four fitted models. Select a latency range to highlight matching requests across all five sequences. The displayed range is 45–100 ms. Distributions use the full sequences.

Training agents with deployment latency

We replay deployment latency in simulation while the world keeps moving. A teacher learns to act under these delays through reinforcement learning. We then fine-tune the VLA using the teacher’s action labels.

The agent interacts with the simulator under delays sampled from the latency profile. ActionReady marks when a predicted action becomes available. The curves show schematic latency profiles.

Key Findings

1. Profile-trained policies achieve higher returns

Training with the full latency profile exposes agents to varying delays and consecutive slow requests. Policies trained with the full profile achieve higher returns than those trained at the mean on three of four tasks. Flappy Bird returns are similar.

OpenVLA · one action per request · mean episode return ± standard deviation
Training	Flappy Bird	Deadly Corridor	Ant	InterceptGrabFast
Fixed mean	384.82± 116.79	1620.80± 913.62	1453.84± 693.73	3.54± 7.07
Full profile	373.78± 126.83	2091.23± 579.61	1853.33± 508.69	18.20± 15.34
Profile gain	-2.9%	+29.0%	+27.5%	+413.4%

2. Transfer is stronger from high to low latency

Transfer is stronger from longer delays to shorter delays than in the reverse direction. Flappy Bird performs best near its training delay. Scores use a policy trained at the evaluation delay as the reference.

Relative to matched-delay policy

0%25%50%75%100%

Flappy Bird

Eval latency (raw frames)

4	1.1%	1.3%	1.1%	2.7%	100.0%
3	1.5%	1.4%	1.3%	100.0%	7.0%
2	1.5%	2.2%	100.0%	3.7%	1.8%
1	4.1%	100.0%	1.6%	1.6%	1.4%
0	100.0%	19.7%	1.8%	1.5%	1.4%
	0	1	2	3	4

Train latency (raw frames)

Demon Attack

Eval latency (raw frames)

32	4.8%	8.1%	37.8%	87.5%	100.0%
24	3.3%	6.0%	24.3%	100.0%	65.2%
16	4.6%	27.4%	100.0%	66.5%	51.8%
8	4.3%	100.0%	84.9%	64.6%	32.6%
0	100.0%	64.0%	36.5%	24.6%	23.0%
	0	8	16	24	32

Train latency (raw frames)

VizDoom Deadly Corridor

Eval latency (raw frames)

32	36.2%	-0.2%	65.4%	52.9%	100.0%
24	90.3%	76.0%	323.2%	100.0%	278.2%
16	65.2%	44.1%	100.0%	121.7%	122.9%
8	45.3%	100.0%	74.2%	79.8%	68.4%
0	100.0%	26.8%	50.0%	50.1%	25.9%
	0	8	16	24	32

Train latency (raw frames)

InterceptGrabFast

Eval latency (raw frames)

4	13.2%	19.3%	33.3%	80.7%	100.0%
3	33.8%	50.8%	84.6%	100.0%	89.2%
2	66.9%	91.9%	100.0%	91.9%	67.4%
1	60.1%	100.0%	100.0%	80.9%	64.9%
0	100.0%	96.4%	96.4%	77.9%	62.6%
	0	1	2	3	4

Train latency (raw frames)

Arrows show training → evaluation latency. High → low latency is the mean of the ten cell percentages in the lower-right triangle. Low → high latency is the corresponding mean for the upper-left triangle.

3. Latency training transfers to other tasks

We train on Demon Attack with mixed latencies and evaluate the policy on other games. It achieves higher returns under delay on Atlantis and AirRaid than a policy trained without latency. On Asterix, it outperforms this baseline at two frames and underperforms at four.

Zero-latency trainMixed-latency train

Demon Attack

Asterix

Atlantis

AirRaid

Both policies train on Demon Attack. Each point shows mean return over 20 episodes. Returns use each task’s native scale.

4. Explicit latency information improves control

The same observation can require different actions at different latencies. Stating the latency in the prompt lets the agent distinguish these cases. On Flappy Bird, mean return rises from below 20 to 268–434 across delays of 0–4 frames. On Demon Attack, mean return improves at delays of 0, 1, 2, and 4 frames; the means are similar at three frames.

Mean return ± 95% confidence interval · 20 episodes per condition
Task	Delay in prompt	0 frames	1 frame	2 frames	3 frames	4 frames
Flappy Bird	Yes	433.8± 15.7	434.0± 22.2	341.3± 69.2	290.3± 81.2	267.6± 69.0
Flappy Bird	No	18.8± 7.3	14.3± 7.3	9.4± 3.1	6.5± 1.6	4.5± 0.4
Demon Attack	Yes	7409.0± 302.6	6558.0± 509.7	5551.2± 474.2	4038.0± 896.4	3630.0± 717.4
Demon Attack	No	3814.8± 947.9	4392.5± 851.8	4478.8± 516.9	4091.2± 877.5	1995.2± 705.5

The prompt states the delay shown in each column.

5. Visual history improves control

Visual history shows how the scene changes over time. Ghost trails improve Flappy Bird performance, KV memory gives the largest gain on Demon Attack, and a frame grid performs best on Deadly Corridor.

Return (% of single-frame baseline)

Flappy Bird

3 frames of delay

Demon Attack

6 frames of delay

Deadly Corridor

6 frames of delay

100% marks the single-frame baseline at the same delay. Delays are three frames on Flappy Bird and six on Demon Attack and Deadly Corridor. Error bars show episode-level 95% intervals from one training run. OpenVLA uses 100 evaluation episodes per condition, and WanOFT uses 20. Ghost trail was not evaluated on Deadly Corridor.

Citation

@misc{chafekar2026lagen,
  title={LAGEN: Are Vision-Language Agents Latency Aware?},
  author={Talha Chafekar and Zihan Wang and Zeju Li and Xinyuan Li and Qineng Wang and Jiajun Wu and Yuke Zhu and Yi Dong and Zhiding Yu and Ruohan Zhang and Manling Li},
  year={2026}
}