GUI‑JEPA replaces token-by-token action decoding with candidate-conditioned latent scoring
Submitted October 27, GUI‑JEPA targets single-step mobile GUI action selection. It transforms JEPA representations pretrained on unlabeled interface screenshots for each action and region candidate, then scores accessibility-tree candidates in one batch instead of asking a 7B vision-language model to decode an action token by token.
Why it matters
Real-time interface agents do not have to obtain every latency gain by shrinking a generative model. When a host can enumerate trusted action candidates, selection can become a bounded discriminative problem with lower latency and memory for local or frequent interaction. That trade moves coverage, catalog quality and planning responsibility into candidate enumeration and the runtime above it.
A closer look
GUI‑JEPA encodes a screenshot with a 224px ViT‑L/16, produces one latent per candidate through an action- and ROI-conditioned residual predictor, and selects with a structural head, text span ranker and click reranker. The model has about 0.50B parameters, roughly 132M trained. AndroidControl training uses 13,594 episodes and 74,714 steps, with three policy seeds reported. Against the authors' Qwen2.5‑VL‑7B reference under the same screenshot, candidates, history, judge, H20 GPU and batch size, GUI‑JEPA reaches 84.1%/61.0% IDD AC‑Low/High versus 91.0%/70.9%, with about 6% as many parameters. On 100 real AC‑High IDD steps, its p50 is 74.3 ms in fp32 or 51.5 ms in bf16, versus 206.7 ms for 7B vLLM, a reported 2.8–4.0× speedup and roughly 4.6–6.4× lower peak GPU memory. Full-test AC‑Low/High is 78.6%/53.8%; two text rankers lift OOD High from 43.85% to 47.67%.
Who this is for
Research and engineering teams designing mobile GUI agents, action catalogs, on-device interaction, low-latency visual policies and layered planning runtimes
Original sources
- GUI-JEPA authors / arXiv and official repository ↗arxiv.org
- Full paper, experiments, limitations and reproducibility statement ↗arxiv.org
- Official repository — README-only at verification ↗github.com
Revisions & corrections (1)
— GUI-JEPA 给出生成式/代理界面实时路径的重要反例：受控候选场景可用专用判别策略替代逐 token 动作生成。论文日期、方法、同硬件延迟、准确率、统计口径与局限已由 arXiv 全文核验，官方代码仓库的空壳状态也已明确记录。