ResearchResearch paper

CanvasObs exposes structured state that canvas agents cannot see in pixels

Submitted October 19, CanvasObs supplements screenshots with object identity, visibility, native relationships and off-screen coordinates, then tests the added state through a fixed action pipeline.

Event 2026-10-19Source published 2026-10-19Indexed 2026-10-22Verified 2026-10-22

Why it matters

Pixels do not reliably reveal covered or off-screen objects, native bindings or exact geometry. A generative-interface runtime may need to expose actionable semantics instead of asking a model to reconstruct all state from the image.

A closer look

The authors run 1,200 trials across 10 targeted configurations, three models and 20 repetitions per condition. Pooled correct-target selection rises from 16.5% to 48.3%, while full-task success rises from 4.3% to 25.0%. Yet only 153 of 290 target-correct CanvasObs trials also place the object correctly (52.8%), separating target discovery from action execution.

Who this is for

Teams building canvas agents, design tools, generative-UI runtimes and evaluations

Original sources

Revisions & corrections (1)

2026-10-22 — Historical catch-up: the original PDF now supplies methods, denominators, confidence intervals and limitations.

Related events

Research2026-10-29

A second, independent VeriCUA turns GUI-task acceptance into active evidence gathering with abstention

A Google and University of Hong Kong team submitted a separate project also named VeriCUA on October 29. It inspects an environment or recording before constructing a task rubric, then gathers missing evidence through GUI, accessibility-tree, file and database reads, marking requirements met, unmet or unknown. In online experiments it reduces false positives against baselines on AndroidWorld, OSWorld and ScienceBoard.

Google and University of Hong Kong VeriCUA arXiv paperResearch paper

Research2026-10-27

GUI‑JEPA replaces token-by-token action decoding with candidate-conditioned latent scoring

Submitted October 27, GUI‑JEPA targets single-step mobile GUI action selection. It transforms JEPA representations pretrained on unlabeled interface screenshots for each action and region candidate, then scores accessibility-tree candidates in one batch instead of asking a 7B vision-language model to decode an action token by token.

GUI-JEPA authors / arXiv and official repositoryResearch paper

Research2026-10-27

TAM turns verified task advancement into online state and reusable operations for GUI agents

Submitted October 27, Task Advancement Memory (TAM) uses what has been reliably accomplished at two timescales: it maintains verified progress, remaining goals and visual evidence during a task, then extracts reusable operations from locally successful segments after the task, including segments inside an overall failed workflow.

arXiv paper and full textResearch paper