ResearchResearch paper

IGG moves GUI grounding from clicking what is visible to recovering visibility first

Submitted October 23 and accepted to the EMNLP 2026 main conference, IGG requires an agent to reveal a target through scrolling, hovering, dragging, waiting or a state transition and then click it within eight actions. Its 601 tasks across 16 controllable mock interfaces measure visibility recovery that static-screenshot grounding omits.

Event 2026-10-23Source published 2026-10-23Indexed 2026-10-29Verified 2026-10-29

Why it matters

Real generative interfaces and GUI agents encounter collapsed regions, occlusion, delayed elements and off-screen content. Evaluating only already-visible targets conflates recognizing an element with choosing the interaction, location and parameters that expose it. IGG's interventions isolate the latter as the main bottleneck.

A closer look

The paper reports 298 successes in 315 human trials on a fixed 21-task subset, or 94.6% at 19.8 seconds on average. Across all 601 tasks, GPT‑5.5 reaches 79.4%, the strongest open baseline GUI‑Owl‑1.5 reaches 31.6%, and Qwen3‑VL reaches 30.8%. Naming the reveal-action category raises Qwen3‑VL only to 33.9%, while executing the ground-truth reveal for it raises accuracy to 81.9%, indicating that recognition after exposure is not the main gap. Frequency-balanced SFT plus browser-feedback GRPO on 479 training and 120 held-out validation tasks raises Qwen3‑VL from 40.0% to 59.2%, InternVL3.5 from 28.3% to 47.5%, and GUI‑Owl‑1.5 from 35.0% to 52.5%. The official repository provides 16 self-contained Next.js mock apps, an evaluator and 481 public tasks; the 120 validation tasks remain private to reduce contamination.

THE LIMITS OF THE EVIDENCE

The evidence is the authors' paper and official code; the results were not independently reproduced here. The main benchmark uses a fixed 1440×900 viewport, deterministic mock interfaces and an eight-action budget. It does not reproduce live-app network variance, permissions, navigation, destructive side effects or long-horizon task success, and the authors explicitly say the mockups do not replace live-interface evaluation. Only 127 tasks remain valid across the five-viewport study, which tests layout change after fixed training rather than new-task generalization. The paired real-interface check has one task per app, 16 total. The human study covers only 21 tasks, and scrolls and hovers do not count against the human click-or-drag limit, making budgets comparable but not identical. Training gains are reported on a held-out validation split, while the public repository contains only 481 tasks, so external teams cannot yet fully audit the 120-task validation results. Commercial-model cost, latency and deployment conditions also differ from the open models; the paper provides no independent replication, confidence intervals or production-safety conclusion.

Who this is for

Research and engineering teams evaluating GUI agents, computer-use models, generative-UI visibility recovery, interaction-policy training and browser feedback

Original sources

Revisions & corrections (1)

2026-10-29 — Backfill a verified primary-source study that tests the missing visibility-recovery layer between static grounding and successful GUI interaction.

IGG moves GUI grounding from clicking what is visible to recovering visibility first

Why it matters

A closer look

Who this is for

Original sources

Related events

A second, independent VeriCUA turns GUI-task acceptance into active evidence gathering with abstention

GUI‑JEPA replaces token-by-token action decoding with candidate-conditioned latent scoring

TAM turns verified task advancement into online state and reusable operations for GUI agents