ResearchResearch paper

RELIEF turns interaction feedback into revision training data for generated UI

Revised October 20, RELIEF has a coding agent repair generated interfaces from feedback, then folds those improvements back into the original generation traces for code-model fine-tuning.

Event 2026-10-20Source published 2026-10-20Indexed 2026-10-22Verified 2026-10-22

Why it matters

Many usability failures only appear when people click, fill forms and complete tasks. This work connects post-interaction feedback to training rather than scoring only the first screenshot.

A closer look

The authors build a dataset of 200 synthetic websites, 355 interaction recordings and 24 participants, comparing critiques from code inspection, automated interaction, explicit user feedback and inferred human-interaction feedback. Experiments cover two code-generation models, Qwen 27B and Inkling; under their setup, critiques inferred from human interaction recordings achieve the highest observed Elo in both model families.

Who this is for

Teams working on generative-UI training, interaction feedback, usability evaluation and code models

Original sources

Revisions & corrections (1)

2026-10-22 — Primary paper verifies the revision method, dataset, feedback sources, results and explicit limitations.

Related events

Research2026-10-29

A second, independent VeriCUA turns GUI-task acceptance into active evidence gathering with abstention

A Google and University of Hong Kong team submitted a separate project also named VeriCUA on October 29. It inspects an environment or recording before constructing a task rubric, then gathers missing evidence through GUI, accessibility-tree, file and database reads, marking requirements met, unmet or unknown. In online experiments it reduces false positives against baselines on AndroidWorld, OSWorld and ScienceBoard.

Google and University of Hong Kong VeriCUA arXiv paperResearch paper

Research2026-10-27

GUI‑JEPA replaces token-by-token action decoding with candidate-conditioned latent scoring

Submitted October 27, GUI‑JEPA targets single-step mobile GUI action selection. It transforms JEPA representations pretrained on unlabeled interface screenshots for each action and region candidate, then scores accessibility-tree candidates in one batch instead of asking a 7B vision-language model to decode an action token by token.

GUI-JEPA authors / arXiv and official repositoryResearch paper

Research2026-10-27

TAM turns verified task advancement into online state and reusable operations for GUI agents

Submitted October 27, Task Advancement Memory (TAM) uses what has been reliably accomplished at two timescales: it maintains verified progress, remaining goals and visual evidence during a task, then extracts reusable operations from locally successful segments after the task, including segments inside an overall failed workflow.

arXiv paper and full textResearch paper