Interface Quilt turns computer-use agents and generative UI into persistent patches for existing applications
Interface Quilt from Autodesk Research combines computer-use agents with generative UI. Agents first explore controls and observable state through rendered pixels, then a coding agent generates an interface patch over the existing application. The paper demonstrates reordered controls, voice and marking-menu input, in-situ parameters, and delayed or confirmed sends across PowerPoint, Affinity, Fusion, Slack, and Overleaf without source code, application APIs, or accessibility data.
Why it matters
The work moves generative UI from a one-off answer toward a reusable application-modification layer. The generated interface continuously reads host state and translates interaction back into host actions, making pixels, synchronization, OS input, and generated-code permissions part of the product boundary.
A closer look
The system separates build time from runtime. OmniParser v2, Apple Vision OCR, and Holo2-8B identify controls and coordinates before Claude Code generates TypeScript patches. On macOS, a transparent Electron surface hosts patches, a Node controller dispatches events, and a Swift daemon performs OS input and exposes local perception services. Patches can keep sliders, situated menus, or usage traces synchronized with the application and map patch actions back to original controls; some examples also call external speech, transcription, or analysis models.
Who this is for
Product, design, and engineering teams studying malleable software, computer-use agents, desktop automation, assistive interaction, and generative-UI runtimes.
Original sources
Revisions & corrections (1)
— Backfill a newly discovered primary research source directly about generative UI and computer-use agents, preserving the real submission date, system architecture, demonstrations, and explicit privacy, security, synchronization, and evaluation limits.