Building TaskPilot: An AI Agent That Sees Your Screen and Takes Control

TaskPilot is an AI agent built for the Gemini Live Agent Challenge that uses Gemini's multimodal vision to observe a user's screen and execute actions via natural language, replacing brittle selector-based automation. It features two modes—a local desktop agent (TypeScript/Node.js) and a cloud-based browser agent (Python/Playwright on Cloud Run)—unified by an Electron frontend, and employs a 5-layer execution pipeline that escalates from cheap programmatic methods (CDP, regex) to expensive vision-based LLM calls only when necessary.