Building ExpertLens: Real-time AI Coaching for Software You Control Directly

ExpertLens is a real-time voice and vision coaching agent built on Gemini Live API, designed for desktop GUI apps like Blender, Affinity Photo, and Unreal Engine where AI cannot directly control the software. It identifies a 'Human-Control Gap'—closed-binary applications with no programmatic interface—where coaching is the only viable AI assistance. The system supports desktop, Android, and iOS through adaptive screen capture methods and adds curated, up-to-date domain knowledge beyond what general-purpose models provide.