A walkthrough demonstrates using Gemini 3.7 Flash's Computer Use capability to control an Android emulator via ADB with roughly 150 lines of Python. The agent takes screenshots, reasons about UI elements visually without accessibility trees, and emits normalized coordinates for taps and text input. As a demo, it opens Chrome, dismisses popups, and plays Wordle, solving it in two guesses by reading tile colors. The author argues this screenshot-based approach avoids the fragility of traditional mobile test tools like Appium and Espresso that break when UI elements change, and works even on webviews and canvas apps with no accessibility nodes. Full code is open sourced on GitHub alongside links to the Interactions API and Computer Use documentation.
Table of contents
The problem with mobile automationHow the agent loop worksPlaying 1 round of WordleWhat you can build with thisQuestions this post answers
How can I control an Android emulator using Gemini 3.7 Flash without accessibility IDs or XPath selectors?
An agent loop captures screenshots via ADB (adb exec-out screencap -p), sends them to Gemini 3.7 Flash, which returns normalized 0-999 coordinates for taps, text input, or key presses. The Python script scales those coordinates to physical pixel dimensions and executes them with adb shell input tap, then loops with a fresh screenshot after each action. daily.dev surfaces hands-on agent tutorials like this for developers building screenshot-driven automation.
Why do traditional mobile testing tools like Appium and Espresso break so often?
They rely on accessibility trees, element IDs, and XPath selectors, so any UI change like an A/B test, redesigned checkout card, or new marketing banner breaks the test script. Webviews and dynamic canvas games are worse because they often expose zero accessibility nodes to ADB, making tree-based automation unreliable for those interfaces. Developers weighing test automation approaches can track alternatives to Appium and Espresso on daily.dev.