Midscene

Midscene is an open-source UI automation framework with more than 15,300 stars on GitHub that lets you test and operate interfaces in natural language — it sees screens the way a human does.

Overview

Traditional end-to-end tests break whenever a button moves or a class name changes, because they locate elements by selectors. Midscene takes a different approach: it sends screenshots to a multimodal AI model, asks the model to find and describe what is on screen, and then acts on those coordinates. You write steps in plain language — or simple YAML — such as “click the login button and check the welcome message appears,” and Midscene handles the visual grounding. It works across web pages, Android, iOS, and desktop interfaces, integrating with tools like Playwright and Puppeteer for browser testing and extending to phones and computer-use scenarios. Because instructions are semantic rather than selector-based, tests survive redesigns far better and read like documentation. Midscene comes from ByteDance’s web infrastructure team, is MIT licensed, supports models such as GPT-4o, Qwen-VL, and other vision models, and is under very active development in 2026.

Key Features

• Natural-language actions – describe clicks, inputs, and assertions in plain words instead of selectors.
• Vision-based grounding – a multimodal model locates elements from screenshots, so tests survive redesigns.
• Cross-platform – automate web, Android, iOS, and desktop interfaces with one approach.
• Playwright and Puppeteer integration – drop Midscene into existing JavaScript test stacks.
• Data extraction and assertions – query structured data from the screen and assert visual state.
• Action caching – cache located elements to cut model calls, cost, and flakiness on repeat runs.
• Visual reports – replay runs with screenshots to see exactly what the agent saw and did.

💳 Pricing

Midscene is free and open source under the MIT license. You run it locally and only pay for whichever vision-capable model API you connect to it — or nothing extra if you use a self-hosted open model.

Who Is It For

QA engineers, frontend developers, and automation builders who are tired of brittle selector-based tests, and teams exploring GUI agents that can operate real interfaces on web and mobile.