PyWinAssistant is an open-source computer-using agent framework published on GitHub that operates Windows graphical interfaces from natural-language instructions. The project has gathered around 1,300 GitHub stars and was last updated in 2025.
Most computer-use agents read screenshots pixel by pixel. PyWinAssistant takes a different route: it perceives applications through Windows’ native UI Automation APIs, building a symbolic map of windows, controls and their states, then plans and executes synthetic mouse and keyboard interactions with Visualization-of-Thought and Chain-of-Thought reasoning. Released at the end of 2023 — before Anthropic’s Computer Use and Microsoft’s VoT paper — it demonstrated cross-application workflows driven by a single sentence, such as finding information in one app and posting it from another, with self-healing behaviour when interface elements move or change. The framework unifies GUI, web and API automation under one Python API, includes reasoning agents that simulate actions before performing them, and is aimed as much at accessibility — controlling complex software by voice or simple text — as at automated UI testing.
PyWinAssistant is free and open source under the MIT license. You bring your own LLM API access for the reasoning models it calls.
Researchers and developers exploring computer-use agents, UI-testing engineers, and accessibility builders who want an open, symbolic alternative to screenshot-based automation.