macOSpilot is an open-source AI assistant for macOS that combines voice and vision: it can see what is on your screen, answer questions about any application in context, and reply in audio. The project has collected about 1,200 stars on GitHub and was last updated in 2023.
Most AI assistants live in a chat window, disconnected from the app you are actually using. macOSpilot was built around a different interaction: you talk to your Mac and ask about what is in front of you, and the assistant looks at the current application context before answering out loud. That makes it useful for moments when your hands are busy or you are stuck inside unfamiliar software: ask what a dialog means, how to find a setting, or what an error message is telling you, and get a spoken answer grounded in the screen rather than a generic guess. Built during the first wave of multimodal assistants using speech recognition and vision-capable models, the project is a compact demonstration of a voice-first, screen-aware copilot for the desktop, and its code remains a reference for developers building similar macOS experiences.
The interaction model leans on short spoken exchanges rather than long dictation: you ask a question, the assistant takes in the screen context, and a concise spoken answer comes back, which keeps it usable while you are presenting, designing or working through an unfamiliar app. As an open project from the early multimodal era, it also documents the practical plumbing of connecting speech input, screenshots and speech output on macOS.
macOSpilot is free and open source: the code is published on GitHub at no cost. You run it on your own Mac and supply your own API access for the underlying speech and vision models it calls.
macOSpilot is for Mac users curious about voice-first, screen-aware assistants, and for developers who want a compact reference implementation of a multimodal desktop copilot.