vimGPT is an open-source experiment that lets a multimodal model browse the real web: GPT-4V looks at screenshots of pages and drives Chrome through the Vimium extension’s keyboard interface. The project has around 2,600 GitHub stars.
Most browsing agents read the page DOM as text. vimGPT asks a different question: can a vision model navigate using only what it sees? Vimium overlays every clickable element on a page with a short key label, which gives the model a reliable way to say what it wants to click without coordinates or DOM parsing. The agent loops through screenshots, decides the next keystrokes, and types, scrolls and follows links like a keyboard-only user. It is a neat demonstration of an idea that later became mainstream: giving multimodal models a structured, labelled view of the screen instead of raw pixels or markup. The code is a compact research prototype rather than a polished product, and it was last updated in 2024. The repository includes setup instructions for loading Vimium locally and installing the Python requirements. You supply your own OpenAI API key for GPT-4V access.
vimGPT is free and open source under the MIT license. Running it requires your own OpenAI API key, and GPT-4V usage is billed by OpenAI.
AI researchers and tinkerers interested in vision-driven browser agents and in how far screenshot-only control can go.