Ferret is Apple’s open-source multimodal large language model, built to refer to and ground objects in images at any granularity โ from whole regions down to precise points and shapes. Released by Apple’s machine-learning research team, the project has collected more than 8,600 stars on GitHub.
Most vision-language models can describe an image but struggle to say exactly where something is, or to understand a region a user points at. Ferret closes that gap with a hybrid region representation that encodes points, boxes and free-form shapes into a form the language model can both consume and produce. As a result it can answer questions about a specific patch of an image, output bounding boxes for phrases it mentions, and handle referring expression comprehension and phrase grounding in one unified model. The repository ships the model code, checkpoints and evaluation tooling from the original research release. Ferret’s region-aware design also influenced Apple’s later on-device screen-understanding work, making the repository a useful reference for anyone designing interfaces where an assistant must point at, and talk about, specific elements on a screen.
Ferret is free to use from Apple’s public repository under its research license terms (intended for research use); you supply your own compute for inference.
The repository includes the checkpoints and evaluation suite from the original release, so researchers can reproduce the paper’s grounding and referring results before building on top of the model or adapting its region representation to new multimodal systems.
It is for multimodal-AI researchers and engineers working on visual grounding, region-level image understanding and grounded dialogue systems.