Caption Anything is an open-source image captioning tool that combines Meta’s Segment Anything model, visual captioning models, and ChatGPT. It has 1,778 stars on GitHub and was last updated in 2023. Instead of describing a whole image generically, it lets you point at any object and get a caption written for that object, in the style you ask for.
The workflow is interactive: click on an object in an image and Segment Anything isolates it; a captioning model drafts a description; and ChatGPT refines it under your language controls. You can steer length, sentiment, factuality, and language, chat with the tool about the selected object for a deeper understanding, or caption everything in the scene as a paragraph. The project, from Tencent ARC and partner labs, ships with public demos on Hugging Face Spaces and a Colab tutorial, and it accompanies a 2023 research paper. The system was one of the earliest popular combinations of Segment Anything with a large language model, and it helped popularize the click-to-describe interaction pattern now common in vision tools. Instructions and checkpoints for running the demo locally are included in the repository.
Caption Anything is free and open source under the BSD-3-Clause license. Running the full pipeline locally may require a ChatGPT API key for the language refinement step.
It is for computer vision researchers, dataset builders, and accessibility experimenters who need controllable, object-level image descriptions rather than one generic caption per image. It remains a frequently cited example of composing foundation models into a new capability.