TaskMatrix connects ChatGPT with a series of visual foundation models so you can send and receive images in the middle of a chat — asking for a picture to be drawn, edited, segmented, or explained in plain language.
TaskMatrix is an open-source research project from Microsoft, hosted on GitHub with around 34,000 stars; it was most active through 2023 and early 2024. It is the codebase behind the Visual ChatGPT paper: a large language model acts as a general interface and planner, while specialized foundation models act as domain experts for vision tasks. TaskMatrix introduced templates — pre-defined execution flows that help the LLM assemble complex multi-model tasks, such as locating an object with GroundingDINO, segmenting it with Segment Anything, and editing it with diffusion-based inpainting.
The repository also hosts two companion efforts that grew out of the project: TaskMatrix.AI, a broader multi-model platform, and LowCodeLLM, which explores building LLM workflows with far less code. New visual foundation models can be registered as additional tools the LLM can call, and community members contributed templates that package multi-model solutions — such as infinite outpainting — into reusable flows other users can run directly.
TaskMatrix is free and open source. You run it yourself with Python and provide your own OpenAI API key; the individual foundation models it loads carry their own licenses, which users are responsible for following.
Researchers and developers exploring multimodal agents — especially anyone studying how an LLM can plan and call chains of vision models, or prototyping chat-driven image editing workflows.