Microsoft JARVIS — the system behind the HuggingGPT paper — connects large language models with the wider machine-learning community, letting an LLM plan a task and delegate each step to a specialist model.
JARVIS is an open-source research project from Microsoft, hosted on GitHub with around 25,000 stars and updates into 2025. Its architecture has four stages: task planning, where ChatGPT breaks a request into sub-tasks; model selection, where it picks suitable expert models from Hugging Face; task execution, where those models run; and response generation, where the results are combined into one answer. This lets a single conversational request span vision, audio, and language — describing an image, transcribing speech, and reasoning about both in one flow.
JARVIS can be tried in two ways: a local deployment that downloads and serves expert models on your own GPUs, or a configuration that calls hosted inference endpoints for heavier models. Its Gradio-style interface shows the plan the controller made and the intermediate model results, which makes the system a useful teaching tool for how planning, tool selection, and execution fit together in an agent stack.
JARVIS is free and open source under the MIT license. You run it yourself; depending on configuration you will need an OpenAI API key for the controller and compute for the expert models you load.
Researchers and engineers studying LLM orchestration and agent design — JARVIS remains a canonical example of how a language model can act as a controller over an ecosystem of specialist models.