KTransformers

KTransformers is an open-source framework on GitHub, with more than 19,600 stars, for running and fine-tuning very large language models on modest hardware through CPU–GPU heterogeneous computing. Developed by the MADSys Lab at Tsinghua University together with Approaching.AI and 9#AISoft, it is famous for running DeepSeek-V3 and R1-class mixture-of-experts models on a single consumer GPU paired with plenty of system RAM. The project is updated constantly, with new model support landing throughout 2026.

Overview

KTransformers splits a model’s computation between devices: hot experts and attention live on the GPU, while the bulk of MoE expert weights sit in CPU memory and are computed with kernels optimized for Intel AMX and AVX instruction sets. Its kt-kernel engine accelerates quantized INT4 and INT8 inference, integrates with SGLang for production serving, and adds a three-layer GPU–CPU–disk prefix cache. A companion fine-tuning stack built with LlamaFactory brings the same heterogeneous tricks to training, enabling LoRA and full-parameter fine-tuning of ultra-large MoE models with far less GPU memory than conventional offloading.

Key Features

• CPU–GPU hybrid inference for huge MoE models on consumer hardware
• AMX and AVX512/AVX2 optimized kernels for quantized INT4/INT8 weights
• Day-zero support for new open models such as DeepSeek, Kimi, GLM, and Qwen releases
• Clean Python API and integration with the SGLang serving framework
• GPU–CPU–disk three-layer prefix caching for long contexts
• MoE fine-tuning with LlamaFactory, including LoRA and full-parameter runs
• Support for Intel Arc GPUs, AMD ROCm, and Ascend NPUs

💳 Pricing

KTransformers is completely free and open source under the Apache-2.0 license.

Who Is It For

KTransformers is for researchers, homelab builders, and infrastructure engineers who want to serve or fine-tune frontier-scale open models without a rack of data-center GPUs. If you have a workstation with a single strong GPU and a lot of RAM, it is currently one of the most capable ways to run the largest open mixture-of-experts models locally.