Kimi K3 with 2.78 trillion parameters runs on 8GB of memory. Developers open source lightweight C inference engine

Comparatively, a developer recently tried to run the Kimi K3 model with 2.78 trillion parameters on a device with only 8GB of RAM, an open source project called kimi-k3-in-c. The project is only 176 KB, written in pure C99, does not rely on GPU, CUDA, PyTorch, or BLAS, and can complete model inference using only the CPU.

The solution leverages Kimi K3's MoE (Hybrid Expert) architectural features. Although the total number of model parameters reached 2.78T, only 16 of the 896 experts on each layer were activated, so instead of loading the full 1.56 TB model weights into memory, the developers stored most of the expert weights in the NVMe hard disk and read them in real time according to inference requirements; at the same time, some dense layers (dense trunks) also used a layer-by-layer streaming loading method.

However, the solution still has significant performance limitations. In 8GB memory mode, the model takes about 32.7 seconds to generate a token, and requires close to 1.7 TB of high-speed storage support.

The developers said that this solution is currently more like an experimental exploration of the direction of large model inference infrastructure optimization. It has no actual production and use value, but its method of streaming hard disk loading+MoE sparse activation provides a new idea for running hyperscale models at low cost in the future.