Course contentsShow
← AI Engineering
Lesson 1052 of 1,88626. Self-Hosted LLM DeploymentFree lesson

llama.cpp: Building and Running Models

Compiling llama.cpp, converting model formats, and running efficient CPU inference with GGUF models.

llama.cpp: Building and Running Models

What you'll learn: How to compile llama.cpp from source, convert models to the GGUF format, and run efficient CPU-based inference.

What is llama.cpp?

While Ollama provides a polished interface, llama.cpp is the underlying engine that makes CPU inference practical. It's a C++ implementation of LLaMA (and other) models optimized for local hardware, enabling you to run quantized models without a GPU. Think of it as the high-performance engine under the hood—bare-metal control for maximum efficiency.

Building llama.cpp

Clone and compile the repository:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

This produces executables like main (for inference) and quantize (for model conversion). On Apple Silicon, use LLAMA_METAL=1 make to enable GPU acceleration.

Converting Models to GGUF

llama.cpp uses the GGUF format—a custom format optimized for efficient loading and quantization. Convert a Hugging Face model:

python convert.py /path/to/model --outfile model.gguf

Then quantize it (e.g., to 4-bit):

./quantize model.gguf model-q4.gguf Q4_K_M

Running Inference

Execute the compiled binary with your quantized model:

./main -m model-q4.gguf -p "Explain quantum computing" -n 128

This runs entirely on CPU (or Metal on Mac), with parameters controlling context length, temperature, and more.

Key Takeaway: llama.cpp gives you direct control over model compilation, quantization formats, and CPU-optimized inference—perfect when you need maximum customization or want to understand the internals of local LLM deployment.