llama.cpp: Building and Running Models
What you'll learn: How to compile llama.cpp from source, convert models to the GGUF format, and run efficient CPU-based inference.
What is llama.cpp?
While Ollama provides a polished interface, llama.cpp is the underlying engine that makes CPU inference practical. It's a C++ implementation of LLaMA (and other) models optimized for local hardware, enabling you to run quantized models without a GPU. Think of it as the high-performance engine under the hood—bare-metal control for maximum efficiency.
Building llama.cpp
Clone and compile the repository:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make
This produces executables like main (for inference) and quantize (for model conversion). On Apple Silicon, use LLAMA_METAL=1 make to enable GPU acceleration.
Converting Models to GGUF
llama.cpp uses the GGUF format—a custom format optimized for efficient loading and quantization. Convert a Hugging Face model:
python convert.py /path/to/model --outfile model.gguf
Then quantize it (e.g., to 4-bit):
./quantize model.gguf model-q4.gguf Q4_K_M
Running Inference
Execute the compiled binary with your quantized model:
./main -m model-q4.gguf -p "Explain quantum computing" -n 128
This runs entirely on CPU (or Metal on Mac), with parameters controlling context length, temperature, and more.
Key Takeaway: llama.cpp gives you direct control over model compilation, quantization formats, and CPU-optimized inference—perfect when you need maximum customization or want to understand the internals of local LLM deployment.