gguf-cpp
GitHub ↗
ONE ENGINE · THREE WORKFLOWS
A focused Python package for running local language models, generating media, and editing GGUF files — all powered by one unified gguf.cpp build.
01 / FIRST RUN
Install the package, launch the unified GUI, and keep the CLI close when you need reproducibility.
The bundled engine compiles during installation. Use a backend flag when you want GPU acceleration.
$ pip install gguf-cpp# NVIDIA
$ GGUF_CPP_CUDA=1 pip install gguf-cppAlso available: GGUF_CPP_HIP=1 · GGUF_CPP_VULKAN=1. Metal is enabled by default on macOS.
One local server mounts Server, Diffuser, and Editor at predictable routes on one port.
$ gguf-cpp/server/LLM chat + API/diffuser/media generation/editor/GGUF tools02 / THE WORKSPACE
Each panel is useful on its own. Together they cover the path from model file to working application.
LOCAL API
Load a GGUF model, tune context and GPU layers, then connect any OpenAI SDK or your own HTTP client. Streaming is supported.
Read server manual →03 / TECHNICAL MANUAL
Copy a working baseline, then change one variable at a time as you tune performance.
Start the GUI or hand raw arguments to gguf-server.
$ gguf-cpp server engine -- \
--model model.gguf \
--port 8888 \
--ctx-size 8192 \
--n-gpu-layers 999Use gguf-cpp server --help for panel options, or add --no-browser in headless environments.
Use the bundled diffusion CLI for a deterministic first test.
$ gguf-cpp diffuser engine -- \
-m sd.gguf \
-p "a lighthouse at dusk" \
-o out.pngFor image-to-image, masks, control images, and text encoders, use the Diffuser GUI or pass the engine's full CLI flags.
Convert a model and select a default tensor type.
$ gguf-cpp editor quantize \
-m in.gguf \
-o out-q4_k.gguf \
--type q4_k
$ gguf-cpp editor devicesKeep the original file. Use --tensor-type-rules when select tensors need a different quantization type.
04 / INTEGRATION
The server exposes familiar endpoints, so local inference can fit into tools you already use.
/v1/modelsList models/v1/chat/completionsChat + streaming/v1/completionsText completion/v1/embeddingsVector embeddings/healthReadiness check$ curl http://127.0.0.1:8888/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "local-model",
"messages": [{"role":"user", "content":"Hello"}],
"stream": true
}'05 / FOR DEVELOPERS
The repository keeps Python orchestration and the vendored C++ engine together, with one CMake configure.
Editable installs compile and place the engine artifacts beside each panel.
$ pip install -e .Use CMake flags or environment variables to opt into CUDA, HIP, Vulkan, or Metal.
$ CMAKE_ARGS="-DGGUF_CPP_CUDA=ON" \
pip install -e .Check panel help and the quantizer device list before running a model.
$ gguf-cpp --version
$ gguf-cpp editor devices