Lightweight web admin panel for llama.cpp server management
2.7K
Lightweight web admin panel for llama.cpp server management.
.gguf files, view name/size/modifiedllama-server subprocess from the browser/health endpoint, show status badgepip install llama-wrangler
llama-wrangler manages a llama-server process on the host machine. Make sure you have:
llama-server binary (build instructions)# Start the admin panel
llama-wrangler --host 0.0.0.0 --port 7860
# With custom config
llama-wrangler --config /path/to/config.json
Then open http://localhost:7860 in your browser.
Config is stored at ~/.config/llama-wrangler/config.json:
{
"llama_server_path": "/path/to/llama-server",
"models_dir": "/path/to/models",
"default_args": {
"host": "0.0.0.0",
"port": 8080,
"n_gpu_layers": 99,
"ctx_size": 8192,
"flash_attn": true,
"batch_size": 2048,
"ubatch_size": 512,
"threads": 0,
"parallel": 1,
"cont_batching": true,
"metrics": true
}
}
The following must be set up on the host machine before running the container:
--gpus flag to work:
# Ubuntu/Debian
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
llama-server binarydocker run --rm --gpus all ubuntu nvidia-smi# Build
docker build -t llama-wrangler .
# Run
docker run --gpus all --network host \
-e LD_LIBRARY_PATH=/opt/llama.cpp/bin:/opt/cuda/lib64 \
-e LLAMA_SERVER_PATH=/opt/llama.cpp/bin/llama-server \
-v /path/to/models:/mnt/data/models \
-v /path/to/llama.cpp/build/bin:/opt/llama.cpp/bin:ro \
-v /sys:/sys:ro \
-v ~/.config/llama-wrangler:/root/.config/llama-wrangler \
llama-wrangler
Volume mounts explained:
| Mount | Purpose |
|---|---|
-v .../build/bin:/opt/llama.cpp/bin:ro | llama-server binary and shared libs from host |
-v /path/to/models:/mnt/data/models | GGUF model files (read/write for downloads) |
-v /sys:/sys:ro | Sensor data (disk/NVMe temperatures via psutil) |
-v ~/.config/llama-wrangler:... | Persist configuration across restarts |
--gpus all | GPU access (nvidia-smi, CUDA for llama-server) |
-e LD_LIBRARY_PATH=... | Shared lib search path for host-compiled binaries |
-e LLAMA_SERVER_PATH=... | Override default llama-server binary path |
Environment variable overrides (precedence: env > config file > default):
| Variable | Default | Purpose |
|---|---|---|
LLAMA_SERVER_PATH | /opt/llama-server | Path to llama-server binary |
LLAMA_MODELS_DIR | /mnt/data/models | Path to GGUF model directory |
Note: CPU and RAM metrics work out of the box in Docker — psutil reads
/procwhich is shared from the host. GPU monitoring requires--gpus allvia nvidia-container-toolkit.
llama-wrangler works without a GPU (CPU-only inference). Simply omit --gpus all:
docker run --network host \
-e LD_LIBRARY_PATH=/opt/llama.cpp/bin \
-e LLAMA_SERVER_PATH=/opt/llama.cpp/bin/llama-server \
-v /path/to/models:/mnt/data/models \
-v /path/to/llama.cpp/build/bin:/opt/llama.cpp/bin:ro \
llama-wrangler
The GPU section on the dashboard will be hidden automatically.
huggingface-hub, psutil onlyMIT
Content type
Image
Digest
sha256:2f7b03fab…
Size
60.4 MB
Last updated
5 months ago
docker pull oaklight/llama-wrangler