Sign inSign up

oaklight/llama-wrangler

By oaklight

•Updated 2 months ago

Lightweight web admin panel for llama.cpp server management

Image
0

2.7K

oaklight/llama-wrangler repository overview

⁠llama-wrangler

Lightweight web admin panel for llama.cpp⁠ server management.

⁠Features

  • Model Browser — Scan local directory for .gguf files, view name/size/modified
  • Model Download — Search HuggingFace for GGUF models, download with progress tracking
  • Server Lifecycle — Start/stop/restart llama-server subprocess from the browser
  • Parameter Config — Visual editor for llama-server flags (context size, GPU layers, batch size, flash attention, etc.)
  • System Monitoring — Real-time GPU (VRAM, temp, utilization, power), CPU, RAM, and disk usage
  • Log Viewer — Stream llama-server stdout/stderr via Server-Sent Events
  • Health Monitoring — Poll /health endpoint, show status badge
  • i18n — English and Chinese interface, switchable at runtime

⁠Install

pip install llama-wrangler
⁠Prerequisites

llama-wrangler manages a llama-server process on the host machine. Make sure you have:

  • llama.cpp compiled with llama-server binary (build instructions⁠)
  • NVIDIA GPU driver installed (for GPU inference and monitoring)

⁠Quick Start

# Start the admin panel
llama-wrangler --host 0.0.0.0 --port 7860

# With custom config
llama-wrangler --config /path/to/config.json

Then open http://localhost:7860 in your browser.

⁠Configuration

Config is stored at ~/.config/llama-wrangler/config.json:

{
  "llama_server_path": "/path/to/llama-server",
  "models_dir": "/path/to/models",
  "default_args": {
    "host": "0.0.0.0",
    "port": 8080,
    "n_gpu_layers": 99,
    "ctx_size": 8192,
    "flash_attn": true,
    "batch_size": 2048,
    "ubatch_size": 512,
    "threads": 0,
    "parallel": 1,
    "cont_batching": true,
    "metrics": true
  }
}

⁠Docker

⁠Host prerequisites

The following must be set up on the host machine before running the container:

  1. NVIDIA GPU driver — install from NVIDIA⁠ or your distro's package manager
  2. NVIDIA Container Toolkit — required for --gpus flag to work:
    # Ubuntu/Debian
    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
      sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
      sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
      sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
    sudo nvidia-ctk runtime configure --runtime=docker
    sudo systemctl restart docker
    
    See the official install guide⁠ for other distros.
  3. llama.cpp compiled on the host with llama-server binary
  4. Verify everything works: docker run --rm --gpus all ubuntu nvidia-smi
⁠Build and run
# Build
docker build -t llama-wrangler .

# Run
docker run --gpus all --network host \
  -e LD_LIBRARY_PATH=/opt/llama.cpp/bin:/opt/cuda/lib64 \
  -e LLAMA_SERVER_PATH=/opt/llama.cpp/bin/llama-server \
  -v /path/to/models:/mnt/data/models \
  -v /path/to/llama.cpp/build/bin:/opt/llama.cpp/bin:ro \
  -v /sys:/sys:ro \
  -v ~/.config/llama-wrangler:/root/.config/llama-wrangler \
  llama-wrangler

Volume mounts explained:

MountPurpose
-v .../build/bin:/opt/llama.cpp/bin:rollama-server binary and shared libs from host
-v /path/to/models:/mnt/data/modelsGGUF model files (read/write for downloads)
-v /sys:/sys:roSensor data (disk/NVMe temperatures via psutil)
-v ~/.config/llama-wrangler:...Persist configuration across restarts
--gpus allGPU access (nvidia-smi, CUDA for llama-server)
-e LD_LIBRARY_PATH=...Shared lib search path for host-compiled binaries
-e LLAMA_SERVER_PATH=...Override default llama-server binary path

Environment variable overrides (precedence: env > config file > default):

VariableDefaultPurpose
LLAMA_SERVER_PATH/opt/llama-serverPath to llama-server binary
LLAMA_MODELS_DIR/mnt/data/modelsPath to GGUF model directory

Note: CPU and RAM metrics work out of the box in Docker — psutil reads /proc which is shared from the host. GPU monitoring requires --gpus all via nvidia-container-toolkit.

⁠Without GPU

llama-wrangler works without a GPU (CPU-only inference). Simply omit --gpus all:

docker run --network host \
  -e LD_LIBRARY_PATH=/opt/llama.cpp/bin \
  -e LLAMA_SERVER_PATH=/opt/llama.cpp/bin/llama-server \
  -v /path/to/models:/mnt/data/models \
  -v /path/to/llama.cpp/build/bin:/opt/llama.cpp/bin:ro \
  llama-wrangler

The GPU section on the dashboard will be hidden automatically.

⁠Tech Stack

  • Backend: Python asyncio (zero-framework, vendored HTTP server)
  • Frontend: Single-file vanilla HTML/CSS/JS
  • Dependencies: huggingface-hub, psutil only
  • No: Flask, FastAPI, React, npm, database

⁠License

MIT

Tag summary

Content type

Image

Digest

sha256:2f7b03fab…

Size

60.4 MB

Last updated

5 months ago

docker pull oaklight/llama-wrangler