Sign inSign up

oaklight/llm-comply

By oaklight

•Updated 2 months ago

LLM API compliance tester: validate endpoints against OpenAI Chat, Open Responses and more.

Image
0

4.4K

oaklight/llm-comply repository overview

⁠llm-comply

中文⁠ | English

Multi-format LLM API compliance testing tool — validates endpoints against official specs for OpenAI Chat Completions, Open Responses, Anthropic Messages, Google GenAI generateContent, and Google GenAI Interactions.

Try it online →⁠

⁠Install

pip install llm-comply
# For colored terminal output:
pip install llm-comply[rich]

⁠Quick Start

# Open Responses (default)
llm-comply -u https://api.openai.com/v1 -k $OPENAI_API_KEY -m gpt-4o-mini

# OpenAI Chat Completions
llm-comply --format chat_completions -u https://api.openai.com/v1 -k $OPENAI_API_KEY -m gpt-4o-mini

# OpenAI Responses API
llm-comply --format openai_responses -u https://api.openai.com/v1 -k $OPENAI_API_KEY -m gpt-4o-mini

# Anthropic Messages
llm-comply --format messages -u https://api.anthropic.com/v1 -k $KEY -m claude-haiku-4-5

# Google GenAI — generateContent
llm-comply --format google_generate -u https://generativelanguage.googleapis.com -k $KEY \
  -m gemini-2.5-flash

# Google GenAI — Interactions
llm-comply --format google_interactions -u https://generativelanguage.googleapis.com -k $KEY \
  -m gemini-2.5-flash

Auth headers and defaults are selected per format: Authorization + Bearer for the OpenAI-facing formats, x-api-key for Messages (with anthropic-version), and x-goog-api-key for Google. --auth-header selects the header name only; the prefix follows --bearer/--no-bearer when given, otherwise the format default. -H name:value overrides or adds headers, but cannot replace the resolved credential header.

⁠Options

-u, --base-url URL     API base URL (required)
-k, --api-key KEY      API key (or set OPENRESPONSES_API_KEY env)
-m, --model MODEL      Model name (default: gpt-4o-mini)
--format FORMAT        API format: open_responses, openai_responses,
                       chat_completions, messages,
                       google_generate, google_interactions
                       (legacy aliases: open-responses, openai-responses,
                       openai-chat, anthropic, google-genai,
                       google-interactions)
-f, --filter IDS       Comma-separated test IDs to run
-i, --ignore PATTERNS  Ignore errors matching substrings (e.g. refusal,verbosity)
-H, --header K:V       Extra headers (e.g. anthropic-version:2023-06-01)
--auth-header NAME     Auth header name (default: per format: Authorization,
                       x-api-key, or x-goog-api-key)
--bearer               Prepend "Bearer " to the API key (overrides the format default)
--no-bearer            Don't prepend "Bearer " to API key (overrides the format default)
-v, --verbose          Show request/response on failure
--json                 JSON output for CI
--list                 List available test IDs

⁠Spec Targets

Bundled spec versions are tracked in src/llm_comply/specs/manifest.json⁠ and rendered by llm-comply specs. The manifest is committed, so every master commit and release tag records exactly which upstream spec version it targets.

FormatBundled specVersionSchemasValidationRefreshed
open_responsesopenresponses.json2026-04-24108Schema2026-10-21
openai_responsesopenai_responses.json2.3.0357Schema2026-10-21
chat_completionsopenai_chat.json2.3.02094Schema2026-10-21
messagesanthropic.jsonunknown (no version in spec)2008Schema2026-10-21
google_generategoogle_generate.jsonv1beta252Semantic2026-10-21
google_interactionsgoogle_interactions.jsonv1beta213Semantic2026-10-21
llm-comply specs            # human-readable table
llm-comply specs --json     # machine-readable manifest
llm-comply specs --check    # CI: fail if the manifest drifts from bundled files

Refresh bundled specs from their upstream sources (the manifest is updated too):

llm-comply update-spec --all                     # every upstream-backed format
llm-comply update-spec --format open_responses   # a single format
llm-comply update-spec --format messages         # Anthropic Messages

messages is pulled from the official Anthropic SDK's bundled scripts/mock-spec.json.gz (the Python and TypeScript copies are byte-identical). chat_completions and openai_responses come from the same upstream OpenAI OpenAPI document; the Responses spec bundles only the /responses paths and their transitive $ref closure. Google specs come from the official OpenAPI discovery document (google_generate) and the Interactions API document (google_interactions). google_generate responses validate against GenerateContentResponse; the Interactions spec is bundled for version tracking while runtime validation stays semantic. The discovery schemas declare types and enums but no required arrays, so checking that fields are present remains the semantic validators' job. A weekly spec-update workflow runs update-spec --all and opens a PR when any upstream spec changes.

Where live provider responses disagree with their bundled document, a documented load-time relaxation applies instead of failing on a field the provider omits (see src/llm_comply/spec_relax.py); every relaxation is pinned by make verify.

⁠Web UI

llm-comply --web                                      # http://127.0.0.1:8090
llm-comply --web --host 0.0.0.0 \
  --tls-cert /path/cert.pem --tls-key /path/key.pem   # https://0.0.0.0:8090

The UI accepts an API key in the browser, so serve it over TLS whenever it is reachable beyond localhost. --tls-cert and --tls-key must be provided together; TLS 1.2 is the minimum.

GET /health returns the version and cumulative counters (runs_started, runs_completed, runs_aborted, runs_failed, requests_completed, client_disconnects), so runs_started reconciles with the terminal counters. A stream that raises sends a final event: error frame and increments runs_failed. Closing the browser tab aborts the run before the next test starts; the test already in flight still finishes because its API call has been sent.

⁠Docker

docker run -p 8090:8090 oaklight/llm-comply:latest

Reference compose.yaml for production deployment:

services:
  llm-comply:
    image: oaklight/llm-comply:latest
    container_name: llm-comply
    restart: unless-stopped
    ports:
      - 127.0.0.1:8090:8090
    read_only: true
    tmpfs:
      - /tmp:size=64m,exec
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    mem_limit: 256m
    cpus: 0.5

Note: The binary image uses Nuitka onefile format, which extracts to /tmp at startup. The tmpfs mount with exec is required when using read_only: true.

⁠Test Coverage

FormatTestsWhat's Validated
open_responses16Schema + semantic (lifecycle, phases, compaction, multi-turn tools, structured output, file input, thinking)
openai_responses12Schema + semantic (lifecycle, streaming events, tools, reasoning, file input)
chat_completions13Schema + semantic (choices, finish_reason, delta, stream chunks, developer role, structured output, web search, audio output, error envelope)
messages19Schema + semantic (content blocks, stop_reason, max_tokens/stop_sequence, streaming tool deltas, thinking, prompt caching, document input, tool_result round-trip, web search, code execution, structured output, token counting)
google_generate12Schema + semantic (candidates, parts, finishReason, safetyRatings, thinking, structured output, grounding, code execution)
google_interactions13Semantic only (steps, model_output, thinking, google search, code execution, maps, url context)

chat-audio-output treats the model name as a capability hint: it skips unless the name contains audio (case-insensitive), so deployments with role-named audio aliases should run the test with a name that includes it.

The built-in Google Interactions tool tests are optional: a provider that does not support a tool and answers with 400, 404, 405, 422, or 501 reports the test as skipped. A successful response that lacks the tool's output is reported as a warning, so an intrinsic tool cannot fail an otherwise compliant API. Protocol failures — a 5xx, a non-JSON body, a schema violation, or an authentication, throttling, or transport error — still fail. The Google GenAI grounding and code-execution, Anthropic web-search and code-execution, and chat web-search tests follow this rule.

⁠License

MIT

Tag summary

Content type

Image

Digest

sha256:c34c9f5e3…

Size

13.3 MB

Last updated

2 months ago

docker pull oaklight/llm-comply