dynagate-cpp/README.md

# Dynagate LLM gateway (C++ port)

Dynagate is a lightweight, high-performance LLM gateway that acts as a proxy for OpenAI-compatible APIs. This directory is a modern **C++20** port of the Go reference implementation (`dynagate`), using a native RFC 8259 JSON engine (`json.cpp`) and **libcurl** for HTTP networking.

It provides seamless model routing, automatic key and model failover, live configuration hot-reloading without restarts, token authentication, and custom header injection.

## Features

1. **Multi-API Protocol Endpoints (Stateless)**:
   - `/v1/models` and `/models` (GET): Dynamically lists unique active model IDs.
   - `/v1/chat/completions` and `/chat/completions` (POST): Proxies non-streaming and streaming (`text/event-stream`) completions.
   - `/v1/responses` and `/responses` (POST): Stateless-style OpenAI Responses API endpoint (automatically routed to upstream chat completions endpoints).
   - `/v1/messages` and `/messages` (POST): Anthropic-compatible Messages API endpoint (automatically routed to upstream chat completions endpoints).
   - `/v1/images/generations` and `/images/generations` (POST): Proxies image generation requests.
   > **Note** Dynagate operates in a completely stateless manner across all API formats (`/v1/chat/completions`, `/v1/responses`, `/v1/messages`). Dynagate does not store or persist server-side conversation state. Clients must include the full conversation history in each request for multi-turn dialogues.

2. **Zero-downtime live-reloading**:
   - Watches `models.csv` (overridable via command-line flags) continuously using a background thread and automatically reloads configuration updates without dropping active connections.

3. **Fallback and retry orchestration**:
   - Attempts keys configured for the requested model sequentially using Fibonacci backoff delays (`-retry-delay`).
   - Cascades automatically to the next model in the priority chain (wrapping around to cover all entries) if all keys for the requested model fail.
   - Updates the `"model"` field dynamically in the outgoing request body before proxying, ensuring upstream compatibility.

4. **Flexible authorization**:
   - Optional gateway token authentication via command-line key file.
   - Key options (`""`, `"-"`) to strip authorization headers when proxying to local or open upstreams.
   - Special `"-blank-"` key config to pass an empty `Authorization: Bearer` header.

5. **External config updater**:
   - Executes a periodic CLI command (`-csv-updater`) at a custom interval (`-csv-update-interval`) to fetch, generate, or download fresh model listings.

6. **Custom header injection (`extra` field)**:
   - Sets or overrides custom headers dynamically per model deployment using a JSON configuration string.

## Tech stack

- **Language**: C++20 (`-std=c++20`)
- **Dependencies**:
  - `libcurl` (HTTP/HTTPS upstream requests and SSE streaming)
  - Native RFC 8259 JSON parser and serializer in `json.cpp` (no extra or vendored third-party JSON dependency)
  - POSIX sockets and standard C++20 threads/atomics/filesystem
- **Target Platforms**: Linux, FreeBSD, macOS, Haiku, and any POSIX-compliant OS (64-bit and 32-bit hosts)

## Repository layout

```text
dynagate.hpp             Shared declarations, data structures, and interfaces
json.cpp                 Native RFC 8259 JSON parser and serializer
config.cpp               RFC 4180 CSV parser, ConfigManager, hot-reloader, CSV updater
handler.cpp              Auth, normalization, protocol conversions, libcurl proxying
server.cpp               Multi-threaded POSIX HTTP/1.1 server and streaming writer
main.cpp                 CLI flag parsing, signal handling, server lifecycle
gateway_test.cpp         Comprehensive unit and end-to-end test suite
Makefile                 POSIX-compliant Makefile (bmake, gmake, smake compatible)
models.csv               Example model configuration file
free-model-collector.sh  Free model discovery and verification script
freekeys.sh              Helper script for collecting free keys
```

## Prerequisites

- A C++20 compiler (`c++`, `g++`, or `clang++`)
- Any POSIX `make` (`bmake`, `gmake`, `make`)
- `libcurl` development headers and library:
  ```bash
  # Debian / Ubuntu
  apt-get install build-essential libcurl4-openssl-dev
  # Fedora / RHEL
  dnf install gcc-c++ libcurl-devel
  # FreeBSD
  pkg install curl
  # macOS (Xcode Command Line Tools include libcurl, or via Homebrew)
  xcode-select --install
  ```

## Getting started

### 1. Building from source

```bash
make            # builds ./dynagate
make 32bit      # builds a 32-bit ./dynagate binary with large file support
make debug      # builds ./dynagate with AddressSanitizer + UBSan
make test       # builds and runs the test suite
make clean      # removes build artifacts
make install    # installs to $(DESTDIR)/usr/local/bin/dynagate
```

If `libcurl` or custom headers live in a non-standard prefix, override flags on the `make` command line:
```bash
make CXX=clang++ CPPFLAGS="-I/opt/homebrew/include -I." LDLIBS="-L/opt/homebrew/lib -lcurl -pthread"
```

### 2. Configure models
Create a `models.csv` file in the same directory. The file supports both header and header-less formats.

**Example `models.csv` with Header**:
```csv
model,key,endpoint,extra
gpt-4o,sk-proj-key1,https://api.openai.com/v1,"{""Openai-Organization"":""org-123""}"
gpt-4o,sk-proj-key2,https://api.openai.com/v1,
gpt-3.5-turbo,-blank-,https://api.openai.com/v1,
deepseek-chat,sk-ds-key,https://api.deepseek.com/v1,"{""X-Custom-Env"":""production""}"
ollama-llama3,-,http://localhost:11434/v1,
```

### 3. Setup gateway key (optional)
To restrict access to your gateway, write one or more plaintext keys (one per line) to a file (e.g. `gatekey.txt`):
```text
my-secure-gateway-token-1
my-secure-gateway-token-2
```

### 4. Run the server
```bash
./dynagate -port 8080 -csv models.csv -key gatekey.txt
```

## Configuration reference

### Command line flags

| Flag | Type | Default | Description |
| ---- | ---- | ------- | ----------- |
| `-port` | int | `8080` | Port the gateway listens on. |
| `-csv` | string | `models.csv` | Path to the model configuration CSV. |
| `-key` | string | `""` | Path to a file containing the gateway's access tokens (one per line). If blank, authentication is disabled. |
| `-csv-updater` | string | `""` | Command/script to run periodically to update `models.csv`. |
| `-csv-update-interval`| int | `10` | Frequency in minutes to invoke `-csv-updater`. |
| `-retry-delay` | int | `100` | Base delay in milliseconds for Fibonacci failover retries (0 to disable). |

### Model CSV format rules
The gateway maps columns dynamically by looking at the header row. If no header is present, it defaults to:
- Column 0: `model` (Model ID)
- Column 1: `key` (API Key / Authorization Token)
- Column 2: `endpoint` (Upstream base URL endpoint)
- Column 3: `extra` (Optional JSON string object containing header mappings)

#### Auth override key values:
- **`""` or `"-"`**: The gateway will strip the `Authorization` header completely (useful for local LLM endpoints or unauthenticated proxies).
- **`"-blank-"`**: The gateway will attach `Authorization: Bearer` without any key appended.
- **Any other string**: The gateway will attach `Authorization: Bearer <key>`.

### Stateless Operation & Multi-Turn Conversations

Dynagate is designed as a lightweight, zero-state proxy gateway. All supported request formats (`/v1/chat/completions`, `/v1/responses`, `/v1/messages`) are processed statelessly without storing session context or conversation state on disk or in memory.

To maintain context in multi-turn conversations, clients **must include the entire conversation history** in every request payload:
- **OpenAI Chat Completions (`/v1/chat/completions`)**: Pass the full array of system, user, and assistant messages in `messages`.
- **OpenAI Responses API (`/v1/responses`)**: Pass previous conversation turns in `input` (or `messages`) along with `instructions`.
- **Anthropic Messages API (`/v1/messages`)**: Pass the full history of user and assistant messages in `messages` along with `system`.

### Manual testing with curl

#### 1. Model listing (authenticated)
```bash
curl -i -H "Authorization: Bearer my-secure-gateway-token" http://localhost:8080/v1/models
```

#### 2. Chat completions proxying
```bash
curl -i -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer my-secure-gateway-token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Explain gravity in one sentence"}]
  }'
```

#### 3. Chat completions streaming
```bash
curl -i -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer my-secure-gateway-token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "stream": true,
    "messages": [{"role": "user", "content": "Count from 1 to 5"}]
  }'
```

#### 4. OpenAI-compatible Responses API (`/v1/responses`)
```bash
curl -i -X POST http://localhost:8080/v1/responses \
  -H "Authorization: Bearer my-secure-gateway-token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "instructions": "Be concise",
    "input": "Explain relativity in one sentence"
  }'
```

#### 5. Anthropic-compatible Messages API (`/v1/messages`)
```bash
curl -i -X POST http://localhost:8080/v1/messages \
  -H "Authorization: Bearer my-secure-gateway-token" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-sonnet",
    "system": "You are a helpful assistant",
    "messages": [{"role": "user", "content": "Hello world"}]
  }'
```

## Credits

Created by Luxferre in 2026, released into the public domain with no warranties.