luxferre/dynagate
1
0
Fork 0
Code Actions Activity

initial upload

Browse Source
This commit is contained in:
Luxferre
2026-07-09 11:43:10 +03:00
commit 3ec7a19680
8 changed files with 1782 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+15
View File
@@ -0,0 +1,15 @@
.PHONY: all build clean test run
all: build
build:
go build -ldflags="-s -w" -o dynagate .
clean:
rm -f dynagate
test:
go test -v ./...
run: build
./dynagate -port 8080 -csv models.csv
README.md
+171
View File
@@ -0,0 +1,171 @@
# Dynagate LLM gateway
Dynagate is a lightweight, high-performance LLM gateway written in Go that acts as a proxy for OpenAI-compatible APIs. It provides seamless model routing, automatic key and model failover, live configuration hot-reloading without restarts, token authentication, and custom header injection with zero third-party dependencies.
## Features
1. **OpenAI-compatible endpoints**:
- `/v1/models` (GET): Dynamically lists unique active model IDs.
- `/v1/chat/completions` (POST): Proxies non-streaming and streaming (`text/event-stream`) completions.
- `/v1/images/generations` (POST): Proxies image generation requests.
2. **Zero-downtime live-reloading**:
- Watches `models.csv` (overridable via command-line flags) continuously using a background thread and automatically reloads configuration updates without dropping active connections.
3. **Fallback and retry orchestration**:
- Attempts keys configured for the requested model sequentially.
- Cascades automatically to the next model in the priority chain (wrapping around to cover all entries) if all keys for the requested model fail.
- Updates the `"model"` field dynamically in the outgoing request body before proxying, ensuring upstream compatibility.
4. **Flexible authorization**:
- Optional gateway token authentication via command-line key file.
- Key options (`""`, `"-"`) to strip authorization headers when proxying to local or open upstreams.
- Special `"-blank-"` key config to pass an empty `Authorization: Bearer` header.
5. **External config updater**:
- Executes a periodic CLI command (`-csv-updater`) at a custom interval (`-csv-update-interval`) to fetch, generate, or download fresh model listings.
6. **Custom header injection (`extra` field)**:
- Sets or overrides custom headers dynamically per model deployment using a JSON configuration string.
## Tech stack
- **Language**: Go 1.26+
- **Dependencies**: None (pure Go standard library)
- **Deployment Platform**: Linux/Docker/bare VPS (runs as a standalone static binary)
## Prerequisites
- Go 1.26 or higher (if building from source)
- Make (optional, for build commands)
- Curl (for testing API endpoints)
## Getting started
### 1. Clone and build
```bash
git clone https://codeberg.org/luxferre/dynagate.git
cd dynagate
make build
```
This produces a static binary `dynagate` (~6.5MB) in the root directory.
### 2. Configure models
Create a `models.csv` file in the same directory. The file supports both header and header-less formats.
**Example `models.csv` with Header**:
```csv
model,key,endpoint,extra
gpt-4o,sk-proj-key1,https://api.openai.com/v1,{"Openai-Organization":"org-123"}
gpt-4o,sk-proj-key2,https://api.openai.com/v1,
gpt-3.5-turbo,-blank-,https://api.openai.com/v1,
deepseek-chat,sk-ds-key,https://api.deepseek.com/v1,{"X-Custom-Env":"production"}
ollama-llama3,-,http://localhost:11434/v1,
```
### 3. Setup gateway key (optional)
To restrict access to your gateway, write a single plaintext key to a file (e.g. `gatekey.txt`):
```text
my-secure-gateway-token
```
### 4. Run the server
```bash
./dynagate -port 8080 -csv models.csv -key gatekey.txt
```
## Configuration reference
### Command line flags
| Flag | Type | Default | Description |
| ---- | ---- | ------- | ----------- |
| `-port` | int | `8080` | Port the gateway listens on. |
| `-csv` | string | `models.csv` | Path to the model configuration CSV. |
| `-key` | string | `""` | Path to a file containing the gateway's access token. If blank, authentication is disabled. |
| `-csv-updater` | string | `""` | Command/script to run periodically to update `models.csv`. |
| `-csv-update-interval`| int | `10` | Frequency in minutes to invoke `-csv-updater`. |
### Model CSV format rules
The gateway maps columns dynamically by looking at the header row. If no header is present, it defaults to:
- Column 0: `model` (Model ID)
- Column 1: `key` (API Key / Authorization Token)
- Column 2: `endpoint` (Upstream base URL endpoint)
- Column 3: `extra` (Optional JSON string object containing header mappings)
#### Auth override key values:
- **`""` or `"-"`**: The gateway will strip the `Authorization` header completely (useful for local LLM endpoints or unauthenticated proxies).
- **`"-blank-"`**: The gateway will attach exactly `Authorization: Bearer` without any key appended.
- **Any other string**: The gateway will attach `Authorization: Bearer <key>`.
### Manual testing with curl
#### 1. Model listing (authenticated)
```bash
curl -i -H "Authorization: Bearer my-secure-gateway-token" http://localhost:8080/v1/models
```
#### 2. Chat completions proxying
```bash
curl -i -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer my-secure-gateway-token" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Explain gravity in one sentence"}]
}'
```
#### 3. Chat completions streaming
```bash
curl -i -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer my-secure-gateway-token" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o",
"stream": true,
"messages": [{"role": "user", "content": "Count from 1 to 5"}]
}'
```
## Troubleshooting
### Dynamic loading failures
- **Symptom**: Config manager reports file not found warnings at startup.
- **Solution**: The gateway is built to run even if `models.csv` is initially missing. Create or write to the CSV configuration file and the hot-reloader will parse and activate it within 1 second.
### Port conflicts
- **Symptom**: `Server listen failed: listen tcp :8080: bind: address already in use`.
- **Solution**: Change the listening port using the `-port` command line flag (e.g. `./dynagate -port 9090`).
### 401 Unauthorized errors
- **Symptom**: Requests return `{"error": {"code": "invalid_api_key", ...}}`.
- **Solution**: Check that you are passing the correct Bearer token configured in the `-key` token file. Trim any whitespace, and verify it starts with `Bearer `.
## FAQ
### Why roll yet another LLM gateway?
First, security reasons. After the supply chain attacks on LiteLLM and other large gateway projects, the author stopped trusting them and wanted to build something that doesn't have any third-party dependencies. As such, Dynagate can be considered a drop-in replacement for LiteLLM proxy mode if you only need OpenAI-compatible completions and model listing endpoints.
Second, flexibility. A lot of existing LLM interaction tools, including gateways, don't support keyless or empty-key access that local inference engines or some public LLM providers offer, and also make it complicated and/or impossible to inject custom HTTP headers into outgoing requests.
Third, simplicity. Dynagate was designed with instant failover (without any extra setup) and instant configuration reload in mind, but without any extra overhead. This makes it much faster than even e.g. Bifrost, which is also written in Go and considered pretty fast on its own.
Fourth, pluggable external commands to **dynamically** update the entire model configuration, running in a specified interval. This is something yet unheard in any existing LLM gateway.
### Is it compatible with OpenClaw?
Yes, but that's not tested yet. It is 100% compatible at least with [PicoClaw](https://github.com/sipeed/picoclaw), and was particularly tuned to address PicoClaw's incompatibility with keyless LLM setups, and these two services work together very nicely. The author personally uses Dynagate + PicoClaw setup on a quite weak fileserver machine in combination with free-tier (keyless access) OpenCode Zen models (see the included [models.csv](models.csv) as an example).
To configure models accessible through Dynagate in PicoClaw, select "LiteLLM" as a provider, then use `[your_dynagate_url]/v1` as an endpoint. Specify any string as an API key if you didn't enable authentication in Dynagate.
### Why was CSV format chosen for model configuration (as opposed to JSON, YAML, TOML etc)?
The entire initial idea behind Dynagate was for it to be able to accept dynamically generated configs on the fly. The means of generating these configs may be as straighforward as using POSIX AWK, command-line tools like Miller and/or plain shell scripts. The only format simpler than CSV to fulfill this task would be TSV (tab-separated values), but it is prone to errors when manually copying the contents between environments may inadvertently convert tab characters to spaces. As a universal tabular data format, CSV is also widely supported by various spreadsheet applications, where one may save current model setup, exporting it to `models.csv` as necessary without even having to restart the gateway.
Additionally, as you may have seen in the example, you can specify the same model ID with different keys and endpoints, which will automatically enable same-model failover. With a hierarchical format like JSON, this setup would require extra parameter nesting or some other complications, or just making the config look much bulkier than it should.
## Credits
Created by Luxferre in 2026, released into the public domain with no warranties.
config.go
+218
View File
@@ -0,0 +1,218 @@
// Dynagate config manager
package main
import (
"context"
"encoding/csv"
"log"
"os"
"strings"
"sync"
"time"
)
type ModelConfig struct {
Model string
Key string
Endpoint string
Extra string
}
type ConfigManager struct {
filePath string
mu sync.RWMutex
configs []ModelConfig
uniqueModels []string
lastModTime time.Time
lastSize int64
}
func NewConfigManager(filePath string) *ConfigManager {
cm := &ConfigManager{
filePath: filePath,
}
if err := cm.Load(); err != nil {
log.Printf("Initial configuration load failed or file not found (will retry on change): %v", err)
}
return cm
}
func (cm *ConfigManager) Load() error {
file, err := os.Open(cm.filePath)
if err != nil {
return err
}
defer file.Close()
stat, err := file.Stat()
if err != nil {
return err
}
reader := csv.NewReader(file)
reader.FieldsPerRecord = -1
rows, err := reader.ReadAll()
if err != nil {
return err
}
var configs []ModelConfig
if len(rows) == 0 {
cm.update(configs, stat.ModTime(), stat.Size())
return nil
}
// Detect header row
hasHeader := false
firstRow := rows[0]
for _, col := range firstRow {
colLower := strings.ToLower(strings.TrimSpace(col))
if colLower == "model" || colLower == "key" || colLower == "endpoint" {
hasHeader = true
break
}
}
var startIdx int
modelIdx, keyIdx, endpointIdx, extraIdx := 0, 1, 2, -1
if hasHeader {
startIdx = 1
modelIdx, keyIdx, endpointIdx, extraIdx = -1, -1, -1, -1
for i, col := range firstRow {
colLower := strings.ToLower(strings.TrimSpace(col))
switch colLower {
case "model":
modelIdx = i
case "key":
keyIdx = i
case "endpoint":
endpointIdx = i
case "extra":
extraIdx = i
}
}
// Fallbacks if columns are not named exactly
if modelIdx == -1 {
modelIdx = 0
}
if keyIdx == -1 {
keyIdx = 1
}
if endpointIdx == -1 {
endpointIdx = 2
}
} else {
startIdx = 0
}
for i := startIdx; i < len(rows); i++ {
row := rows[i]
maxIdx := modelIdx
if keyIdx > maxIdx {
maxIdx = keyIdx
}
if endpointIdx > maxIdx {
maxIdx = endpointIdx
}
if len(row) <= maxIdx {
continue
}
m := strings.TrimSpace(row[modelIdx])
k := strings.TrimSpace(row[keyIdx])
ep := strings.TrimSpace(row[endpointIdx])
if m == "" || k == "" || ep == "" {
continue
}
extra := ""
if hasHeader {
if extraIdx != -1 && len(row) > extraIdx {
extra = strings.TrimSpace(row[extraIdx])
}
} else {
if len(row) > 3 {
extra = strings.TrimSpace(row[3])
}
}
configs = append(configs, ModelConfig{
Model: m,
Key: k,
Endpoint: ep,
Extra: extra,
})
}
cm.update(configs, stat.ModTime(), stat.Size())
log.Printf("Loaded %d model configurations from %s", len(configs), cm.filePath)
return nil
}
func (cm *ConfigManager) update(configs []ModelConfig, modTime time.Time, size int64) {
cm.mu.Lock()
defer cm.mu.Unlock()
cm.configs = configs
cm.lastModTime = modTime
cm.lastSize = size
var uniqueModels []string
seen := make(map[string]bool)
for _, cfg := range configs {
if !seen[cfg.Model] {
seen[cfg.Model] = true
uniqueModels = append(uniqueModels, cfg.Model)
}
}
cm.uniqueModels = uniqueModels
}
func (cm *ConfigManager) GetConfigs() []ModelConfig {
cm.mu.RLock()
defer cm.mu.RUnlock()
res := make([]ModelConfig, len(cm.configs))
copy(res, cm.configs)
return res
}
func (cm *ConfigManager) GetUniqueModels() []string {
cm.mu.RLock()
defer cm.mu.RUnlock()
res := make([]string, len(cm.uniqueModels))
copy(res, cm.uniqueModels)
return res
}
func (cm *ConfigManager) Watch(ctx context.Context) {
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
stat, err := os.Stat(cm.filePath)
if err != nil {
continue
}
cm.mu.RLock()
lastMod := cm.lastModTime
lastSz := cm.lastSize
cm.mu.RUnlock()
if stat.ModTime().After(lastMod) || stat.Size() != lastSz {
log.Printf("Detected changes in %s, reloading...", cm.filePath)
if err := cm.Load(); err != nil {
log.Printf("Error reloading configuration: %v", err)
} else {
log.Printf("Configuration reloaded successfully")
}
}
}
}
}
gateway_test.go
+689
View File
@@ -0,0 +1,689 @@
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
func TestURLBuilding(t *testing.T) {
tests := []struct {
endpoint string
reqPath string
expected string
}{
{"https://api.openai.com/v1", "/v1/chat/completions", "https://api.openai.com/v1/chat/completions"},
{"https://api.openai.com/v1/", "/v1/chat/completions", "https://api.openai.com/v1/chat/completions"},
{"https://api.openai.com", "/v1/chat/completions", "https://api.openai.com/v1/chat/completions"},
{"http://localhost:8000/api/v1", "/v1/chat/completions", "http://localhost:8000/api/v1/chat/completions"},
{"http://localhost:11434", "/v1/chat/completions", "http://localhost:11434/v1/chat/completions"},
{"https://custom.endpoint/v1/custom", "/v1/chat/completions", "https://custom.endpoint/v1/custom/v1/chat/completions"},
}
for _, tc := range tests {
t.Run(tc.endpoint+"+"+tc.reqPath, func(t *testing.T) {
got := buildURL(tc.endpoint, tc.reqPath)
if got != tc.expected {
t.Errorf("buildURL(%q, %q) = %q; want %q", tc.endpoint, tc.reqPath, got, tc.expected)
}
})
}
}
func TestTrialOrdering(t *testing.T) {
configs := []ModelConfig{
{Model: "gpt-4", Key: "key-a1", Endpoint: "ep-a"},
{Model: "gpt-4", Key: "key-a2", Endpoint: "ep-a"},
{Model: "gpt-3.5", Key: "key-b1", Endpoint: "ep-b"},
{Model: "claude", Key: "key-c1", Endpoint: "ep-c"},
}
uniqueModels := []string{"gpt-4", "gpt-3.5", "claude"}
// Test case 1: Requested model matches the first model "gpt-4"
{
trials := getTrialConfigs(configs, uniqueModels, "gpt-4")
expected := []ModelConfig{
{Model: "gpt-4", Key: "key-a1", Endpoint: "ep-a"},
{Model: "gpt-4", Key: "key-a2", Endpoint: "ep-a"},
{Model: "gpt-3.5", Key: "key-b1", Endpoint: "ep-b"},
{Model: "claude", Key: "key-c1", Endpoint: "ep-c"},
}
if !reflect.DeepEqual(trials, expected) {
t.Errorf("Trial ordering for gpt-4 mismatch.\nGot: %+v\nWant: %+v", trials, expected)
}
}
// Test case 2: Requested model matches the second model "gpt-3.5"
{
trials := getTrialConfigs(configs, uniqueModels, "gpt-3.5")
expected := []ModelConfig{
{Model: "gpt-3.5", Key: "key-b1", Endpoint: "ep-b"},
{Model: "claude", Key: "key-c1", Endpoint: "ep-c"},
{Model: "gpt-4", Key: "key-a1", Endpoint: "ep-a"},
{Model: "gpt-4", Key: "key-a2", Endpoint: "ep-a"},
}
if !reflect.DeepEqual(trials, expected) {
t.Errorf("Trial ordering for gpt-3.5 mismatch.\nGot: %+v\nWant: %+v", trials, expected)
}
}
// Test case 3: Requested model matches the last model "claude"
{
trials := getTrialConfigs(configs, uniqueModels, "claude")
expected := []ModelConfig{
{Model: "claude", Key: "key-c1", Endpoint: "ep-c"},
{Model: "gpt-4", Key: "key-a1", Endpoint: "ep-a"},
{Model: "gpt-4", Key: "key-a2", Endpoint: "ep-a"},
{Model: "gpt-3.5", Key: "key-b1", Endpoint: "ep-b"},
}
if !reflect.DeepEqual(trials, expected) {
t.Errorf("Trial ordering for claude mismatch.\nGot: %+v\nWant: %+v", trials, expected)
}
}
// Test case 4: Requested model doesn't match any config
{
trials := getTrialConfigs(configs, uniqueModels, "non-existent")
expected := []ModelConfig{
{Model: "gpt-4", Key: "key-a1", Endpoint: "ep-a"},
{Model: "gpt-4", Key: "key-a2", Endpoint: "ep-a"},
{Model: "gpt-3.5", Key: "key-b1", Endpoint: "ep-b"},
{Model: "claude", Key: "key-c1", Endpoint: "ep-c"},
}
if !reflect.DeepEqual(trials, expected) {
t.Errorf("Trial ordering for non-existent model mismatch.\nGot: %+v\nWant: %+v", trials, expected)
}
}
}
func TestConfigManagerCSVWatcher(t *testing.T) {
tmpDir, err := os.MkdirTemp("", "dynagate-test")
if err != nil {
t.Fatalf("failed to create temp dir: %v", err)
}
defer os.RemoveAll(tmpDir)
csvFile := filepath.Join(tmpDir, "models.csv")
// Write initial content
initialContent := `model,key,endpoint
gpt-4,key-1,http://localhost:8001
gpt-3.5,key-2,http://localhost:8002
`
if err := os.WriteFile(csvFile, []byte(initialContent), 0644); err != nil {
t.Fatalf("failed to write csv file: %v", err)
}
cm := NewConfigManager(csvFile)
configs := cm.GetConfigs()
if len(configs) != 2 || configs[0].Model != "gpt-4" || configs[1].Model != "gpt-3.5" {
t.Fatalf("unexpected initial configs: %+v", configs)
}
// Start watching in background
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go cm.Watch(ctx)
// Wait briefly, modify file, and check if reload occurs
time.Sleep(100 * time.Millisecond)
updatedContent := `model,key,endpoint
gpt-4,key-1,http://localhost:8001
gpt-3.5,key-2,http://localhost:8002
claude,key-3,http://localhost:8003
`
// Make sure we sleep slightly to ensure file modtime actually changes on modern filesystems
time.Sleep(1 * time.Second)
if err := os.WriteFile(csvFile, []byte(updatedContent), 0644); err != nil {
t.Fatalf("failed to update csv file: %v", err)
}
// Poll GetConfigs until it gets updated (timeout after 3 seconds)
deadline := time.Now().Add(3 * time.Second)
var updatedConfigs []ModelConfig
for time.Now().Before(deadline) {
updatedConfigs = cm.GetConfigs()
if len(updatedConfigs) == 3 {
break
}
time.Sleep(100 * time.Millisecond)
}
if len(updatedConfigs) != 3 || updatedConfigs[2].Model != "claude" {
t.Errorf("Hot-reload failed. Got configs: %+v", updatedConfigs)
}
}
func TestGatewayCompletionsFailover(t *testing.T) {
// We will setup 3 mock upstream servers.
// Server A: Always returns 500
// Server B: Always returns 401
// Server C: Succeeds and returns a chat completion response
serverA := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
_, _ = w.Write([]byte(`{"error": "Internal Server Error A"}`))
}))
defer serverA.Close()
serverB := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusUnauthorized)
_, _ = w.Write([]byte(`{"error": "Unauthorized B"}`))
}))
defer serverB.Close()
serverC := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Read request body to verify model was modified
var body map[string]any
_ = json.NewDecoder(r.Body).Decode(&body)
modelSent := body["model"].(string)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(fmt.Sprintf(`{"choices":[{"message":{"content":"Hello from %s"}}]}`, modelSent)))
}))
defer serverC.Close()
// Configure gateway ConfigManager with endpoints pointing to these servers
// We want to verify that when we call completions:
// - First it tries model-1 (pointing to serverA) -> fails
// - Then tries model-1 next key (pointing to serverB) -> fails
// - Then retries with next model model-2 (pointing to serverC) -> succeeds!
configs := []ModelConfig{
{Model: "model-1", Key: "key-fail-1", Endpoint: serverA.URL},
{Model: "model-1", Key: "key-fail-2", Endpoint: serverB.URL},
{Model: "model-2", Key: "key-success", Endpoint: serverC.URL},
}
cm := &ConfigManager{
configs: configs,
uniqueModels: []string{"model-1", "model-2"},
}
handler := handleChatCompletions(cm, "")
// 1. Test non-streaming failover request starting with model-1
reqBodyObj := map[string]any{
"model": "model-1",
"messages": []map[string]string{
{"role": "user", "content": "Hi"},
},
}
reqBytes, _ := json.Marshal(reqBodyObj)
req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewReader(reqBytes))
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
resp := w.Result()
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("Expected status 200, got %d", resp.StatusCode)
}
var respObj map[string]any
_ = json.NewDecoder(resp.Body).Decode(&respObj)
choices, ok := respObj["choices"].([]any)
if !ok || len(choices) == 0 {
t.Fatalf("Invalid response structure: %+v", respObj)
}
content := choices[0].(map[string]any)["message"].(map[string]any)["content"].(string)
expectedContent := "Hello from model-2"
if content != expectedContent {
t.Errorf("Expected content %q, got %q (failover to model-2 did not happen or model was not updated)", expectedContent, content)
}
}
func TestGatewayCompletionsStreamingFailover(t *testing.T) {
// We will setup 2 mock upstream servers.
// Server A: Always returns 500
// Server B: Streams chat completions chunks
serverA := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer serverA.Close()
serverB := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.WriteHeader(http.StatusOK)
flusher, _ := w.(http.Flusher)
_, _ = w.Write([]byte("data: chunk1\n\n"))
flusher.Flush()
_, _ = w.Write([]byte("data: chunk2\n\n"))
flusher.Flush()
}))
defer serverB.Close()
configs := []ModelConfig{
{Model: "model-1", Key: "key-fail", Endpoint: serverA.URL},
{Model: "model-2", Key: "key-success", Endpoint: serverB.URL},
}
cm := &ConfigManager{
configs: configs,
uniqueModels: []string{"model-1", "model-2"},
}
handler := handleChatCompletions(cm, "")
reqBodyObj := map[string]any{
"model": "model-1",
"stream": true,
"messages": []map[string]string{
{"role": "user", "content": "Hi"},
},
}
reqBytes, _ := json.Marshal(reqBodyObj)
req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewReader(reqBytes))
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
resp := w.Result()
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("Expected status 200, got %d", resp.StatusCode)
}
respBytes, _ := io.ReadAll(resp.Body)
respStr := string(respBytes)
if !strings.Contains(respStr, "data: chunk1") || !strings.Contains(respStr, "data: chunk2") {
t.Errorf("Expected streaming chunks to be forwarded, got: %q", respStr)
}
}
func TestModelsEndpoint(t *testing.T) {
configs := []ModelConfig{
{Model: "model-1", Key: "key-1", Endpoint: "ep-1"},
{Model: "model-1", Key: "key-2", Endpoint: "ep-1"},
{Model: "model-2", Key: "key-3", Endpoint: "ep-2"},
}
cm := &ConfigManager{
configs: configs,
uniqueModels: []string{"model-1", "model-2"},
}
handler := handleModels(cm, "")
req := httptest.NewRequest("GET", "/v1/models", nil)
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
resp := w.Result()
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("Expected status 200, got %d", resp.StatusCode)
}
var respObj map[string]any
_ = json.NewDecoder(resp.Body).Decode(&respObj)
data, ok := respObj["data"].([]any)
if !ok || len(data) != 2 {
t.Fatalf("Expected 2 models in response data, got: %+v", respObj)
}
model1 := data[0].(map[string]any)["id"].(string)
model2 := data[1].(map[string]any)["id"].(string)
if model1 != "model-1" || model2 != "model-2" {
t.Errorf("Unexpected models. Got: %s, %s", model1, model2)
}
}
func TestGatewayAuthentication(t *testing.T) {
configs := []ModelConfig{
{Model: "model-1", Key: "key-1", Endpoint: "ep-1"},
}
cm := &ConfigManager{
configs: configs,
uniqueModels: []string{"model-1"},
}
// Set up handlers expecting "secret-token"
modelsHandler := handleModels(cm, "secret-token")
completionsHandler := handleChatCompletions(cm, "secret-token")
// Case 1: No Authorization header
{
req := httptest.NewRequest("GET", "/v1/models", nil)
w := httptest.NewRecorder()
modelsHandler.ServeHTTP(w, req)
if w.Result().StatusCode != http.StatusUnauthorized {
t.Errorf("Expected 401 Unauthorized for missing token, got %d", w.Result().StatusCode)
}
}
// Case 2: Incorrect Authorization header
{
req := httptest.NewRequest("GET", "/v1/models", nil)
req.Header.Set("Authorization", "Bearer wrong-token")
w := httptest.NewRecorder()
modelsHandler.ServeHTTP(w, req)
if w.Result().StatusCode != http.StatusUnauthorized {
t.Errorf("Expected 401 Unauthorized for wrong token, got %d", w.Result().StatusCode)
}
}
// Case 3: Correct Authorization header
{
req := httptest.NewRequest("GET", "/v1/models", nil)
req.Header.Set("Authorization", "Bearer secret-token")
w := httptest.NewRecorder()
modelsHandler.ServeHTTP(w, req)
if w.Result().StatusCode != http.StatusOK {
t.Errorf("Expected 200 OK for correct token, got %d", w.Result().StatusCode)
}
}
// Case 4: Completions endpoint with correct token
{
reqBodyObj := map[string]any{
"model": "model-1",
}
reqBytes, _ := json.Marshal(reqBodyObj)
req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewReader(reqBytes))
req.Header.Set("Authorization", "Bearer secret-token")
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
completionsHandler.ServeHTTP(w, req)
// It should proceed past auth and return 502/503/500 because "ep-1" is not a valid endpoint,
// but critically, it should NOT return 401.
status := w.Result().StatusCode
if status == http.StatusUnauthorized {
t.Errorf("Expected completions request to pass authentication, but got 401 Unauthorized")
}
}
}
func TestCSVUpdater(t *testing.T) {
tmpDir, err := os.MkdirTemp("", "dynagate-updater-test")
if err != nil {
t.Fatalf("failed to create temp dir: %v", err)
}
defer os.RemoveAll(tmpDir)
csvFile := filepath.Join(tmpDir, "models.csv")
initialContent := "model,key,endpoint\ngpt-4,key-1,http://localhost:8001\n"
if err := os.WriteFile(csvFile, []byte(initialContent), 0644); err != nil {
t.Fatalf("failed to write csv file: %v", err)
}
// Create a command to append a model row to the CSV file
command := fmt.Sprintf("echo 'gpt-3.5,key-2,http://localhost:8002' >> %s", csvFile)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Start background updater with a fast interval (100ms)
go startCSVUpdater(ctx, command, 100*time.Millisecond)
// The updater runs once immediately at startup, so we expect the file to have the appended line
// almost immediately. Let's poll to check.
deadline := time.Now().Add(2 * time.Second)
var contentBytes []byte
var containsNewModel bool
for time.Now().Before(deadline) {
contentBytes, err = os.ReadFile(csvFile)
if err == nil && strings.Contains(string(contentBytes), "gpt-3.5") {
containsNewModel = true
break
}
time.Sleep(50 * time.Millisecond)
}
if !containsNewModel {
t.Fatalf("Expected CSV file to contain the updater-appended model, but it didn't. Content: %s", string(contentBytes))
}
}
func TestGatewayImageGenerations(t *testing.T) {
// We will setup a mock upstream server that handles image generation.
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/images/generations" {
w.WriteHeader(http.StatusNotFound)
return
}
var body map[string]any
_ = json.NewDecoder(r.Body).Decode(&body)
modelSent := body["model"].(string)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(fmt.Sprintf(`{"created":1589478378,"data":[{"url":"http://image-url/generated-by-%s"}]}`, modelSent)))
}))
defer server.Close()
configs := []ModelConfig{
{Model: "dall-e-3", Key: "dalle-key", Endpoint: server.URL},
}
cm := &ConfigManager{
configs: configs,
uniqueModels: []string{"dall-e-3"},
}
handler := handleImageGenerations(cm, "")
reqBodyObj := map[string]any{
"prompt": "a beautiful kitten",
"model": "dall-e-3",
}
reqBytes, _ := json.Marshal(reqBodyObj)
req := httptest.NewRequest("POST", "/v1/images/generations", bytes.NewReader(reqBytes))
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
resp := w.Result()
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("Expected status 200, got %d", resp.StatusCode)
}
var respObj map[string]any
_ = json.NewDecoder(resp.Body).Decode(&respObj)
data, ok := respObj["data"].([]any)
if !ok || len(data) == 0 {
t.Fatalf("Expected data field to contain image urls, got: %+v", respObj)
}
urlVal := data[0].(map[string]any)["url"].(string)
expectedURL := "http://image-url/generated-by-dall-e-3"
if urlVal != expectedURL {
t.Errorf("Expected URL %q, got %q", expectedURL, urlVal)
}
}
func TestGatewayNoAuthHeaderOnEmptyKey(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if auth := r.Header.Get("Authorization"); auth != "" {
t.Errorf("Expected no Authorization header, but got %q", auth)
w.WriteHeader(http.StatusBadRequest)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"Hello"}}]}n`))
}))
defer server.Close()
configs := []ModelConfig{
{Model: "model-no-key", Key: "", Endpoint: server.URL},
{Model: "model-no-key-2", Key: "-", Endpoint: server.URL},
}
cm := &ConfigManager{
configs: configs,
uniqueModels: []string{"model-no-key", "model-no-key-2"},
}
handler := handleChatCompletions(cm, "")
// Case 1: Empty Key
{
reqBodyObj := map[string]any{
"model": "model-no-key",
"messages": []map[string]string{
{"role": "user", "content": "Hi"},
},
}
reqBytes, _ := json.Marshal(reqBodyObj)
req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewReader(reqBytes))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer some-client-token")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
if w.Result().StatusCode != http.StatusOK {
t.Errorf("Expected status 200 for empty key, got %d", w.Result().StatusCode)
}
}
// Case 2: Dash Key
{
reqBodyObj := map[string]any{
"model": "model-no-key-2",
"messages": []map[string]string{
{"role": "user", "content": "Hi"},
},
}
reqBytes, _ := json.Marshal(reqBodyObj)
req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewReader(reqBytes))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer some-client-token")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
if w.Result().StatusCode != http.StatusOK {
t.Errorf("Expected status 200 for dash key, got %d", w.Result().StatusCode)
}
}
}
func TestGatewayBlankKey(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
auth := r.Header.Get("Authorization")
if auth != "Bearer" {
t.Errorf("Expected Authorization header to be exactly 'Bearer', but got %q", auth)
w.WriteHeader(http.StatusBadRequest)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"Hello"}}]}n`))
}))
defer server.Close()
configs := []ModelConfig{
{Model: "model-blank-key", Key: "-blank-", Endpoint: server.URL},
}
cm := &ConfigManager{
configs: configs,
uniqueModels: []string{"model-blank-key"},
}
handler := handleChatCompletions(cm, "")
reqBodyObj := map[string]any{
"model": "model-blank-key",
"messages": []map[string]string{
{"role": "user", "content": "Hi"},
},
}
reqBytes, _ := json.Marshal(reqBodyObj)
req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewReader(reqBytes))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer some-client-token")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
if w.Result().StatusCode != http.StatusOK {
t.Errorf("Expected status 200 for -blank- key, got %d", w.Result().StatusCode)
}
}
func TestGatewayExtraHeaders(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if h := r.Header.Get("X-My-Custom-Header"); h != "HeaderValue" {
t.Errorf("Expected X-My-Custom-Header to be 'HeaderValue', got %q", h)
w.WriteHeader(http.StatusBadRequest)
return
}
if h := r.Header.Get("X-Another-Header"); h != "123" {
t.Errorf("Expected X-Another-Header to be '123', got %q", h)
w.WriteHeader(http.StatusBadRequest)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"Hello"}}]}n`))
}))
defer server.Close()
configs := []ModelConfig{
{Model: "model-extra-headers", Key: "my-key", Endpoint: server.URL, Extra: `{"X-My-Custom-Header":"HeaderValue","X-Another-Header":123}`},
}
cm := &ConfigManager{
configs: configs,
uniqueModels: []string{"model-extra-headers"},
}
handler := handleChatCompletions(cm, "")
reqBodyObj := map[string]any{
"model": "model-extra-headers",
"messages": []map[string]string{
{"role": "user", "content": "Hi"},
},
}
reqBytes, _ := json.Marshal(reqBodyObj)
req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewReader(reqBytes))
req.Header.Set("Content-Type", "application/json")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
if w.Result().StatusCode != http.StatusOK {
t.Errorf("Expected status 200, got %d", w.Result().StatusCode)
}
}
go.mod
+3
View File
@@ -0,0 +1,3 @@
module dynagate
go 1.26.4
handler.go
+552
View File
@@ -0,0 +1,552 @@
// Dynagate LLM request proxying handlers
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"strings"
)
var httpClient = &http.Client{}
func checkAuth(expectedToken string, r *http.Request) bool {
if expectedToken == "" {
return true
}
authHeader := r.Header.Get("Authorization")
if !strings.HasPrefix(authHeader, "Bearer ") {
return false
}
token := strings.TrimPrefix(authHeader, "Bearer ")
return token == expectedToken
}
func sendUnauthorized(w http.ResponseWriter) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusUnauthorized)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "Incorrect API key provided.",
"type": "invalid_request_error",
"param": nil,
"code": "invalid_api_key",
},
})
}
func handleModels(cm *ConfigManager, expectedToken string) http.HandlerFunc {
type ModelData struct {
ID string `json:"id"`
Object string `json:"object"`
Created int64 `json:"created"`
OwnedBy string `json:"owned_by"`
}
type ModelsResponse struct {
Object string `json:"object"`
Data []ModelData `json:"data"`
}
return func(w http.ResponseWriter, r *http.Request) {
if !checkAuth(expectedToken, r) {
sendUnauthorized(w)
return
}
if r.Method != http.MethodGet {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusMethodNotAllowed)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "Method not allowed",
"type": "invalid_request_error",
},
})
return
}
models := cm.GetUniqueModels()
data := make([]ModelData, len(models))
for i, m := range models {
data[i] = ModelData{
ID: m,
Object: "model",
Created: 1686935002,
OwnedBy: "dynagate",
}
}
resp := ModelsResponse{
Object: "list",
Data: data,
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(resp)
}
}
func handleChatCompletions(cm *ConfigManager, expectedToken string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if !checkAuth(expectedToken, r) {
sendUnauthorized(w)
return
}
if r.Method != http.MethodPost {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusMethodNotAllowed)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "Method not allowed",
"type": "invalid_request_error",
},
})
return
}
bodyBytes, err := io.ReadAll(r.Body)
if err != nil {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "Failed to read request body",
"type": "invalid_request_error",
},
})
return
}
var bodyMap map[string]any
if err := json.Unmarshal(bodyBytes, &bodyMap); err != nil {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "Invalid JSON in request body",
"type": "invalid_request_error",
},
})
return
}
var requestedModel string
if m, ok := bodyMap["model"]; ok {
if s, ok := m.(string); ok {
requestedModel = s
}
}
configs := cm.GetConfigs()
uniqueModels := cm.GetUniqueModels()
if len(configs) == 0 {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "No model configurations loaded",
"type": "gateway_error",
},
})
return
}
trialConfigs := getTrialConfigs(configs, uniqueModels, requestedModel)
if len(trialConfigs) == 0 {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "No valid trial configuration candidates",
"type": "gateway_error",
},
})
return
}
var isStream bool
if s, ok := bodyMap["stream"]; ok {
if b, ok := s.(bool); ok {
isStream = b
}
}
log.Printf("Received completion request for model %q (stream=%t). Found %d config trials.", requestedModel, isStream, len(trialConfigs))
for i, trial := range trialConfigs {
log.Printf("Trial %d/%d: model=%s endpoint=%s key_len=%d", i+1, len(trialConfigs), trial.Model, trial.Endpoint, len(trial.Key))
bodyMap["model"] = trial.Model
modifiedBody, err := json.Marshal(bodyMap)
if err != nil {
log.Printf("Trial %d: Failed to marshal body for %s: %v", i+1, trial.Model, err)
continue
}
targetURL := buildURL(trial.Endpoint, r.URL.Path)
outReq, err := http.NewRequestWithContext(r.Context(), "POST", targetURL, bytes.NewReader(modifiedBody))
if err != nil {
log.Printf("Trial %d: Failed to create outgoing request to %s: %v", i+1, targetURL, err)
continue
}
// Copy headers from incoming request, excluding host and auth
for k, vv := range r.Header {
kLower := strings.ToLower(k)
if kLower == "authorization" || kLower == "host" || kLower == "content-length" {
continue
}
for _, v := range vv {
outReq.Header.Add(k, v)
}
}
if trial.Key == "-blank-" {
outReq.Header.Set("Authorization", "Bearer")
} else if trial.Key != "" && trial.Key != "-" {
outReq.Header.Set("Authorization", "Bearer "+trial.Key)
}
outReq.Header.Set("Content-Type", "application/json")
if trial.Extra != "" {
var extraHeaders map[string]any
if err := json.Unmarshal([]byte(trial.Extra), &extraHeaders); err != nil {
log.Printf("Trial %d: Failed to parse extra headers JSON: %v", i+1, err)
} else {
for hk, hv := range extraHeaders {
var valStr string
switch v := hv.(type) {
case string:
valStr = v
default:
valStr = fmt.Sprintf("%v", v)
}
outReq.Header.Set(hk, valStr)
}
}
}
resp, err := httpClient.Do(outReq)
if err != nil {
log.Printf("Trial %d: Request to %s failed: %v", i+1, targetURL, err)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
errBody, _ := io.ReadAll(io.LimitReader(resp.Body, 1024))
resp.Body.Close()
log.Printf("Trial %d: Request to %s returned error status %d: %s", i+1, targetURL, resp.StatusCode, strings.TrimSpace(string(errBody)))
continue
}
log.Printf("Trial %d: Connection established with status %d. Proxying response.", i+1, resp.StatusCode)
if !isStream {
respBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
log.Printf("Trial %d: Failed to read non-streaming response body: %v", i+1, readErr)
continue
}
for k, vv := range resp.Header {
for _, v := range vv {
w.Header().Add(k, v)
}
}
w.WriteHeader(resp.StatusCode)
_, _ = w.Write(respBody)
return
}
// Handle streaming
defer resp.Body.Close()
flusher, ok := w.(http.Flusher)
if !ok {
log.Printf("Trial %d: Flusher not supported on current ResponseWriter", i+1)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusInternalServerError)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "Response flusher not supported",
"type": "gateway_error",
},
})
return
}
for k, vv := range resp.Header {
for _, v := range vv {
w.Header().Add(k, v)
}
}
w.WriteHeader(resp.StatusCode)
flusher.Flush()
buf := make([]byte, 4096)
for {
n, rErr := resp.Body.Read(buf)
if n > 0 {
if _, wErr := w.Write(buf[:n]); wErr != nil {
log.Printf("Trial %d: Client disconnected or write error: %v", i+1, wErr)
return
}
flusher.Flush()
}
if rErr != nil {
if rErr != io.EOF {
log.Printf("Trial %d: Error reading response stream: %v", i+1, rErr)
}
break
}
}
return
}
log.Printf("All trials failed. Returning Bad Gateway.")
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadGateway)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "All configured models and keys failed to respond.",
"type": "gateway_error",
"param": nil,
"code": "all_endpoints_failed",
},
})
}
}
func handleImageGenerations(cm *ConfigManager, expectedToken string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if !checkAuth(expectedToken, r) {
sendUnauthorized(w)
return
}
if r.Method != http.MethodPost {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusMethodNotAllowed)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "Method not allowed",
"type": "invalid_request_error",
},
})
return
}
bodyBytes, err := io.ReadAll(r.Body)
if err != nil {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "Failed to read request body",
"type": "invalid_request_error",
},
})
return
}
var bodyMap map[string]any
if err := json.Unmarshal(bodyBytes, &bodyMap); err != nil {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadRequest)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "Invalid JSON in request body",
"type": "invalid_request_error",
},
})
return
}
var requestedModel string
if m, ok := bodyMap["model"]; ok {
if s, ok := m.(string); ok {
requestedModel = s
}
}
configs := cm.GetConfigs()
uniqueModels := cm.GetUniqueModels()
if len(configs) == 0 {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "No model configurations loaded",
"type": "gateway_error",
},
})
return
}
trialConfigs := getTrialConfigs(configs, uniqueModels, requestedModel)
if len(trialConfigs) == 0 {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "No valid trial configuration candidates",
"type": "gateway_error",
},
})
return
}
log.Printf("Received image generation request for model %q. Found %d config trials.", requestedModel, len(trialConfigs))
for i, trial := range trialConfigs {
log.Printf("Trial %d/%d: model=%s endpoint=%s key_len=%d", i+1, len(trialConfigs), trial.Model, trial.Endpoint, len(trial.Key))
bodyMap["model"] = trial.Model
modifiedBody, err := json.Marshal(bodyMap)
if err != nil {
log.Printf("Trial %d: Failed to marshal body for %s: %v", i+1, trial.Model, err)
continue
}
targetURL := buildURL(trial.Endpoint, r.URL.Path)
outReq, err := http.NewRequestWithContext(r.Context(), "POST", targetURL, bytes.NewReader(modifiedBody))
if err != nil {
log.Printf("Trial %d: Failed to create outgoing request to %s: %v", i+1, targetURL, err)
continue
}
for k, vv := range r.Header {
kLower := strings.ToLower(k)
if kLower == "authorization" || kLower == "host" || kLower == "content-length" {
continue
}
for _, v := range vv {
outReq.Header.Add(k, v)
}
}
if trial.Key == "-blank-" {
outReq.Header.Set("Authorization", "Bearer")
} else if trial.Key != "" && trial.Key != "-" {
outReq.Header.Set("Authorization", "Bearer "+trial.Key)
}
outReq.Header.Set("Content-Type", "application/json")
if trial.Extra != "" {
var extraHeaders map[string]any
if err := json.Unmarshal([]byte(trial.Extra), &extraHeaders); err != nil {
log.Printf("Trial %d: Failed to parse extra headers JSON: %v", i+1, err)
} else {
for hk, hv := range extraHeaders {
var valStr string
switch v := hv.(type) {
case string:
valStr = v
default:
valStr = fmt.Sprintf("%v", v)
}
outReq.Header.Set(hk, valStr)
}
}
}
resp, err := httpClient.Do(outReq)
if err != nil {
log.Printf("Trial %d: Request to %s failed: %v", i+1, targetURL, err)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
errBody, _ := io.ReadAll(io.LimitReader(resp.Body, 1024))
resp.Body.Close()
log.Printf("Trial %d: Request to %s returned error status %d: %s", i+1, targetURL, resp.StatusCode, strings.TrimSpace(string(errBody)))
continue
}
log.Printf("Trial %d: Connection established with status %d. Proxying response.", i+1, resp.StatusCode)
respBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
log.Printf("Trial %d: Failed to read response body: %v", i+1, readErr)
continue
}
for k, vv := range resp.Header {
for _, v := range vv {
w.Header().Add(k, v)
}
}
w.WriteHeader(resp.StatusCode)
_, _ = w.Write(respBody)
return
}
log.Printf("All trials failed. Returning Bad Gateway.")
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadGateway)
_ = json.NewEncoder(w).Encode(map[string]any{
"error": map[string]any{
"message": "All configured models and keys failed to respond.",
"type": "gateway_error",
"param": nil,
"code": "all_endpoints_failed",
},
})
}
}
func getTrialConfigs(configs []ModelConfig, uniqueModels []string, requestedModel string) []ModelConfig {
if len(configs) == 0 {
return nil
}
reqIdx := -1
for i, m := range uniqueModels {
if m == requestedModel {
reqIdx = i
break
}
}
var orderedModels []string
if reqIdx != -1 {
orderedModels = append(orderedModels, uniqueModels[reqIdx:]...)
orderedModels = append(orderedModels, uniqueModels[:reqIdx]...)
} else {
orderedModels = uniqueModels
}
var trialConfigs []ModelConfig
for _, modelName := range orderedModels {
for _, cfg := range configs {
if cfg.Model == modelName {
trialConfigs = append(trialConfigs, cfg)
}
}
}
return trialConfigs
}
func buildURL(endpoint string, reqPath string) string {
endpoint = strings.TrimSuffix(endpoint, "/")
reqPath = strings.TrimPrefix(reqPath, "/")
if strings.HasSuffix(endpoint, "/v1") {
if strings.HasPrefix(reqPath, "v1/") {
reqPath = strings.TrimPrefix(reqPath, "v1/")
}
}
return endpoint + "/" + reqPath
}
main.go
+127
View File
@@ -0,0 +1,127 @@
/*
Dynagate LLM Gateway: dynamically change models on the fly
Created by Luxferre in 2026, released into the public domain
*/
package main
import (
"context"
"flag"
"fmt"
"log"
"net/http"
"os"
"os/exec"
"os/signal"
"strings"
"syscall"
"time"
)
func main() {
port := flag.Int("port", 8080, "Port to listen on")
csvPath := flag.String("csv", "models.csv", "Path to the CSV configuration file")
keyPath := flag.String("key", "", "Path to plaintext file containing expected auth token")
csvUpdater := flag.String("csv-updater", "", "Command to run to update the configuration CSV file")
csvUpdateInterval := flag.Int("csv-update-interval", 10, "Interval in minutes with which to run the CSV updater command")
flag.Parse()
var authToken string
if *keyPath != "" {
content, err := os.ReadFile(*keyPath)
if err != nil {
log.Fatalf("Failed to read auth token file: %v", err)
}
authToken = strings.TrimSpace(string(content))
log.Printf("Starting Dynagate LLM Gateway. Config: %s, Port: %d, Auth: Required", *csvPath, *port)
} else {
log.Printf("Starting Dynagate LLM Gateway. Config: %s, Port: %d, Auth: None", *csvPath, *port)
}
// Initialize configuration manager
cm := NewConfigManager(*csvPath)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Start background config watcher
go cm.Watch(ctx)
// Start background CSV updater if specified
if *csvUpdater != "" {
mins := *csvUpdateInterval
if mins <= 0 {
log.Printf("Warning: csv-update-interval must be positive, defaulting to 10 minutes")
mins = 10
}
interval := time.Duration(mins) * time.Minute
log.Printf("Starting background CSV updater: command=%q, interval=%v", *csvUpdater, interval)
go startCSVUpdater(ctx, *csvUpdater, interval)
}
// Register routes
mux := http.NewServeMux()
mux.HandleFunc("/v1/models", handleModels(cm, authToken))
mux.HandleFunc("/v1/chat/completions", handleChatCompletions(cm, authToken))
mux.HandleFunc("/v1/images/generations", handleImageGenerations(cm, authToken))
server := &http.Server{
Addr: fmt.Sprintf(":%d", *port),
Handler: mux,
}
// Graceful shutdown setup
shutdownSig := make(chan os.Signal, 1)
signal.Notify(shutdownSig, os.Interrupt, syscall.SIGTERM)
go func() {
log.Printf("Server listening on port %d", *port)
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("Server listen failed: %v", err)
}
}()
<-shutdownSig
log.Println("Shutting down Dynagate server...")
cancel() // stop config watcher
shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second)
defer shutdownCancel()
if err := server.Shutdown(shutdownCtx); err != nil {
log.Fatalf("Server shutdown failed: %v", err)
}
log.Println("Server exited cleanly")
}
func startCSVUpdater(ctx context.Context, command string, interval time.Duration) {
runUpdater := func() {
log.Printf("Running CSV updater command: %s", command)
cmd := exec.Command("/bin/sh", "-c", command)
output, err := cmd.CombinedOutput()
if err != nil {
log.Printf("CSV updater command failed: %v, output: %q", err, strings.TrimSpace(string(output)))
} else {
log.Printf("CSV updater command completed successfully")
}
}
// Run once immediately at startup
runUpdater()
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
runUpdater()
}
}
}
models.csv
+7
View File
@@ -0,0 +1,7 @@
model,key,endpoint,extra
big-pickle,-,https://opencode.ai/zen,
deepseek-v4-flash-free,-,https://opencode.ai/zen,
mimo-v2.5-free,-,https://opencode.ai/zen,
hy3-free,-,https://opencode.ai/zen,
nemotron-3-ultra-free,-,https://opencode.ai/zen,
north-mini-code-free,-,https://opencode.ai/zen,
1 model key endpoint extra
2 big-pickle - https://opencode.ai/zen
3 deepseek-v4-flash-free - https://opencode.ai/zen
4 mimo-v2.5-free - https://opencode.ai/zen
5 hy3-free - https://opencode.ai/zen
6 nemotron-3-ultra-free - https://opencode.ai/zen
7 north-mini-code-free - https://opencode.ai/zen