feat: implement universal Gradio to OpenAI proxy gateway

2026-09-07 07:45:50 +03:00
commit d1d7422099
6 changed files with 2717 additions and 0 deletions
@@ -0,0 +1,2 @@
+bin/
+gr2gw
@@ -0,0 +1,13 @@
+.PHONY: all build clean test
+
+all: build
+
+build:
+	mkdir -p bin
+	go build -trimpath -ldflags="-s -w" -o bin/gr2gw gr2gw.go
+
+clean:
+	rm -rf bin
+
+test:
+	go test -v ./...
@@ -0,0 +1,251 @@
+# gr2gw: Universal Gradio to OpenAI LLM Gateway
+
+A zero-dependency, high-performance Go proxy server that introspects any Gradio chat space (such as Hugging Face Spaces or custom deployments) and exposes a standards-compliant OpenAI `/v1/chat/completions` and `/v1/models` HTTP API.
+
+Default demo space: `https://ghost2513-openai-gpt-oss-120b.hf.space`
+
+---
+
+## Features
+
+- **Zero External Dependencies**: Pure Go standard library (`net/http`, `encoding/json`, `bufio`, etc.).
+- **Automatic Space Introspection**: Dynamically queries `/gradio_api/info`, `/config`, and Hugging Face space metadata to discover models, endpoints, and input parameter mappings.
+- **Universal Multi-turn Handling**:
+  - Automatically formats conversation history into structured inputs when the space supports them.
+  - Transparently composes multi-turn dialogue (`System`, `User`, `Assistant`) into single prompt inputs when the space only accepts a single message textbox.
+  - Automatically pads hidden/State inputs (e.g. Gradio State components) to prevent backend argument count mismatches.
+- **Real-Time Streaming & Accumulation Filter**:
+  - Automatically computes token deltas from cumulative or incremental Gradio SSE output streams.
+  - Emits standards-compliant `chat.completion.chunk` SSE events in real time.
+- **Thinking & Reasoning Token Separation**:
+  - Detects `<think>...</think>` tags in real time.
+  - Separates reasoning into `delta.reasoning_content` (streaming) and `message.reasoning_content` (non-streaming).
+  - Keeps `content` clean without tag leakage.
+- **Full Tool Calling & Function Interception**:
+  - Formats schemas into system prompts with strict function calling instructions.
+  - **`StreamToolCallFilter`**: Stateful sliding-window filter that prevents `<tool_call>` tags from leaking into `delta.content`. Emits structured OpenAI `delta.tool_calls` chunks and sets `finish_reason: "tool_calls"`.
+  - Seamlessly maintains multi-turn context when tool results are submitted back via `role: "tool"`.
+- **Built-in SOCKS5 Proxy Client**:
+  - Full RFC 1928 / RFC 1929 implementation with domain resolution (`socks5h://`), IPv4, IPv6, and username/password auth.
+- **Dynamic Space Override**:
+  - Switch the target Gradio space on-the-fly per request using the `X-Gradio-Space` or `X-Space-URL` HTTP headers.
+- **Fibonacci Retry Engine**:
+  - Resilient backoff retry mechanism (1s, 1s, 2s, 3s, 5s) for transient network hiccups.
+
+---
+
+## Build
+
+```bash
+make build
+```
+
+Binary will be compiled to `bin/gr2gw`.
+
+To run tests:
+```bash
+make test
+```
+
+---
+
+## Usage
+
+### Quick Start
+
+Run with the default space (`https://ghost2513-openai-gpt-oss-120b.hf.space`):
+```bash
+./bin/gr2gw -port 8080
+```
+
+Target any other Gradio space:
+```bash
+./bin/gr2gw -space https://ericsqin-hy3.hf.space -port 8080
+```
+
+With SOCKS5 proxy:
+```bash
+./bin/gr2gw -space https://ghost2513-openai-gpt-oss-120b.hf.space -socks socks5://127.0.0.1:1080
+```
+
+### CLI Flags
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `-space`, `-url` | `https://ghost2513-openai-gpt-oss-120b.hf.space` | Target Gradio space URL |
+| `-port` | `8080` | Port to listen on |
+| `-host` | `0.0.0.0` | Host interface to bind to |
+| `-socks`, `-proxy`, `-socks5` | `""` | SOCKS5 proxy URL (`socks5://user:pass@host:port`) |
+| `-user-agent`, `-ua` | Firefox string | Custom User-Agent header |
+| `-timeout` | `300` | Upstream request timeout in seconds |
+
+### Environment Variables
+
+- `GRADIO_SPACE_URL`: Default Gradio space URL fallback.
+- `ALL_PROXY`, `SOCKS5_PROXY`, `SOCKS_PROXY`: Default SOCKS5 proxy URL fallback.
+
+---
+
+## API Examples
+
+### List Models
+
+```bash
+curl http://localhost:8080/v1/models
+```
+
+Response:
+```json
+{
+  "object": "list",
+  "data": [
+    {
+      "id": "openai/gpt-oss-120b",
+      "object": "model",
+      "created": 1788756307,
+      "owned_by": "gradio"
+    },
+    {
+      "id": "gpt-oss-120b",
+      "object": "model",
+      "created": 1788756307,
+      "owned_by": "gradio"
+    }
+  ]
+}
+```
+
+### Chat Completions (Non-Streaming)
+
+```bash
+curl http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "openai/gpt-oss-120b",
+    "messages": [
+      {"role": "user", "content": "What is the capital of France?"}
+    ]
+  }'
+```
+
+Response:
+```json
+{
+  "id": "chatcmpl-16425f9d-c350-47a1-9a6d-e9ce10871545",
+  "object": "chat.completion",
+  "created": 1788756310,
+  "model": "openai/gpt-oss-120b",
+  "choices": [
+    {
+      "index": 0,
+      "message": {
+        "role": "assistant",
+        "content": "Paris is the capital of France."
+      },
+      "finish_reason": "stop"
+    }
+  ],
+  "usage": {
+    "prompt_tokens": 0,
+    "completion_tokens": 0,
+    "total_tokens": 0
+  }
+}
+```
+
+### Chat Completions (Streaming)
+
+```bash
+curl -N http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "openai/gpt-oss-120b",
+    "messages": [
+      {"role": "user", "content": "Count from 1 to 5."}
+    ],
+    "stream": true
+  }'
+```
+
+### Tool Calling
+
+```bash
+curl http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "openai/gpt-oss-120b",
+    "messages": [
+      {"role": "user", "content": "What is the weather in Tokyo?"}
+    ],
+    "tools": [{
+      "type": "function",
+      "function": {
+        "name": "get_weather",
+        "description": "Get current weather in a location",
+        "parameters": {
+          "type": "object",
+          "properties": {
+            "location": {"type": "string", "description": "City name"}
+          },
+          "required": ["location"]
+        }
+      }
+    }]
+  }'
+```
+
+Response:
+```json
+{
+  "id": "chatcmpl-32727c62-ef2a-4866-855b-f1c7ec2b8023",
+  "object": "chat.completion",
+  "created": 1788756322,
+  "model": "openai/gpt-oss-120b",
+  "choices": [
+    {
+      "index": 0,
+      "message": {
+        "role": "assistant",
+        "content": null,
+        "tool_calls": [
+          {
+            "id": "call_3d4c016a",
+            "type": "function",
+            "function": {
+              "name": "get_weather",
+              "arguments": "{\"location\":\"Tokyo\"}"
+            }
+          }
+        ]
+      },
+      "finish_reason": "tool_calls"
+    }
+  ],
+  "usage": {
+    "prompt_tokens": 0,
+    "completion_tokens": 0,
+    "total_tokens": 0
+  }
+}
+```
+
+### Dynamic Target Space Override
+
+Override the target space per request without restarting the server:
+
+```bash
+curl http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "X-Gradio-Space: https://ericsqin-hy3.hf.space" \
+  -d '{
+    "messages": [
+      {"role": "user", "content": "Hello!"}
+    ]
+  }'
+```
+
+---
+
+## License
+
+Released into the public domain under Creative Commons Zero (CC0) or Unlicense.
@@ -0,0 +1,3 @@
+module gr2gw
+
+go 1.26
@@ -0,0 +1,249 @@
+package main
+
+import (
+  "bytes"
+  "encoding/json"
+  "fmt"
+  "net/http"
+  "net/http/httptest"
+  "strings"
+  "testing"
+  "time"
+)
+
+func TestParseSOCKS5URL(t *testing.T) {
+  tests := []struct {
+    input    string
+    expected *SOCKS5Config
+  }{
+    {"", nil},
+    {"127.0.0.1:1080", &SOCKS5Config{Address: "127.0.0.1:1080"}},
+    {"socks5://127.0.0.1:9050", &SOCKS5Config{Address: "127.0.0.1:9050"}},
+    {"socks5h://user:pass@10.0.0.1:1080", &SOCKS5Config{Address: "10.0.0.1:1080", Username: "user", Password: "pass"}},
+  }
+
+  for _, tc := range tests {
+    cfg, err := ParseSOCKS5URL(tc.input)
+    if err != nil {
+      t.Fatalf("unexpected error for %q: %v", tc.input, err)
+    }
+    if tc.expected == nil {
+      if cfg != nil {
+        t.Errorf("expected nil config, got %+v", cfg)
+      }
+      continue
+    }
+    if cfg.Address != tc.expected.Address || cfg.Username != tc.expected.Username || cfg.Password != tc.expected.Password {
+      t.Errorf("for %q, expected %+v, got %+v", tc.input, tc.expected, cfg)
+    }
+  }
+}
+
+func TestChatMessageGetContentString(t *testing.T) {
+  m1 := ChatMessage{Role: "user", Content: "hello world"}
+  if m1.GetContentString() != "hello world" {
+    t.Errorf("expected 'hello world', got %q", m1.GetContentString())
+  }
+
+  m2 := ChatMessage{
+    Role: "user",
+    Content: []interface{}{
+      map[string]interface{}{"type": "text", "text": "part 1 "},
+      map[string]interface{}{"type": "text", "text": "part 2"},
+    },
+  }
+  if m2.GetContentString() != "part 1 part 2" {
+    t.Errorf("expected 'part 1 part 2', got %q", m2.GetContentString())
+  }
+}
+
+func TestExtractThinking(t *testing.T) {
+  content := "<think>Let me calculate 2+2.</think>The answer is 4."
+  clean, reasoning := ExtractThinking(content)
+  if reasoning != "Let me calculate 2+2." {
+    t.Errorf("expected reasoning 'Let me calculate 2+2.', got %q", reasoning)
+  }
+  if clean != "The answer is 4." {
+    t.Errorf("expected clean 'The answer is 4.', got %q", clean)
+  }
+}
+
+func TestDetectToolCalls(t *testing.T) {
+  xmlContent := `<tool_call>
+{"name": "get_weather", "arguments": {"city": "Paris"}}
+</tool_call>`
+  calls, rem, ok := DetectToolCalls(xmlContent)
+  if !ok || len(calls) != 1 {
+    t.Fatalf("expected 1 tool call, got %d (ok: %v)", len(calls), ok)
+  }
+  if calls[0].Function.Name != "get_weather" {
+    t.Errorf("expected function name get_weather, got %q", calls[0].Function.Name)
+  }
+  if rem != "" {
+    t.Errorf("expected empty remaining content, got %q", rem)
+  }
+
+  jsonContent := `{"name": "calculator", "arguments": {"expr": "1+1"}}`
+  calls2, rem2, ok2 := DetectToolCalls(jsonContent)
+  if !ok2 || len(calls2) != 1 {
+    t.Fatalf("expected 1 tool call from JSON, got %d", len(calls2))
+  }
+  if calls2[0].Function.Name != "calculator" {
+    t.Errorf("expected function calculator, got %q", calls2[0].Function.Name)
+  }
+  if rem2 != "" {
+    t.Errorf("expected empty remaining, got %q", rem2)
+  }
+}
+
+func TestStreamThinkingFilter(t *testing.T) {
+  filter := NewStreamThinkingFilter()
+  var contentParts []string
+  var reasoningParts []string
+
+  onContent := func(s string) { contentParts = append(contentParts, s) }
+  onReasoning := func(s string) { reasoningParts = append(reasoningParts, s) }
+
+  chunks := []string{"<thi", "nk>Thinking de", "eply</th", "ink>Here is your answer."}
+  for _, c := range chunks {
+    filter.Feed(c, onContent, onReasoning)
+  }
+  filter.Flush(onContent, onReasoning)
+
+  fullReasoning := strings.Join(reasoningParts, "")
+  fullContent := strings.Join(contentParts, "")
+
+  if fullReasoning != "Thinking deeply" {
+    t.Errorf("expected reasoning 'Thinking deeply', got %q", fullReasoning)
+  }
+  if fullContent != "Here is your answer." {
+    t.Errorf("expected content 'Here is your answer.', got %q", fullContent)
+  }
+}
+
+func TestStreamToolCallFilter(t *testing.T) {
+  filter := NewStreamToolCallFilter()
+  var contentParts []string
+  var toolCalls []ToolCall
+
+  onContent := func(s string) { contentParts = append(contentParts, s) }
+  onToolCall := func(tc ToolCall) { toolCalls = append(toolCalls, tc) }
+
+  chunks := []string{
+    "Searching now: ",
+    "<tool_c",
+    "all>\n{\"name\": \"search_web\", \"arguments\": {\"query\": \"golang\"}}\n</tool_",
+    "call>",
+    " Done.",
+  }
+
+  for _, c := range chunks {
+    filter.Feed(c, onContent, onToolCall)
+  }
+  filter.Flush(onContent, onToolCall)
+
+  if len(toolCalls) != 1 {
+    t.Fatalf("expected 1 emitted tool call, got %d", len(toolCalls))
+  }
+  if toolCalls[0].Function.Name != "search_web" {
+    t.Errorf("expected tool name 'search_web', got %q", toolCalls[0].Function.Name)
+  }
+  fullContent := strings.Join(contentParts, "")
+  if fullContent != "Searching now:  Done." {
+    t.Errorf("expected 'Searching now:  Done.', got %q", fullContent)
+  }
+}
+
+func TestMockGradioServerCompletion(t *testing.T) {
+  // Setup a mock Gradio server
+  ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+    if r.URL.Path == "/gradio_api/info" {
+      resp := GradioAPIInfoResponse{
+        NamedEndpoints: map[string]GradioEndpointInfo{
+          "/chat_fn": {
+            Parameters: []GradioParamInfo{
+              {ParameterName: "message", Component: "Textbox"},
+            },
+            Returns: []GradioParamInfo{
+              {ParameterName: "response", Component: "Json"},
+            },
+          },
+        },
+      }
+      w.Header().Set("Content-Type", "application/json")
+      json.NewEncoder(w).Encode(resp)
+      return
+    }
+
+    if r.URL.Path == "/gradio_api/call/chat_fn" {
+      w.Header().Set("Content-Type", "application/json")
+      json.NewEncoder(w).Encode(GradioJoinResponse{EventID: "evt_123"})
+      return
+    }
+
+    if r.URL.Path == "/gradio_api/call/chat_fn/evt_123" {
+      w.Header().Set("Content-Type", "text/event-stream")
+      flusher, ok := w.(http.Flusher)
+      if !ok {
+        t.Fatal("expected flusher")
+      }
+      fmt.Fprintf(w, "event: generating\ndata: [\"Hello \", null]\n\n")
+      flusher.Flush()
+      fmt.Fprintf(w, "event: generating\ndata: [\"Hello world!\", null]\n\n")
+      flusher.Flush()
+      fmt.Fprintf(w, "event: complete\ndata: [\"Hello world!\", null]\n\n")
+      flusher.Flush()
+      return
+    }
+
+    http.NotFound(w, r)
+  }))
+  defer ts.Close()
+
+  gw := NewGradioGateway(ts.URL, "", 10*time.Second)
+
+  // 1. Test Non-streaming request
+  reqBody := ChatCompletionRequest{
+    Model: "test-model",
+    Messages: []ChatMessage{
+      {Role: "user", Content: "Hi"},
+    },
+    Stream: false,
+  }
+  b, _ := json.Marshal(reqBody)
+  httpReq := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBuffer(b))
+  httpReq.Header.Set("Content-Type", "application/json")
+  rec := httptest.NewRecorder()
+
+  err := gw.ExecuteChatCompletion(rec, httpReq, reqBody)
+  if err != nil {
+    t.Fatalf("unexpected completion error: %v", err)
+  }
+
+  var resp ChatCompletionResponse
+  if err := json.NewDecoder(rec.Body).Decode(&resp); err != nil {
+    t.Fatalf("failed to decode completion response: %v", err)
+  }
+  if len(resp.Choices) != 1 {
+    t.Fatalf("expected 1 choice, got %d", len(resp.Choices))
+  }
+  if resp.Choices[0].Message.GetContentString() != "Hello world!" {
+    t.Errorf("expected 'Hello world!', got %q", resp.Choices[0].Message.GetContentString())
+  }
+
+  // 2. Test Streaming request
+  reqBodyStream := reqBody
+  reqBodyStream.Stream = true
+  recStream := httptest.NewRecorder()
+  err = gw.ExecuteChatCompletion(recStream, httpReq, reqBodyStream)
+  if err != nil {
+    t.Fatalf("unexpected streaming error: %v", err)
+  }
+  streamOutput := recStream.Body.String()
+  if !strings.Contains(streamOutput, "data: [DONE]") {
+    t.Errorf("expected stream to contain [DONE], got:\n%s", streamOutput)
+  }
+  if !strings.Contains(streamOutput, "Hello world!") && !strings.Contains(streamOutput, "world!") {
+    t.Errorf("expected stream output to contain delta tokens, got:\n%s", streamOutput)
+  }
+}