diff --git a/README.md b/README.md index f5a8bd3..d8796d3 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# gr2gw: Universal Gradio to OpenAI LLM Gateway +# gr2gw: Universal Gradio to OpenAI LLM gateway A zero-dependency, high-performance Go proxy server that introspects any Gradio chat space (such as Hugging Face Spaces or custom deployments) and exposes a standards-compliant OpenAI `/v1/chat/completions` and `/v1/models` HTTP API. @@ -8,34 +8,34 @@ Default demo space: `https://ghost2513-openai-gpt-oss-120b.hf.space` ## Features -- **Zero External Dependencies**: Pure Go standard library (`net/http`, `encoding/json`, `bufio`, etc.). -- **Automatic Space Introspection**: Dynamically queries `/gradio_api/info`, `/config`, and Hugging Face space metadata to discover models, endpoints, and input parameter mappings. -- **Native Tencent Hunyuan 3 (`tencent-hy3`) Support**: +- **Zero external dependencies**: Pure Go standard library (`net/http`, `encoding/json`, `bufio`, etc.). +- **Automatic space introspection**: Dynamically queries `/gradio_api/info`, `/config`, and Hugging Face space metadata to discover models, endpoints, and input parameter mappings. +- **Native Tencent Hunyuan 3 (`tencent-hy3`) support**: - Full native zero-degradation handling for official spaces like `https://tencent-hy3.hf.space`. - Maps `functions_json_str` natively without polluting the system prompt. - Preserves multi-turn reasoning content and tool call history in standard OpenAI message schemas. - Maps `reasoning_effort` (`no_think`, `low`, `high`) directly to `think_level`. -- **Universal Multi-turn Handling**: +- **Universal multi-turn handling**: - Automatically formats conversation history into structured inputs when the space supports them. - Transparently composes multi-turn dialogue (`System`, `User`, `Assistant`) into single prompt inputs when the space only accepts a single message textbox. - Automatically pads hidden/State inputs (e.g. Gradio State components) to prevent backend argument count mismatches. -- **Real-Time Streaming & Accumulation Filter**: +- **Real-time streaming & accumulation filter**: - Automatically computes token deltas from cumulative or incremental Gradio SSE output streams (including 2D Hy3 frames `[[content, reasoning, tool_calls, history]]`). - Emits standards-compliant `chat.completion.chunk` SSE events in real time. -- **Thinking & Reasoning Token Separation**: +- **Thinking & reasoning token separation**: - Streams native reasoning chunks as `delta.reasoning_content` in real time. - Detects `...` tags in real time as fallback for standard spaces. - Separates reasoning into `delta.reasoning_content` (streaming) and `message.reasoning_content` (non-streaming). - Keeps `content` clean without tag leakage. -- **Full Tool Calling & Function Interception**: +- **Full tool calling & function interception**: - Formats schemas into native `functions_json_str` (Hy3) or system prompts (standard spaces). - **`StreamToolCallFilter`**: Stateful sliding-window filter that prevents `` tags from leaking into `delta.content`. Emits structured OpenAI `delta.tool_calls` chunks and sets `finish_reason: "tool_calls"`. - Seamlessly maintains multi-turn context when tool results are submitted back via `role: "tool"`. -- **Built-in SOCKS5 Proxy Client**: +- **Built-in SOCKS5 proxy client**: - Full RFC 1928 / RFC 1929 implementation with domain resolution (`socks5h://`), IPv4, IPv6, and username/password auth. -- **Dynamic Space Override**: +- **Dynamic space override**: - Switch the target Gradio space on-the-fly per request using the `X-Gradio-Space` or `X-Space-URL` HTTP headers. -- **Fibonacci Retry Engine**: +- **Fibonacci retry engine**: - Resilient backoff retry mechanism (1s, 1s, 2s, 3s, 5s) for transient network hiccups. --- @@ -57,7 +57,7 @@ make test ## Usage -### Quick Start +### Quick start Run with the default space (`https://ghost2513-openai-gpt-oss-120b.hf.space`): ```bash @@ -74,7 +74,7 @@ With SOCKS5 proxy: ./bin/gr2gw -space https://ghost2513-openai-gpt-oss-120b.hf.space -socks socks5://127.0.0.1:1080 ``` -### CLI Flags +### CLI flags | Flag | Default | Description | |------|---------|-------------| @@ -85,16 +85,16 @@ With SOCKS5 proxy: | `-user-agent`, `-ua` | Firefox string | Custom User-Agent header | | `-timeout` | `300` | Upstream request timeout in seconds | -### Environment Variables +### Environment variables -- `GRADIO_SPACE_URL`: Default Gradio space URL fallback. -- `ALL_PROXY`, `SOCKS5_PROXY`, `SOCKS_PROXY`: Default SOCKS5 proxy URL fallback. +- `GRADIO_SPACE_URL`: default Gradio space URL fallback. +- `ALL_PROXY`, `SOCKS5_PROXY`, `SOCKS_PROXY`: default SOCKS5 proxy URL fallback. --- -## API Examples +## API examples -### List Models +### List models ```bash curl http://localhost:8080/v1/models @@ -121,7 +121,7 @@ Response: } ``` -### Chat Completions (Non-Streaming) +### Chat completions (non-streaming) ```bash curl http://localhost:8080/v1/chat/completions \ @@ -159,7 +159,7 @@ Response: } ``` -### Chat Completions (Streaming) +### Chat completions (streaming) ```bash curl -N http://localhost:8080/v1/chat/completions \ @@ -173,7 +173,7 @@ curl -N http://localhost:8080/v1/chat/completions \ }' ``` -### Tool Calling +### Tool calling ```bash curl http://localhost:8080/v1/chat/completions \ @@ -235,7 +235,7 @@ Response: } ``` -### Dynamic Target Space Override +### Dynamic target space override Override the target space per request without restarting the server: diff --git a/gr2gw.go b/gr2gw.go index 35abeb7..4a8aff6 100644 --- a/gr2gw.go +++ b/gr2gw.go @@ -31,7 +31,7 @@ var ( ) // --------------------------------------------------------------------------- -// OpenAI API Data Structures +// OpenAI API data structures // --------------------------------------------------------------------------- type ModelItem struct { @@ -155,7 +155,7 @@ type StreamResponse struct { } // --------------------------------------------------------------------------- -// SOCKS5 Proxy Client (RFC 1928 / RFC 1929) +// SOCKS5 proxy client (RFC 1928 / RFC 1929) // --------------------------------------------------------------------------- type SOCKS5Config struct { @@ -336,7 +336,7 @@ func DialSOCKS5(ctx context.Context, proxyURL, targetAddr string) (net.Conn, err } // --------------------------------------------------------------------------- -// Helper Utilities +// Helper utilities // --------------------------------------------------------------------------- func GenerateUUID() string { @@ -430,7 +430,7 @@ func EffectiveUserAgent(r *http.Request) string { } // --------------------------------------------------------------------------- -// Tool and Message Processing +// Tool and message processing // --------------------------------------------------------------------------- func BuildToolInstruction(tools []Tool) string { @@ -832,7 +832,7 @@ func ExtractThinking(content string) (string, string) { } // --------------------------------------------------------------------------- -// Response Framing & Streamer +// Response framing & streamer // --------------------------------------------------------------------------- type FinalOutput struct { @@ -942,7 +942,7 @@ func sendStreamChunk(w http.ResponseWriter, flusher http.Flusher, completionID s } // --------------------------------------------------------------------------- -// Stateful Thinking Tag Filter for Streaming +// Stateful thinking tag filter for streaming // --------------------------------------------------------------------------- type StreamThinkingFilter struct { @@ -1030,7 +1030,7 @@ func (f *StreamThinkingFilter) Flush(onContent func(string), onReasoning func(st } // --------------------------------------------------------------------------- -// Stateful Tool Call Tag Filter for Streaming +// Stateful tool call tag filter for streaming // --------------------------------------------------------------------------- type StreamToolCallFilter struct { @@ -1134,7 +1134,7 @@ func (f *StreamToolCallFilter) Flush(onContent func(string), onToolCall func(Too } // --------------------------------------------------------------------------- -// Universal Gradio Space Inspector & Metadata Discovery +// Universal Gradio space inspector & metadata discovery // --------------------------------------------------------------------------- type GradioParamInfo struct { @@ -1575,7 +1575,7 @@ func InspectSpace(client *http.Client, rawURL, userAgent string) (*SpaceDiscover } // --------------------------------------------------------------------------- -// Universal Gradio Gateway Engine +// Universal Gradio gateway engine // --------------------------------------------------------------------------- type GradioJoinResponse struct { @@ -2409,7 +2409,7 @@ func (g *GradioGateway) ExecuteChatCompletion(w http.ResponseWriter, r *http.Req } // --------------------------------------------------------------------------- -// HTTP Routes & Main Server +// HTTP routes & main server // --------------------------------------------------------------------------- func main() {