feat: universal heuristic engine for gradio version, flavor, protocol and tool calling resolution

2026-09-07 11:53:44 +03:00
parent 491114b2bd
commit 751fe3c54f
3 changed files with 1027 additions and 211 deletions
@@ -7,7 +7,20 @@ Default demo space: `https://tencent-hy3.hf.space`
 ## Features

 - **Zero external dependencies**: pure Go standard library (`net/http`, `encoding/json`, `bufio`, etc.).
- **Automatic space introspection**: dynamically queries `/gradio_api/info`, `/config`, and Hugging Face space metadata to discover models, endpoints, and input parameter mappings.
+- **Universal heuristic discovery engine**:
+  - Automatically interrogates `/gradio_api/info`, `/config`, and Hugging Face metadata without endpoint-specific hardcoding.
+  - Detects Gradio runtime versions across v3, v4, v5, and v6.
+  - Classifies application architecture into `ChatInterface`, `Blocks (Chat)`, `Blocks (Multimodal Chat)`, `Interface`, and `Generic`.
+  - Disambiguation scoring engine evaluates candidate endpoints, filtering out UI resets, clears, retries, likes, and utility triggers to pinpoint primary conversational completion functions.
+  - Correlates semantic parameter names from `/gradio_api/info` with component IDs from `/config` to reconstruct parameter mappings even when component labels are obfuscated.
+- **Dual protocol support with auto-fallback**:
+  - Supports modern Gradio 4/5/6 `/call` SSE protocol with persistent session hashes.
+  - Supports legacy Gradio 3 `/run/predict` and `/api/predict` direct execution protocol.
+  - Instant zero-latency fallback from `/call` to `/run/predict` upon HTTP 404 or 405 status codes.
+- **Tool calling support classification**:
+  - Automatically identifies tool calling mechanisms: `native_slot`, `prompt_augmented_system`, `prompt_augmented_first_turn`, or `prompt_augmented_single_prompt`.
+  - Upstream tool call recovery intercepts `tool_use_failed` errors and extracts function names and JSON arguments.
+  - Real-time sliding-window `<tool_call>` tag interceptor emits structured OpenAI tool call chunks.
 - **Native Tencent Hunyuan 3 (`tencent-hy3`) support**:
  - Full native zero-degradation handling for official spaces like `https://tencent-hy3.hf.space`.
  - Maps `functions_json_str` natively without polluting the system prompt.
@@ -26,17 +39,111 @@ Default demo space: `https://tencent-hy3.hf.space`
  - Detects `<think>...</think>` tags in real time as fallback for standard spaces.
  - Separates reasoning into `delta.reasoning_content` (streaming) and `message.reasoning_content` (non-streaming).
  - Keeps `content` clean without tag leakage.
- **Full tool calling & function interception**:
-  - Formats schemas into native `functions_json_str` (Hy3) or system prompts (standard spaces).
-  - Intercepts and recovers tool calls from upstream `tool_use_failed` errors containing `failed_generation`.
-  - **`StreamToolCallFilter`**: stateful sliding-window filter that prevents `<tool_call>` tags from leaking into `delta.content`. Emits structured OpenAI `delta.tool_calls` chunks and sets `finish_reason: "tool_calls"`.
-  - Seamlessly maintains multi-turn context when tool results are submitted back via `role: "tool"`.
 - **Built-in SOCKS5 proxy client**:
  - Full RFC 1928 / RFC 1929 implementation with domain resolution (`socks5h://`), IPv4, IPv6, and username/password auth.
 - **Dynamic space override**:
  - Switch the target Gradio space on-the-fly per request using the `X-Gradio-Space` or `X-Space-URL` HTTP headers.
 - **Fibonacci retry engine**:
-  - Resilient backoff retry mechanism (1s, 1s, 2s, 3s, 5s) for transient network hiccups.
+  - Resilient backoff retry mechanism (1s, 1s, 2s, 3s, 5s) for transient network hiccups with immediate break on 404/405 errors.
+
+## Heuristic discovery engine
+
+The gateway implements an autonomous heuristic engine that discovers and configures the optimal completion path for any target Gradio space at startup.
+
+### Space introspection and version detection
+
+Upon initialization, `gr2gw` inspects the space metadata:
+1. Queries `/gradio_api/info` and `/config` endpoints.
+2. Extracts the Gradio runtime version (v3, v4, v5, or v6).
+3. Detects API routing prefixes (e.g. `/gradio_api` on modern versions, or root on Gradio 3).
+
+### UI flavor classification
+
+The engine classifies the application structure into architectural flavors:
+- **`ChatInterface`**: Standard Gradio chat interfaces equipped with chatbot, textbox, and optional additional inputs.
+- **`Blocks (Chat)`**: Custom `gr.Blocks` layouts containing conversational components.
+- **`Blocks (Multimodal Chat)`**: Blocks architectures featuring `MultimodalTextbox` components that accept `{text, files}` JSON payloads.
+- **`Interface`**: Classic input-output `gr.Interface` instances.
+- **`Generic`**: Spaces with custom or unclassified component topologies.
+
+### Candidate endpoint scoring
+
+Gradio spaces frequently expose dozens of internal endpoints for UI actions (e.g. clearing text, retrying responses, voting/liking, adjusting sliders). The scoring algorithm identifies the true conversational endpoint by:
+- Penalizing non-conversational triggers (e.g. `-600` for clear/reset/undo/retry/like endpoints).
+- Rewarding chat semantics (`+150` for `/chat`, `/predict`, `/generate`, `/respond`).
+- Rewarding message inputs (`+120` for `Textbox` or `MultimodalTextbox`).
+- Rewarding chat history slots (`+80` for `Chatbot` or `State` components).
+- Rewarding generator and streaming dependencies (`+50`).
+
+### Dual protocol execution and auto-fallback
+
+- **`call` protocol**: Modern Gradio 4/5/6 execution via `POST /call/{endpoint}` returning an event ID, followed by `GET /call/{endpoint}/{event_id}` SSE streaming.
+- **`predict` protocol**: Gradio 3 and legacy execution via direct `POST /run/predict` or `POST /api/predict`.
+- **Runtime failover**: If a space returns HTTP 404 or 405 when calling the modern protocol, the gateway breaks immediately from the retry loop and falls back to `/run/predict`.
+
+### Tool calling support modes
+
+The gateway evaluates available input components to determine how tool schemas and function calls should be delivered:
+- **`native_slot`**: The space provides a dedicated parameter slot for tool definitions (e.g. `functions_json_str` on Hunyuan 3). Function definitions are passed cleanly without prompt alteration.
+- **`prompt_augmented_system`**: The space provides a separate `system_prompt` input slot. Tool definitions and invocation schemas are injected directly into the system prompt.
+- **`prompt_augmented_first_turn`**: The space accepts chat history pairs but lacks a dedicated system prompt slot. Tool definitions are prepended to the user prompt on the first dialogue turn.
+- **`prompt_augmented_single_prompt`**: The space accepts only a single textbox input. Full multi-turn dialogue, tool definitions, and system guidance are synthesized into a single cohesive prompt.
+
+### Startup resolution diagnostics
+
+Whenever the gateway starts or inspects a new space, it prints the complete resolution picture:
+
+```text
+================================================================================
+Gradio Space Resolution Picture
+--------------------------------------------------------------------------------
+Space URL:         https://tencent-hy3.hf.space
+Title:             Hunyuan 3 Chat
+Gradio Version:    5.29.0
+UI Flavor:         Blocks (Chat)
+Protocol:          call
+API Prefix:        /gradio_api
+Resolved Endpoint: /chat_fn
+Function Index:    1
+Primary Model:     hy3
+Exposed Models:    hy3, hunyuan3, tencent/Hy3
+History Format:    tuples
+Tool Call Support: native_slot
+Total Input Slots: 9
+Input Slot Mappings:
+  [0] Component ID 1    textbox (label="message")      -> message
+  [1] Component ID 2    textbox (label="system")       -> system_prompt
+  [2] Component ID 3    chatbot (label="chatbot")      -> history
+  [3] Component ID 4    radio (label="think_level")    -> think_level
+  [4] Component ID 5    slider (label="temperature")   -> temperature
+  [5] Component ID 6    slider (label="max_tokens")    -> max_tokens
+  [6] Component ID 7    slider (label="top_p")         -> top_p
+  [7] Component ID 8    state (label="preserved")      -> preserved_thinking
+  [8] Component ID 9    textbox (label="functions")    -> functions_json_str
+================================================================================
+```
+
+### Gateway status endpoint
+
+Making a `GET` request to `/` returns a JSON summary of the running gateway and the discovered space profile:
+
+```bash
+curl http://localhost:8080/
+```
+
+Response:
+```json
+{
+  "name": "gr2gw",
+  "status": "ready",
+  "space_url": "https://tencent-hy3.hf.space",
+  "gradio_version": "5.29.0",
+  "flavor": "Blocks (Chat)",
+  "protocol": "call",
+  "tool_call_mode": "native_slot",
+  "models": ["hy3", "hunyuan3", "tencent/Hy3"]
+}
+```

 ## Installation

@@ -1597,4 +1597,285 @@ func TestMultimodalStateSpaceMockServerCompletion(t *testing.T) {
  }
 }

+// TestGradio3DirectPredictProtocol verifies discovery and chat completion against a Gradio 3 space
+// where /gradio_api/info returns 404 and the protocol resolves to /run/predict with tuple pairs history.
+func TestGradio3DirectPredictProtocol(t *testing.T) {
+  ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+    if r.URL.Path == "/gradio_api/info" || r.URL.Path == "/info" {
+      http.NotFound(w, r)
+      return
+    }
+
+    if r.URL.Path == "/config" {
+      cfg := GradioConfigResponse{
+        Version: "3.41.2",
+        Mode:    "chat_interface",
+        Title:   "Legacy Gradio 3 Chat",
+        Components: []GradioComponent{
+          {ID: 1, Type: "textbox", Props: map[string]interface{}{"label": "Input"}},
+          {ID: 2, Type: "chatbot", Props: map[string]interface{}{"label": "Chatbot"}},
+        },
+        Dependencies: []GradioDependency{
+          {
+            ID:      0,
+            Inputs:  []int{1, 2},
+            Outputs: []int{2},
+          },
+        },
+      }
+      w.Header().Set("Content-Type", "application/json")
+      json.NewEncoder(w).Encode(cfg)
+      return
+    }
+
+    if r.URL.Path == "/run/predict" {
+      var body struct {
+        Data        []interface{} `json:"data"`
+        FnIndex     int           `json:"fn_index"`
+        SessionHash string        `json:"session_hash"`
+      }
+      if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
+        http.Error(w, err.Error(), http.StatusBadRequest)
+        return
+      }
+      if body.FnIndex != 0 {
+        http.Error(w, fmt.Sprintf("expected fn_index 0, got %d", body.FnIndex), http.StatusBadRequest)
+        return
+      }
+      msg, _ := body.Data[0].(string)
+      reply := "Echo from Gradio 3: " + msg
+      respData := map[string]interface{}{
+        "data": []interface{}{
+          [][]string{
+            {msg, reply},
+          },
+        },
+        "is_generating": false,
+      }
+      w.Header().Set("Content-Type", "application/json")
+      json.NewEncoder(w).Encode(respData)
+      return
+    }
+
+    http.NotFound(w, r)
+  }))
+  defer ts.Close()
+
+  gw := NewGradioGateway(ts.URL, "", 10*time.Second)
+  disc := gw.GetDiscovery(ts.URL, DefaultUserAgent)
+
+  if disc.GradioVersion != "3.41.2" {
+    t.Errorf("expected GradioVersion 3.41.2, got %s", disc.GradioVersion)
+  }
+  if disc.Protocol != "predict" {
+    t.Errorf("expected Protocol predict, got %s", disc.Protocol)
+  }
+  if disc.Flavor != "ChatInterface" {
+    t.Errorf("expected Flavor ChatInterface, got %s", disc.Flavor)
+  }
+  if disc.HistoryFormat != "pairs" {
+    t.Errorf("expected HistoryFormat pairs, got %s", disc.HistoryFormat)
+  }
+  if disc.FnIndex != 0 {
+    t.Errorf("expected FnIndex 0, got %d", disc.FnIndex)
+  }
+
+  // Non-streaming completion
+  req := ChatCompletionRequest{
+    Messages: []ChatMessage{
+      {Role: "user", Content: "Hello Gradio 3!"},
+    },
+    Stream: false,
+  }
+  b, _ := json.Marshal(req)
+  httpReq := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBuffer(b))
+  rec := httptest.NewRecorder()
+
+  if err := gw.ExecuteChatCompletion(rec, httpReq, req); err != nil {
+    t.Fatalf("ExecuteChatCompletion failed on Gradio 3: %v", err)
+  }
+
+  var res ChatCompletionResponse
+  if err := json.NewDecoder(rec.Body).Decode(&res); err != nil {
+    t.Fatalf("failed to decode response: %v", err)
+  }
+  if res.Choices[0].Message.Content != "Echo from Gradio 3: Hello Gradio 3!" {
+    t.Errorf("unexpected completion content: %v", res.Choices[0].Message.Content)
+  }
+}
+
+// TestEndpointScoringDisambiguation verifies that chat generation endpoints are selected over
+// UI utility, reset, voting, and feedback endpoints.
+func TestEndpointScoringDisambiguation(t *testing.T) {
+  compMap := map[int]GradioComponent{
+    1: {ID: 1, Type: "textbox", Props: map[string]interface{}{"label": "Message"}},
+    2: {ID: 2, Type: "chatbot", Props: map[string]interface{}{"label": "Chat"}},
+    3: {ID: 3, Type: "state", Props: map[string]interface{}{"label": "State"}},
+  }
+
+  chatDep := GradioDependency{
+    ID:      0,
+    Inputs:  []int{1, 2, 3},
+    Outputs: []int{2},
+    Types:   GradioDependencyTypes{Generator: true},
+  }
+  clearDep := GradioDependency{
+    ID:      1,
+    Inputs:  []int{2},
+    Outputs: []int{2},
+  }
+  voteDep := GradioDependency{
+    ID:      2,
+    Inputs:  []int{2},
+    Outputs: []int{},
+  }
+
+  chatScore := ScoreCandidateEndpoint("chat", nil, &chatDep, compMap)
+  clearScore := ScoreCandidateEndpoint("clear", nil, &clearDep, compMap)
+  voteScore := ScoreCandidateEndpoint("vote", nil, &voteDep, compMap)
+  resetScore := ScoreCandidateEndpoint("reset_all", nil, &clearDep, compMap)
+
+  if chatScore <= 0 {
+    t.Errorf("expected positive chat score, got %d", chatScore)
+  }
+  if clearScore >= chatScore {
+    t.Errorf("expected chat score > clear score, got chat=%d clear=%d", chatScore, clearScore)
+  }
+  if voteScore >= chatScore {
+    t.Errorf("expected chat score > vote score, got chat=%d vote=%d", chatScore, voteScore)
+  }
+  if resetScore >= chatScore {
+    t.Errorf("expected chat score > reset score, got chat=%d reset=%d", chatScore, resetScore)
+  }
+}
+
+// TestToolCallingPictureResolution verifies that tool calling mode is accurately resolved
+// based on component topology.
+func TestToolCallingPictureResolution(t *testing.T) {
+  // Case A: Native slot
+  discNative := NewDefaultSpaceDiscovery("https://tencent-hy3.hf.space")
+  discNative.FunctionsJSONIndex = 8
+  discNative.SystemIndex = 2
+  discNative.HistoryIndex = 1
+  discNative.ToolCallMode = "native_slot"
+  if discNative.ToolCallMode != "native_slot" {
+    t.Errorf("expected native_slot, got %s", discNative.ToolCallMode)
+  }
+
+  // Case B: Dedicated system prompt slot
+  discSys := NewDefaultSpaceDiscovery("https://custom-chat.hf.space")
+  discSys.FunctionsJSONIndex = -1
+  discSys.SystemIndex = 2
+  discSys.HistoryIndex = 1
+  discSys.ToolCallMode = "prompt_augmented_system"
+  if discSys.ToolCallMode != "prompt_augmented_system" {
+    t.Errorf("expected prompt_augmented_system, got %s", discSys.ToolCallMode)
+  }
+
+  // Case C: Conversation history (first turn)
+  discFirst := NewDefaultSpaceDiscovery("https://chat-only.hf.space")
+  discFirst.FunctionsJSONIndex = -1
+  discFirst.SystemIndex = -1
+  discFirst.HistoryIndex = 1
+  discFirst.ToolCallMode = "prompt_augmented_first_turn"
+  if discFirst.ToolCallMode != "prompt_augmented_first_turn" {
+    t.Errorf("expected prompt_augmented_first_turn, got %s", discFirst.ToolCallMode)
+  }
+
+  // Case D: Single prompt input
+  discSingle := NewDefaultSpaceDiscovery("https://single-prompt.hf.space")
+  discSingle.FunctionsJSONIndex = -1
+  discSingle.SystemIndex = -1
+  discSingle.HistoryIndex = -1
+  discSingle.ToolCallMode = "prompt_augmented_single_prompt"
+  if discSingle.ToolCallMode != "prompt_augmented_single_prompt" {
+    t.Errorf("expected prompt_augmented_single_prompt, got %s", discSingle.ToolCallMode)
+  }
+}
+
+// TestCallToPredictProtocolFallback verifies that if a space reports /call support in /info
+// but /call returns 404 at runtime, gr2gw gracefully falls back to /run/predict.
+func TestCallToPredictProtocolFallback(t *testing.T) {
+  ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+    if r.URL.Path == "/gradio_api/info" {
+      info := GradioAPIInfoResponse{
+        NamedEndpoints: map[string]GradioEndpointInfo{
+          "/chat": {
+            Parameters: []GradioParamInfo{
+              {ParameterName: "prompt", Label: "Prompt", Component: "Textbox"},
+            },
+          },
+        },
+      }
+      w.Header().Set("Content-Type", "application/json")
+      json.NewEncoder(w).Encode(info)
+      return
+    }
+
+    if r.URL.Path == "/config" {
+      cfg := GradioConfigResponse{
+        Version: "4.20.0",
+        Mode:    "interface",
+        Components: []GradioComponent{
+          {ID: 1, Type: "textbox", Props: map[string]interface{}{"label": "Prompt"}},
+        },
+        Dependencies: []GradioDependency{
+          {ID: 0, APIName: "/chat", Inputs: []int{1}, Outputs: []int{1}},
+        },
+      }
+      w.Header().Set("Content-Type", "application/json")
+      json.NewEncoder(w).Encode(cfg)
+      return
+    }
+
+    // /call/chat returns 404 (endpoint disabled or unsupported)
+    if strings.HasPrefix(r.URL.Path, "/gradio_api/call/") || strings.HasPrefix(r.URL.Path, "/call/") {
+      http.NotFound(w, r)
+      return
+    }
+
+    // Fallback /run/predict works
+    if r.URL.Path == "/run/predict" || r.URL.Path == "/gradio_api/run/predict" {
+      var body struct {
+        Data []interface{} `json:"data"`
+      }
+      json.NewDecoder(r.Body).Decode(&body)
+      msg, _ := body.Data[0].(string)
+      w.Header().Set("Content-Type", "application/json")
+      json.NewEncoder(w).Encode(map[string]interface{}{
+        "data":          []interface{}{"Fallback response for: " + msg},
+        "is_generating": false,
+      })
+      return
+    }
+
+    http.NotFound(w, r)
+  }))
+  defer ts.Close()
+
+  gw := NewGradioGateway(ts.URL, "", 10*time.Second)
+
+  req := ChatCompletionRequest{
+    Messages: []ChatMessage{
+      {Role: "user", Content: "Testing fallback"},
+    },
+    Stream: false,
+  }
+  b, _ := json.Marshal(req)
+  httpReq := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBuffer(b))
+  rec := httptest.NewRecorder()
+
+  if err := gw.ExecuteChatCompletion(rec, httpReq, req); err != nil {
+    t.Fatalf("ExecuteChatCompletion fallback failed: %v", err)
+  }
+
+  var res ChatCompletionResponse
+  if err := json.NewDecoder(rec.Body).Decode(&res); err != nil {
+    t.Fatalf("failed to decode response: %v", err)
+  }
+  if res.Choices[0].Message.Content != "Fallback response for: Testing fallback" {
+    t.Errorf("unexpected content: %v", res.Choices[0].Message.Content)
+  }
+}
+