feat: initial implementation of qflash gateway

2026-09-05 15:42:07 +03:00
commit d06c025c24
7 changed files with 2215 additions and 0 deletions
@@ -0,0 +1,161 @@
+# qflash: OpenAI Proxy Gateway for Qwen3.8-Flash-Next
+
+Standalone, performant, zero-dependency Go OpenAI proxy gateway for the **Qwen3.8-Flash-Next** Hugging Face Gradio space (`https://halvo78-qwen3-8-flash-next-playground.hf.space`).
+
+Created by Luxferre in 2026, released into the public domain.
+
+---
+
+## Features
+
+- **Zero External Dependencies**: Built entirely with Go standard library packages (`net/http`, `encoding/json`, `bufio`, etc.).
+- **OpenAI-Compatible API**: Implements standard `/v1/chat/completions` (streaming & non-streaming) and `/v1/models`.
+- **Real-Time Token Streaming**: Streams SSE chunks with incremental token delivery directly to clients.
+- **Deep Reasoning Separation**:
+  - Automatically isolates thinking traces from both standard `<think>...</think>` tags and the playground's blockquote thinking blocks (`> 💭 **Thinking Process...**`).
+  - Emits pure thought traces to `delta.reasoning_content` (streaming) and `message.reasoning_content` (non-streaming).
+  - Keeps `delta.content` and `message.content` clean.
+- **Stateful Streaming Tool Call Interception**:
+  - Injects tool schemas into system instructions.
+  - Intercepts `<tool_call>` blocks in real time via `StreamToolCallFilter` without leaking raw XML or JSON into `delta.content`.
+  - Emits structured `delta.tool_calls` chunks and sets `finish_reason: "tool_calls"`.
+- **Reasoning Effort Control**: Respects standard `reasoning_effort: "none"` to switch dynamically into high-speed Instruct Mode.
+- **Zero-Dependency SOCKS5 Proxy Client**:
+  - RFC 1928 and RFC 1929 compliant client with domain resolution (`socks5h://`), IPv4, IPv6, and authentication.
+  - Wireable via `-socks` CLI flag or `ALL_PROXY` / `SOCKS5_PROXY` environment variables.
+- **Bring Your Own Key (BYOK) Pass-through**:
+  - Passes client API keys or custom base URLs directly to upstream inference engines when provided.
+
+---
+
+## Architecture & Model Aliases
+
+The gateway serves the following models under `/v1/models`:
+
+| Model ID | Target Model | Description |
+|---|---|---|
+| `Qwen/Qwen3.8-Flash-Next` | `Qwen/Qwen3.8-Flash-Next` | Primary playground model (125B MoE, 6B activated) |
+| `qwen3.8-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Standard lowercase alias |
+| `qwen-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Shorthand alias |
+| `qwen-flash` | `Qwen/Qwen3.8-Flash-Next` | Quick convenience alias |
+
+Any unlisted custom model name requested by the client is passed through directly.
+
+---
+
+## Build Instructions
+
+Build binary with Go:
+
+```bash
+make qflash
+```
+
+Or run test suite:
+
+```bash
+make test
+```
+
+The resulting binary will be placed at `bin/qflash`.
+
+---
+
+## Configuration Flags & Environment Variables
+
+| Flag | Shorthand | Environment Variable | Default | Description |
+|---|---|---|---|---|
+| `-port` | | `PORT` | `8080` | Port to bind the HTTP server |
+| `-endpoint` | | | `https://halvo78-qwen3-8-flash-next-playground.hf.space` | Upstream Gradio space base URL |
+| `-model` | | | `Qwen/Qwen3.8-Flash-Next` | Default model ID |
+| `-thinking` | `-enable-thinking` | | `true` | Enable chain-of-thought reasoning by default |
+| `-hf-token` | | `HF_TOKEN` | `""` | Hugging Face user access token |
+| `-api-key` | | `OPENAI_API_KEY` / `QWEN_API_KEY` | `""` | Upstream inference engine API key |
+| `-base-url` | | `OPENAI_BASE_URL` / `QWEN_BASE_URL` | `""` | Upstream inference engine base URL |
+| `-user-agent` | `-ua` | `USER_AGENT` | Firefox 153 on Linux | Custom User-Agent header |
+| `-socks` | `-proxy`, `-socks5` | `ALL_PROXY`, `SOCKS5_PROXY` | `""` | SOCKS5 proxy URL (`socks5://127.0.0.1:1080`) |
+
+---
+
+## Usage Examples
+
+### 1. Launch Gateway
+
+```bash
+./bin/qflash -port 8080
+```
+
+### 2. List Models
+
+```bash
+curl http://127.0.0.1:8080/v1/models
+```
+
+### 3. Non-Streaming Chat Completion
+
+```bash
+curl -X POST http://127.0.0.1:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "qwen3.8-flash-next",
+    "messages": [
+      {"role": "user", "content": "Explain QSA micro-blocks in one sentence."}
+    ],
+    "stream": false
+  }'
+```
+
+### 4. Streaming Chat Completion (Real-Time SSE)
+
+```bash
+curl -N -X POST http://127.0.0.1:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "qwen-flash",
+    "messages": [
+      {"role": "user", "content": "Write a quick Python countdown loop."}
+    ],
+    "stream": true
+  }'
+```
+
+### 5. Instruct Mode (Disable Thinking)
+
+```bash
+curl -X POST http://127.0.0.1:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "qwen-flash",
+    "messages": [
+      {"role": "user", "content": "Hello!"}
+    ],
+    "reasoning_effort": "none"
+  }'
+```
+
+### 6. Python OpenAI SDK Integration
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://127.0.0.1:8080/v1",
+    api_key="sk-dummy"
+)
+
+stream = client.chat.completions.create(
+    model="qwen3.8-flash-next",
+    messages=[
+        {"role": "user", "content": "Prove that the sum of the first n odd numbers is n^2."}
+    ],
+    stream=True
+)
+
+for chunk in stream:
+    delta = chunk.choices[0].delta
+    if hasattr(delta, "reasoning_content") and delta.reasoning_content:
+        print(f"[THINK] {delta.reasoning_content}", end="", flush=True)
+    if delta.content:
+        print(delta.content, end="", flush=True)
+print()
+```