luxferre/qflash
1
0
Fork 0
Code Actions Activity

feat: add multi-space support with MicroHERO ZeroGPU default and auto-detection

Browse Source
This commit is contained in:
Luxferre
2026-09-05 16:15:15 +03:00
parent 0da31212b4
commit bb56840d59
3 changed files with 608 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+64 -21
View File
@@ -1,20 +1,47 @@
# Qflash: Free Qwen3.8-Flash-Next LLM gateway
# Qflash: OpenAI-compatible gateway for Qwen3.8 models
## About
Qflash is a standalone, single-binary gateway that exposes the Qwen3.8-Flash-Next Gradio space (`https://halvo78-qwen3-8-flash-next-playground.hf.space`) through an OpenAI-compatible API. It translates the standard `/v1/chat/completions` and `/v1/models` endpoints into Gradio's `/gradio_api/call/chat_response` request and Server-Sent Events (SSE) stream protocol, allowing any standard OpenAI-compatible client, agent, or IDE to interface with Qwen3.8-Flash-Next without modification.
Qflash is a standalone, single-binary gateway that exposes Qwen3.8 model spaces on Hugging Face through a standard OpenAI-compatible API. It automatically detects and supports multiple upstream protocols:
- Gradio `/respond` endpoints (such as `https://microhero-qwen3-8-27b-uncensored-chat.hf.space`, running live on ZeroGPU)
- Direct OpenAI `/v1/chat/completions` endpoints (such as `https://apathy-exe-qwen3-8-flash-next.hf.space`, running `llama-server`)
- Gradio `/chat_response` endpoints (such as `https://halvo78-qwen3-8-flash-next-playground.hf.space`)
It translates standard `/v1/chat/completions` and `/v1/models` requests and Server-Sent Events (SSE) stream protocols into upstream formats, allowing any standard OpenAI-compatible client, agent, or IDE to interface with Qwen3.8 models without modification.
## Features
- OpenAI-compatible chat completions (streaming and non-streaming)
- Deep reasoning extraction with thinking trace passthrough (`<think>` tags and blockquotes mapped to `reasoning_content`)
- Stateful streaming tool call interception (`StreamToolCallFilter`) with zero XML/JSON leakage into `delta.content`
- Real-time token streaming with incremental SSE delivery
- Support for instruct mode via standard `reasoning_effort: "none"`
- Zero-dependency SOCKS5 proxy client (RFC 1928 / RFC 1929) with domain resolution (`socks5h://`), IPv4, IPv6, and auth
- Bring Your Own Key (BYOK) pass-through support via `Authorization: Bearer` or CLI flags
- Fibonacci backoff retry on transient upstream errors
- Zero external dependencies (Go standard library only)
- openai-compatible chat completions (streaming and non-streaming)
- automatic upstream endpoint detection (`respond`, `openai`, `chat_response`)
- deep reasoning extraction with thinking trace passthrough (`<think>` tags and blockquotes mapped to `reasoning_content`)
- stateful streaming tool call interception (`StreamToolCallFilter`) with zero XML or JSON leakage into `delta.content`
- real-time token streaming with incremental SSE delivery
- support for instruct mode via standard `reasoning_effort: "none"`
- zero-dependency SOCKS5 proxy client (RFC 1928 / RFC 1929) with domain resolution (`socks5h://`), IPv4, IPv6, and auth
- bring your own key (BYOK) pass-through support via `Authorization: Bearer` or CLI flags
- zero-gpu quota authentication via free Hugging Face personal tokens (`HF_TOKEN`)
- fibonacci backoff retry on transient upstream errors
- zero external dependencies (Go standard library only)
## Supported upstream spaces
| Space | Model | Hardware | Protocol | Notes |
|---|---|---|---|---|
| `MicroHERO/qwen3.8-27b-uncensored-chat` *(default)* | Qwen3.8-27B Uncensored | ZeroGPU (A10G) | Gradio `/respond` | Fast live inference (~2-5s), vision projector, uncensored build |
| `apathy-exe/Qwen3.8-Flash-Next` | Qwen3.8-Flash-Next (~177B) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | Full 177B model, single execution slot |
| `apathy-exe/Qwen3.8-27B` | Qwen3.8-27B (MTP) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | 27B model with speculative decoding, single execution slot |
| `Halvo78/qwen3-8-flash-next-playground` | Qwen3.8-Flash-Next | CPU basic | Gradio `/chat_response` | Sandbox client; requires BYOK API key/base URL |
### Hugging Face ZeroGPU quota note
Spaces running on ZeroGPU provide free compute. Anonymous requests share a small pool per IP address. To avoid quota limits, set a free Hugging Face token:
```bash
export HF_TOKEN="hf_..."
```
Or pass it via the `-hf-token` flag, or via the `Authorization: Bearer hf_...` header in your client requests. Personal access tokens can be created for free at `https://huggingface.co/settings/tokens`.
## Installation
@@ -50,10 +77,13 @@ The gateway advertises the following models under `/v1/models`:
| Model ID | Target model | Description |
|---|---|---|
| `Qwen/Qwen3.8-Flash-Next` | `Qwen/Qwen3.8-Flash-Next` | Primary playground model (125B MoE, 6B activated) |
| `qwen3.8-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Standard lowercase alias |
| `Qwen/Qwen3.8-27B-Uncensored` | `Qwen/Qwen3.8-27B-Uncensored` | Default primary live model (ZeroGPU) |
| `Qwen/Qwen3.8-Flash-Next` | `Qwen/Qwen3.8-Flash-Next` | Flash-Next model identifier |
| `qwen3.8-27b-uncensored` | `Qwen/Qwen3.8-27B-Uncensored` | Standard lowercase alias |
| `qwen3.8-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Flash-Next lowercase alias |
| `qwen-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Shorthand alias |
| `qwen-flash` | `Qwen/Qwen3.8-Flash-Next` | Quick convenience alias |
| `qwen` | Default model | Generic shorthand alias |
Any unlisted custom model name requested by the client is passed through directly.
@@ -62,16 +92,19 @@ Any unlisted custom model name requested by the client is passed through directl
Run the gateway:
```bash
qflash [-port 8080] [-endpoint https://halvo78-qwen3-8-flash-next-playground.hf.space] [-model Qwen/Qwen3.8-Flash-Next]
qflash
```
By default, this listens on `http://127.0.0.1:8080` and connects to `https://microhero-qwen3-8-27b-uncensored-chat.hf.space`.
Available flags:
- `-port` — TCP port to listen on (default `8080`)
- `-endpoint` — root URL of the Gradio space (default `https://halvo78-qwen3-8-flash-next-playground.hf.space`)
- `-model` — exposed model name (default `Qwen/Qwen3.8-Flash-Next`)
- `-endpoint` — root URL of upstream space (default `https://microhero-qwen3-8-27b-uncensored-chat.hf.space`, `QFLASH_ENDPOINT` env)
- `-mode` — upstream protocol mode: `auto`, `respond`, `chat_response`, `openai` (default `auto`, `QFLASH_MODE` env)
- `-model` — exposed model name (default `Qwen/Qwen3.8-27B-Uncensored`, `QFLASH_MODEL` env)
- `-thinking` / `-enable-thinking` — enable chain-of-thought reasoning by default (default `true`)
- `-hf-token` — optional Hugging Face API token for authenticated spaces (`HF_TOKEN` env)
- `-hf-token` / `-token` — Hugging Face API token for ZeroGPU quota or private spaces (`HF_TOKEN` env)
- `-api-key` — upstream inference engine API key for BYOK mode (`OPENAI_API_KEY` / `QWEN_API_KEY` env)
- `-base-url` — upstream inference engine base URL for BYOK mode (`OPENAI_BASE_URL` / `QWEN_BASE_URL` env)
- `-socks` / `-proxy` / `-socks5` — SOCKS5 proxy URL, e.g. `socks5://127.0.0.1:1080` (`ALL_PROXY` env)
@@ -87,7 +120,7 @@ Example request with curl:
```bash
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen3.8-flash-next","messages":[{"role":"user","content":"Explain QSA micro-blocks in one sentence."}],"stream":false}'
-d '{"model":"qwen","messages":[{"role":"user","content":"Explain quantum superposition in one sentence."}],"stream":false}'
```
Streaming example:
@@ -95,7 +128,7 @@ Streaming example:
```bash
curl -N http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen-flash","messages":[{"role":"user","content":"Count from 1 to 5."}],"stream":true}'
-d '{"model":"qwen","messages":[{"role":"user","content":"Count from 1 to 5."}],"stream":true}'
```
Instruct mode example (disables thinking):
@@ -103,7 +136,17 @@ Instruct mode example (disables thinking):
```bash
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen-flash","messages":[{"role":"user","content":"Hello!"}],"reasoning_effort":"none"}'
-d '{"model":"qwen","messages":[{"role":"user","content":"Hello!"}],"reasoning_effort":"none"}'
```
Using an alternative upstream space:
```bash
# Point to apathy-exe CPU llama-server
qflash -endpoint https://apathy-exe-qwen3-8-flash-next.hf.space -mode openai
# Point to Halvo78 playground with your own API credentials
qflash -endpoint https://halvo78-qwen3-8-flash-next-playground.hf.space -api-key "sk-..." -base-url "https://api.openai.com/v1"
```
Python OpenAI SDK integration:
@@ -117,7 +160,7 @@ client = OpenAI(
)
stream = client.chat.completions.create(
model="qwen3.8-flash-next",
model="qwen",
messages=[
{"role": "user", "content": "Prove that the sum of the first n odd numbers is n^2."}
],