luxferre/qflash
1
0
Fork 0
Code Actions Activity

feat: remove ZeroGPU endpoints from default pool

Browse Source
This commit is contained in:
Luxferre
2026-09-05 21:10:59 +03:00
parent 6877e2b358
commit 404779f8e8
3 changed files with 34 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+20 -18
View File
@@ -4,8 +4,8 @@
Qflash is a standalone, single-binary gateway that exposes Qwen3.8 model spaces on Hugging Face through a standard OpenAI-compatible API. It automatically detects and supports multiple upstream protocols: Qflash is a standalone, single-binary gateway that exposes Qwen3.8 model spaces on Hugging Face through a standard OpenAI-compatible API. It automatically detects and supports multiple upstream protocols:
- gradio `/respond` endpoints (such as `https://microhero-qwen3-8-27b-uncensored-chat.hf.space`, running live on ZeroGPU) - direct OpenAI `/v1/chat/completions` endpoints (such as `https://wanyamaelis-qwen3-8-27b.hf.space` and `https://apathy-exe-qwen3-8-flash-next.hf.space`, running persistent `llama-server` instances on CPU with zero quota limits)
- direct OpenAI `/v1/chat/completions` endpoints (such as `https://wanyamaelis-qwen3-8-27b.hf.space` and `https://apathy-exe-qwen3-8-flash-next.hf.space`, running `llama-server` on CPU) - gradio `/respond` endpoints (supported for custom spaces such as `https://microhero-qwen3-8-27b-uncensored-chat.hf.space`)
- gradio `/chat_response` endpoints (such as `https://halvo78-qwen3-8-flash-next-playground.hf.space`) - gradio `/chat_response` endpoints (such as `https://halvo78-qwen3-8-flash-next-playground.hf.space`)
It translates standard `/v1/chat/completions` and `/v1/models` requests and Server-Sent Events (SSE) stream protocols into upstream formats, allowing any standard OpenAI-compatible client, agent, or IDE to interface with Qwen3.8 models without modification. It translates standard `/v1/chat/completions` and `/v1/models` requests and Server-Sent Events (SSE) stream protocols into upstream formats, allowing any standard OpenAI-compatible client, agent, or IDE to interface with Qwen3.8 models without modification.
@@ -13,7 +13,7 @@ It translates standard `/v1/chat/completions` and `/v1/models` requests and Serv
## Features ## Features
- openai-compatible chat completions (streaming and non-streaming) - openai-compatible chat completions (streaming and non-streaming)
- automatic upstream failover across ZeroGPU and non-ZeroGPU endpoints - automatic upstream failover across configured non-ZeroGPU endpoints
- smart endpoint cooldown (5 minutes on quota exhaustion, 30 seconds on network errors) - smart endpoint cooldown (5 minutes on quota exhaustion, 30 seconds on network errors)
- automatic upstream endpoint detection (`respond`, `openai`, `chat_response`) - automatic upstream endpoint detection (`respond`, `openai`, `chat_response`)
- deep reasoning extraction with thinking trace passthrough (`<think>` tags and blockquotes mapped to `reasoning_content`) - deep reasoning extraction with thinking trace passthrough (`<think>` tags and blockquotes mapped to `reasoning_content`)
@@ -30,21 +30,21 @@ It translates standard `/v1/chat/completions` and `/v1/models` requests and Serv
| Space | Model | Hardware | Protocol | Notes | | Space | Model | Hardware | Protocol | Notes |
|---|---|---|---|---| |---|---|---|---|---|
| `MicroHERO/qwen3.8-27b-uncensored-chat` *(primary)* | Qwen3.8-27B Uncensored | ZeroGPU (A10G) | Gradio `/respond` | Fast live GPU inference (~2s), uncensored weights | | `Wanyamaelis/Qwen3.8-27B` *(primary)* | Qwen3.8-27B (MTP) | CPU basic (8 vCPU) | Native OpenAI `/v1` | Default endpoint, non-ZeroGPU, no quota limits, fast speculative decoding (~5s) |
| `Wanyamaelis/Qwen3.8-27B` *(fallback)* | Qwen3.8-27B (MTP) | CPU basic (8 vCPU) | Native OpenAI `/v1` | Non-ZeroGPU, no quota limits, fast speculative decoding (~5s) | | `apathy-exe/Qwen3.8-27B` *(fallback)* | Qwen3.8-27B (MTP) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | Non-ZeroGPU, no quota limits, speculative decoding |
| `apathy-exe/Qwen3.8-27B` | Qwen3.8-27B (MTP) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | Non-ZeroGPU, no quota limits, speculative decoding | | `apathy-exe/Qwen3.8-Flash-Next` *(fallback)* | Qwen3.8-Flash-Next (~177B) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | Non-ZeroGPU, no quota limits, full 177B Flash-Next model |
| `apathy-exe/Qwen3.8-Flash-Next` | Qwen3.8-Flash-Next (~177B) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | Non-ZeroGPU, no quota limits, full 177B Flash-Next model | | `MicroHERO/qwen3.8-27b-uncensored-chat` | Qwen3.8-27B Uncensored | ZeroGPU (A10G) | Gradio `/respond` | Fast live GPU inference, can be configured via custom `-endpoints` |
| `Halvo78/qwen3-8-flash-next-playground` | Qwen3.8-Flash-Next | CPU basic | Gradio `/chat_response` | Sandbox client; requires BYOK API key/base URL | | `Halvo78/qwen3-8-flash-next-playground` | Qwen3.8-Flash-Next | CPU basic | Gradio `/chat_response` | Sandbox client; requires BYOK API key/base URL |
### Auto-failover and ZeroGPU quota handling ### Auto-failover and quota handling
Spaces running on ZeroGPU provide free A10G compute, but anonymous requests share a small pool per IP address. When ZeroGPU runs limit is reached, upstream returns a quota error (`429` or `ZeroGPU runs limit`). Default endpoints run on persistent non-ZeroGPU hardware with no daily runs limit. When custom ZeroGPU spaces are configured, anonymous requests share a small pool per IP address. When runs limit is reached, upstream returns a quota error (`429` or `ZeroGPU runs limit`).
Qflash handles this seamlessly: Qflash handles upstream reliability seamlessly:
- with auto-failover enabled (default), when a ZeroGPU space hits its runs limit, it is placed on a 5-minute cooldown and the gateway automatically fails over to the next configured endpoint (e.g. `wanyamaelis-qwen3-8-27b`, which runs on CPU with zero quota limits) - with auto-failover enabled (default), when an endpoint encounters an error or timeout, it is placed on cooldown and the gateway automatically fails over to the next configured endpoint
- transient network errors trigger a shorter 30-second cooldown before retrying - transient network errors trigger a 30-second cooldown, while quota limits trigger a 5-minute cooldown
- you can also provide a free Hugging Face personal access token (`https://huggingface.co/settings/tokens`) via the `HF_TOKEN` environment variable, the `-hf-token` CLI flag, or the `Authorization: Bearer hf_...` header to authenticate ZeroGPU requests directly - optional ZeroGPU or private spaces can be authenticated using a Hugging Face personal access token (`https://huggingface.co/settings/tokens`) via the `HF_TOKEN` environment variable, the `-hf-token` CLI flag, or the `Authorization: Bearer hf_...` header
## Installation ## Installation
@@ -80,9 +80,11 @@ The gateway advertises the following models under `/v1/models`:
| Model ID | Target model | Description | | Model ID | Target model | Description |
|---|---|---| |---|---|---|
| `Qwen/Qwen3.8-27B-Uncensored` | `Qwen/Qwen3.8-27B-Uncensored` | Default primary live model | | `Qwen/Qwen3.8-27B` | `Qwen/Qwen3.8-27B` | Default primary live model |
| `Qwen/Qwen3.8-27B-Uncensored` | `Qwen/Qwen3.8-27B-Uncensored` | Uncensored model identifier |
| `Qwen/Qwen3.8-Flash-Next` | `Qwen/Qwen3.8-Flash-Next` | Flash-Next model identifier | | `Qwen/Qwen3.8-Flash-Next` | `Qwen/Qwen3.8-Flash-Next` | Flash-Next model identifier |
| `qwen3.8-27b-uncensored` | `Qwen/Qwen3.8-27B-Uncensored` | Standard lowercase alias | | `qwen3.8-27b` | `Qwen/Qwen3.8-27B` | Standard lowercase alias |
| `qwen3.8-27b-uncensored` | `Qwen/Qwen3.8-27B-Uncensored` | Uncensored lowercase alias |
| `qwen3.8-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Flash-Next lowercase alias | | `qwen3.8-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Flash-Next lowercase alias |
| `qwen-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Shorthand alias | | `qwen-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Shorthand alias |
| `qwen-flash` | `Qwen/Qwen3.8-Flash-Next` | Quick convenience alias | | `qwen-flash` | `Qwen/Qwen3.8-Flash-Next` | Quick convenience alias |
@@ -98,15 +100,15 @@ Run the gateway with default auto-failover endpoints:
qflash qflash
``` ```
By default, this listens on `http://127.0.0.1:8080` with failover configured across `MicroHERO` (ZeroGPU), `wanyamaelis` (CPU non-ZeroGPU), and `apathy-exe` endpoints. By default, this listens on `http://127.0.0.1:8080` with failover configured across persistent non-ZeroGPU endpoints (`wanyamaelis` and `apathy-exe`).
Available flags: Available flags:
- `-port` — tcp port to listen on (default `8080`) - `-port` — tcp port to listen on (default `8080`)
- `-endpoints` / `-endpoint` — comma-separated upstream space or OpenAI URLs (default list of 4 endpoints, `QFLASH_ENDPOINTS` / `QFLASH_ENDPOINT` env) - `-endpoints` / `-endpoint` — comma-separated upstream space or OpenAI URLs (default list of 3 non-ZeroGPU endpoints, `QFLASH_ENDPOINTS` / `QFLASH_ENDPOINT` env)
- `-failover` / `-auto-failover` — enable automatic failover across endpoints on quota exhaustion or error (default `true`, `QFLASH_FAILOVER` env) - `-failover` / `-auto-failover` — enable automatic failover across endpoints on quota exhaustion or error (default `true`, `QFLASH_FAILOVER` env)
- `-mode` — upstream protocol mode: `auto`, `respond`, `chat_response`, `openai` (default `auto`, `QFLASH_MODE` env) - `-mode` — upstream protocol mode: `auto`, `respond`, `chat_response`, `openai` (default `auto`, `QFLASH_MODE` env)
- `-model` — exposed model name (default `Qwen/Qwen3.8-27B-Uncensored`, `QFLASH_MODEL` env) - `-model` — exposed model name (default `Qwen/Qwen3.8-27B`, `QFLASH_MODEL` env)
- `-thinking` / `-enable-thinking` — enable chain-of-thought reasoning by default (default `true`) - `-thinking` / `-enable-thinking` — enable chain-of-thought reasoning by default (default `true`)
- `-hf-token` / `-token` — Hugging Face API token for ZeroGPU quota or private spaces (`HF_TOKEN` env) - `-hf-token` / `-token` — Hugging Face API token for ZeroGPU quota or private spaces (`HF_TOKEN` env)
- `-api-key` — upstream inference engine API key for BYOK mode (`OPENAI_API_KEY` / `QWEN_API_KEY` env) - `-api-key` — upstream inference engine API key for BYOK mode (`OPENAI_API_KEY` / `QWEN_API_KEY` env)
main.go
+9 -6
View File
@@ -32,7 +32,6 @@ var (
ConfiguredModel string ConfiguredModel string
EnableThinkingDefault = true EnableThinkingDefault = true
DefaultEndpoints = []string{ DefaultEndpoints = []string{
"https://microhero-qwen3-8-27b-uncensored-chat.hf.space",
"https://wanyamaelis-qwen3-8-27b.hf.space", "https://wanyamaelis-qwen3-8-27b.hf.space",
"https://apathy-exe-qwen3-8-27b.hf.space", "https://apathy-exe-qwen3-8-27b.hf.space",
"https://apathy-exe-qwen3-8-flash-next.hf.space", "https://apathy-exe-qwen3-8-flash-next.hf.space",
@@ -509,13 +508,15 @@ func EffectiveModelID(reqModel string, defaultModel string) string {
switch strings.ToLower(clean) { switch strings.ToLower(clean) {
case "qwen/qwen3.8-flash-next", "qwen3.8-flash-next", "qwen-flash-next", "qwen-flash", "qwen3.8-flash": case "qwen/qwen3.8-flash-next", "qwen3.8-flash-next", "qwen-flash-next", "qwen-flash", "qwen3.8-flash":
return "Qwen/Qwen3.8-Flash-Next" return "Qwen/Qwen3.8-Flash-Next"
case "qwen/qwen3.8-27b-uncensored", "qwen3.8-27b-uncensored", "qwen-27b", "qwen3.8-27b": case "qwen/qwen3.8-27b", "qwen3.8-27b", "qwen-27b":
return "Qwen/Qwen3.8-27B"
case "qwen/qwen3.8-27b-uncensored", "qwen3.8-27b-uncensored":
return "Qwen/Qwen3.8-27B-Uncensored" return "Qwen/Qwen3.8-27B-Uncensored"
case "qwen": case "qwen":
if defaultModel != "" { if defaultModel != "" {
return defaultModel return defaultModel
} }
return "Qwen/Qwen3.8-Flash-Next" return "Qwen/Qwen3.8-27B"
default: default:
return clean return clean
} }
@@ -1250,7 +1251,7 @@ func parseEndpointList(rawList []string) []*EndpointNode {
func NewQwenService(endpoints []string, modelName, mode, token, apiKey, baseURL, socksProxy string, enableThinking, autoFailover bool) *QwenService { func NewQwenService(endpoints []string, modelName, mode, token, apiKey, baseURL, socksProxy string, enableThinking, autoFailover bool) *QwenService {
if modelName == "" { if modelName == "" {
modelName = "Qwen/Qwen3.8-27B-Uncensored" modelName = "Qwen/Qwen3.8-27B"
} }
if mode == "" { if mode == "" {
mode = "auto" mode = "auto"
@@ -1287,13 +1288,15 @@ func (s *QwenService) ListModels() []ModelItem {
now := time.Now().Unix() now := time.Now().Unix()
primaryID := s.modelName primaryID := s.modelName
if primaryID == "" { if primaryID == "" {
primaryID = "Qwen/Qwen3.8-27B-Uncensored" primaryID = "Qwen/Qwen3.8-27B"
} }
candidates := []string{ candidates := []string{
primaryID, primaryID,
"Qwen/Qwen3.8-27B",
"Qwen/Qwen3.8-27B-Uncensored", "Qwen/Qwen3.8-27B-Uncensored",
"Qwen/Qwen3.8-Flash-Next", "Qwen/Qwen3.8-Flash-Next",
"qwen3.8-27b",
"qwen3.8-27b-uncensored", "qwen3.8-27b-uncensored",
"qwen3.8-flash-next", "qwen3.8-flash-next",
"qwen-flash-next", "qwen-flash-next",
@@ -2204,7 +2207,7 @@ func main() {
autoFailover := flag.Bool("failover", true, "Enable automatic failover across endpoints on error or quota limit") autoFailover := flag.Bool("failover", true, "Enable automatic failover across endpoints on error or quota limit")
flag.BoolVar(autoFailover, "auto-failover", true, "Alias for -failover") flag.BoolVar(autoFailover, "auto-failover", true, "Alias for -failover")
defaultModelVal := "Qwen/Qwen3.8-27B-Uncensored" defaultModelVal := "Qwen/Qwen3.8-27B"
if envModel := os.Getenv("QFLASH_MODEL"); envModel != "" { if envModel := os.Getenv("QFLASH_MODEL"); envModel != "" {
defaultModelVal = envModel defaultModelVal = envModel
} }
qflash_test.go
+5 -1
View File
@@ -71,8 +71,12 @@ func TestEffectiveModelID(t *testing.T) {
"qwen3.8-flash-next": def, "qwen3.8-flash-next": def,
"qwen-flash-next": def, "qwen-flash-next": def,
"qwen-flash": def, "qwen-flash": def,
"qwen3.8-flash": def,
"Qwen/Qwen3.8-Flash-Next": def, "Qwen/Qwen3.8-Flash-Next": def,
"qwen3.8-27b": "Qwen/Qwen3.8-27B",
"qwen-27b": "Qwen/Qwen3.8-27B",
"Qwen/Qwen3.8-27B": "Qwen/Qwen3.8-27B",
"qwen3.8-27b-uncensored": "Qwen/Qwen3.8-27B-Uncensored",
"Qwen/Qwen3.8-27B-Uncensored": "Qwen/Qwen3.8-27B-Uncensored",
"custom-org/my-model": "custom-org/my-model", "custom-org/my-model": "custom-org/my-model",
} }