luxferre/qflash
1
0
Fork 0
Code Actions Activity

feat: add non-ZeroGPU endpoints, automatic failover, and endpoint cooldown

Browse Source
This commit is contained in:
Luxferre
2026-09-05 16:39:35 +03:00
parent bb56840d59
commit 54352f3da7
3 changed files with 433 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+24 -20
View File
@@ -4,15 +4,17 @@
Qflash is a standalone, single-binary gateway that exposes Qwen3.8 model spaces on Hugging Face through a standard OpenAI-compatible API. It automatically detects and supports multiple upstream protocols:
- Gradio `/respond` endpoints (such as `https://microhero-qwen3-8-27b-uncensored-chat.hf.space`, running live on ZeroGPU)
- Direct OpenAI `/v1/chat/completions` endpoints (such as `https://apathy-exe-qwen3-8-flash-next.hf.space`, running `llama-server`)
- Gradio `/chat_response` endpoints (such as `https://halvo78-qwen3-8-flash-next-playground.hf.space`)
- gradio `/respond` endpoints (such as `https://microhero-qwen3-8-27b-uncensored-chat.hf.space`, running live on ZeroGPU)
- direct OpenAI `/v1/chat/completions` endpoints (such as `https://wanyamaelis-qwen3-8-27b.hf.space` and `https://apathy-exe-qwen3-8-flash-next.hf.space`, running `llama-server` on CPU)
- gradio `/chat_response` endpoints (such as `https://halvo78-qwen3-8-flash-next-playground.hf.space`)
It translates standard `/v1/chat/completions` and `/v1/models` requests and Server-Sent Events (SSE) stream protocols into upstream formats, allowing any standard OpenAI-compatible client, agent, or IDE to interface with Qwen3.8 models without modification.
## Features
- openai-compatible chat completions (streaming and non-streaming)
- automatic upstream failover across ZeroGPU and non-ZeroGPU endpoints
- smart endpoint cooldown (5 minutes on quota exhaustion, 30 seconds on network errors)
- automatic upstream endpoint detection (`respond`, `openai`, `chat_response`)
- deep reasoning extraction with thinking trace passthrough (`<think>` tags and blockquotes mapped to `reasoning_content`)
- stateful streaming tool call interception (`StreamToolCallFilter`) with zero XML or JSON leakage into `delta.content`
@@ -28,20 +30,21 @@ It translates standard `/v1/chat/completions` and `/v1/models` requests and Serv
| Space | Model | Hardware | Protocol | Notes |
|---|---|---|---|---|
| `MicroHERO/qwen3.8-27b-uncensored-chat` *(default)* | Qwen3.8-27B Uncensored | ZeroGPU (A10G) | Gradio `/respond` | Fast live inference (~2-5s), vision projector, uncensored build |
| `apathy-exe/Qwen3.8-Flash-Next` | Qwen3.8-Flash-Next (~177B) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | Full 177B model, single execution slot |
| `apathy-exe/Qwen3.8-27B` | Qwen3.8-27B (MTP) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | 27B model with speculative decoding, single execution slot |
| `MicroHERO/qwen3.8-27b-uncensored-chat` *(primary)* | Qwen3.8-27B Uncensored | ZeroGPU (A10G) | Gradio `/respond` | Fast live GPU inference (~2s), uncensored weights |
| `Wanyamaelis/Qwen3.8-27B` *(fallback)* | Qwen3.8-27B (MTP) | CPU basic (8 vCPU) | Native OpenAI `/v1` | Non-ZeroGPU, no quota limits, fast speculative decoding (~5s) |
| `apathy-exe/Qwen3.8-27B` | Qwen3.8-27B (MTP) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | Non-ZeroGPU, no quota limits, speculative decoding |
| `apathy-exe/Qwen3.8-Flash-Next` | Qwen3.8-Flash-Next (~177B) | CPU (OpenMP/AVX512) | Native OpenAI `/v1` | Non-ZeroGPU, no quota limits, full 177B Flash-Next model |
| `Halvo78/qwen3-8-flash-next-playground` | Qwen3.8-Flash-Next | CPU basic | Gradio `/chat_response` | Sandbox client; requires BYOK API key/base URL |
### Hugging Face ZeroGPU quota note
### Auto-failover and ZeroGPU quota handling
Spaces running on ZeroGPU provide free compute. Anonymous requests share a small pool per IP address. To avoid quota limits, set a free Hugging Face token:
Spaces running on ZeroGPU provide free A10G compute, but anonymous requests share a small pool per IP address. When ZeroGPU runs limit is reached, upstream returns a quota error (`429` or `ZeroGPU runs limit`).
```bash
export HF_TOKEN="hf_..."
```
Qflash handles this seamlessly:
Or pass it via the `-hf-token` flag, or via the `Authorization: Bearer hf_...` header in your client requests. Personal access tokens can be created for free at `https://huggingface.co/settings/tokens`.
- with auto-failover enabled (default), when a ZeroGPU space hits its runs limit, it is placed on a 5-minute cooldown and the gateway automatically fails over to the next configured endpoint (e.g. `wanyamaelis-qwen3-8-27b`, which runs on CPU with zero quota limits)
- transient network errors trigger a shorter 30-second cooldown before retrying
- you can also provide a free Hugging Face personal access token (`https://huggingface.co/settings/tokens`) via the `HF_TOKEN` environment variable, the `-hf-token` CLI flag, or the `Authorization: Bearer hf_...` header to authenticate ZeroGPU requests directly
## Installation
@@ -77,7 +80,7 @@ The gateway advertises the following models under `/v1/models`:
| Model ID | Target model | Description |
|---|---|---|
| `Qwen/Qwen3.8-27B-Uncensored` | `Qwen/Qwen3.8-27B-Uncensored` | Default primary live model (ZeroGPU) |
| `Qwen/Qwen3.8-27B-Uncensored` | `Qwen/Qwen3.8-27B-Uncensored` | Default primary live model |
| `Qwen/Qwen3.8-Flash-Next` | `Qwen/Qwen3.8-Flash-Next` | Flash-Next model identifier |
| `qwen3.8-27b-uncensored` | `Qwen/Qwen3.8-27B-Uncensored` | Standard lowercase alias |
| `qwen3.8-flash-next` | `Qwen/Qwen3.8-Flash-Next` | Flash-Next lowercase alias |
@@ -89,18 +92,19 @@ Any unlisted custom model name requested by the client is passed through directl
## Usage
Run the gateway:
Run the gateway with default auto-failover endpoints:
```bash
qflash
```
By default, this listens on `http://127.0.0.1:8080` and connects to `https://microhero-qwen3-8-27b-uncensored-chat.hf.space`.
By default, this listens on `http://127.0.0.1:8080` with failover configured across `MicroHERO` (ZeroGPU), `wanyamaelis` (CPU non-ZeroGPU), and `apathy-exe` endpoints.
Available flags:
- `-port` — TCP port to listen on (default `8080`)
- `-endpoint` — root URL of upstream space (default `https://microhero-qwen3-8-27b-uncensored-chat.hf.space`, `QFLASH_ENDPOINT` env)
- `-port` — tcp port to listen on (default `8080`)
- `-endpoints` / `-endpoint` — comma-separated upstream space or OpenAI URLs (default list of 4 endpoints, `QFLASH_ENDPOINTS` / `QFLASH_ENDPOINT` env)
- `-failover` / `-auto-failover` — enable automatic failover across endpoints on quota exhaustion or error (default `true`, `QFLASH_FAILOVER` env)
- `-mode` — upstream protocol mode: `auto`, `respond`, `chat_response`, `openai` (default `auto`, `QFLASH_MODE` env)
- `-model` — exposed model name (default `Qwen/Qwen3.8-27B-Uncensored`, `QFLASH_MODEL` env)
- `-thinking` / `-enable-thinking` — enable chain-of-thought reasoning by default (default `true`)
@@ -139,11 +143,11 @@ curl http://localhost:8080/v1/chat/completions \
-d '{"model":"qwen","messages":[{"role":"user","content":"Hello!"}],"reasoning_effort":"none"}'
```
Using an alternative upstream space:
Using custom upstream spaces or a single endpoint:
```bash
# Point to apathy-exe CPU llama-server
qflash -endpoint https://apathy-exe-qwen3-8-flash-next.hf.space -mode openai
# Point to a single CPU llama-server without failover
qflash -endpoint https://wanyamaelis-qwen3-8-27b.hf.space -failover=false
# Point to Halvo78 playground with your own API credentials
qflash -endpoint https://halvo78-qwen3-8-flash-next-playground.hf.space -api-key "sk-..." -base-url "https://api.openai.com/v1"