feat: initial release of q38max gateway
This commit is contained in:
Luxferre
@@ -0,0 +1,184 @@
|
||||
# q38max
|
||||
|
||||
Standalone, zero-dependency OpenAI-compatible proxy gateway in Go for the **Qwen 3.8 Max** model (`Qwen/Qwen3.8-Max`) hosted on Hugging Face Spaces (`harpreetsahota-qwen38-max-openlogo-demo.hf.space`).
|
||||
|
||||
## Overview
|
||||
|
||||
`q38max` reverse-engineers the FiftyOne plugin backend operator interface of the Hugging Face space and transforms it into a standard, production-ready OpenAI API endpoint (`/v1/chat/completions` and `/v1/models`).
|
||||
|
||||
### Features
|
||||
|
||||
- **OpenAI Standard Compatibility**: Full drop-in replacement for OpenAI API clients (Curl, Python `openai`, LangChain, LiteLLM, Open-WebUI).
|
||||
- **Zero External Dependencies**: Pure standard library Go implementation (`net/http`, `encoding/json`, `crypto/rand`, `time`).
|
||||
- **Live Streaming SSE & Reasoning**: Streams real-time tokens with separation of reasoning content (`delta.reasoning_content`) and message content (`delta.content`).
|
||||
- **Function / Tool Calling Interception**: Supports OpenAI `tools` specification, system prompt tool schema injection, and stateful streaming interception of tool calls (`delta.tool_calls` and `finish_reason: "tool_calls"`).
|
||||
- **Session Lifecycle Management**: Thread-safe automatic session creation (`/__session/start`), periodic background heartbeats (`/__session/heartbeat`), and auto-reconnect recovery.
|
||||
- **Fibonacci Backoff Retry**: Resilient against network hiccups and transient timeouts.
|
||||
|
||||
---
|
||||
|
||||
## Architecture & Upstream Protocol
|
||||
|
||||
```
|
||||
+---------------------------+ OpenAI HTTP / SSE +------------------------+
|
||||
| Client (Python / Curl / | ===========================> | q38max Gateway |
|
||||
| OpenAI SDK / Open-WebUI) | | (localhost:8080) |
|
||||
+---------------------------+ +------------------------+
|
||||
|
|
||||
| FiftyOne Session &
|
||||
| Operator API
|
||||
v
|
||||
+------------------------+
|
||||
| HuggingFace Space |
|
||||
| FiftyOne Backend |
|
||||
| (Qwen 3.8 Max Model) |
|
||||
+------------------------+
|
||||
```
|
||||
|
||||
### Upstream Flow:
|
||||
1. `POST /__session/start` -> Allocates an ephemeral session token `X-FiftyOne-Session` and dataset clone.
|
||||
2. `POST /operators/execute` -> Dispatches the `@harpreetsahota/qwen38-max/qwen38_chat` operator with method `"ask"`.
|
||||
3. Polling Loops:
|
||||
- `get_thinking_chunk`: Extracts newly generated reasoning tokens in real-time.
|
||||
- `get_stream_chunk`: Extracts newly generated message content tokens in real-time.
|
||||
|
||||
---
|
||||
|
||||
## Build & Run
|
||||
|
||||
### Build
|
||||
```bash
|
||||
make q38max
|
||||
```
|
||||
|
||||
Binary is output to `bin/q38max`.
|
||||
|
||||
### Run
|
||||
```bash
|
||||
./bin/q38max -port 8080
|
||||
```
|
||||
|
||||
### CLI Flags
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `-port` | `8080` | Port to listen on |
|
||||
| `-space-url` | `https://harpreetsahota-qwen38-max-openlogo-demo.hf.space` | Upstream Hugging Face Space URL |
|
||||
| `-sample-path` | `/home/user/datasets/openlogo/data/data_0/logos32plus_002359.jpg` | Container image sample path |
|
||||
| `-model` | `qwen-3.8-max` | Default model identifier |
|
||||
| `-timeout` | `300` | Upstream timeout in seconds |
|
||||
| `-user-agent` / `-ua` | `""` | Custom User-Agent header |
|
||||
| `-hf-token` | `""` | Optional Hugging Face token |
|
||||
|
||||
---
|
||||
|
||||
## API Usage Examples
|
||||
|
||||
### 1. List Models
|
||||
```bash
|
||||
curl http://localhost:8080/v1/models
|
||||
```
|
||||
|
||||
### 2. Non-Streaming Chat Completion
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen-3.8-max",
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is the capital of Germany? Answer in 1 word."}
|
||||
],
|
||||
"reasoning_effort": "none",
|
||||
"max_tokens": 50
|
||||
}'
|
||||
```
|
||||
|
||||
### 3. Streaming Chat Completion with Reasoning
|
||||
```bash
|
||||
curl -N -X POST http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen-3.8-max",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Calculate 25 * 25 and explain in one sentence."}
|
||||
],
|
||||
"stream": true,
|
||||
"reasoning_effort": "medium",
|
||||
"max_tokens": 150
|
||||
}'
|
||||
```
|
||||
|
||||
### 4. Function / Tool Calling
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen-3.8-max",
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is the weather in Berlin?"}
|
||||
],
|
||||
"tools": [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string"}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"reasoning_effort": "none"
|
||||
}'
|
||||
```
|
||||
|
||||
### 5. Python OpenAI Client Example
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8080/v1",
|
||||
api_key="none"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="qwen-3.8-max",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a short haiku about computers."}
|
||||
],
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
delta = chunk.choices[0].delta
|
||||
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||
print(f"[Thinking] {delta.reasoning_content}", end="", flush=True)
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
print()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
Run unit tests:
|
||||
```bash
|
||||
make test
|
||||
```
|
||||
|
||||
Run end-to-end integration tests:
|
||||
```bash
|
||||
./xtest.sh 8080
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
Public Domain / Unlicense
|
||||
Reference in New Issue
Block a user