ini upl
This commit is contained in:
Luxferre
+10
@@ -0,0 +1,10 @@
|
|||||||
|
# Binaries
|
||||||
|
bin/
|
||||||
|
*.exe
|
||||||
|
*.dll
|
||||||
|
*.so
|
||||||
|
*.dylib
|
||||||
|
|
||||||
|
# Test artifacts
|
||||||
|
*.test
|
||||||
|
*.out
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Makefile for k3r053n3 (Kimi K3 LLM Gateway)
|
||||||
|
|
||||||
|
k3r053n3:
|
||||||
|
go build -trimpath -ldflags="-s -w" -o bin/k3r053n3 .
|
||||||
|
|
||||||
|
test:
|
||||||
|
go test -v ./...
|
||||||
|
|
||||||
|
all: k3r053n3
|
||||||
|
|
||||||
|
clean:
|
||||||
|
rm -rf bin/
|
||||||
|
|
||||||
|
.PHONY: all clean k3r053n3 test
|
||||||
@@ -0,0 +1,307 @@
|
|||||||
|
# k3r053n3: Kimi K3 OpenAI-compatible proxy gateway
|
||||||
|
|
||||||
|
A standalone, high-performance, zero-dependency Go proxy gateway that exposes a standard OpenAI-compatible API (`/v1/chat/completions` and `/v1/models`) for the **Kimi K3** model hosted on the [`cw-105/kimi-k3-gguf-demo`](https://cw-105-kimi-k3-gguf-demo.hf.space) Gradio space.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Zero external dependencies**: pure Go standard library (`net/http`, `encoding/json`, `bufio`, `bytes`, `crypto/rand`, `flag`, `strings`, `time`).
|
||||||
|
- **SOCKS5 proxy support**: zero-dependency built-in RFC 1928 / RFC 1929 SOCKS5 client supporting domain name resolution (`socks5h://`), IPv4/IPv6, and username/password authentication (via `-socks`, `-proxy`, or `ALL_PROXY`/`all_proxy`/`SOCKS5_PROXY`/`socks5_proxy` environment variables).
|
||||||
|
- **Reasoning effort support**: defaults to `"max"` reasoning effort out of the box, with support for `"max"`, `"high"`, `"low"`, and `"default"` (via `reasoning_effort` request field or CLI flag).
|
||||||
|
- **Real-time token streaming**: Server-Sent Events (SSE) streaming engine (`stream: true`) with separate token-by-token emission for `delta.reasoning_content` and `delta.content`.
|
||||||
|
- **Reasoning extraction**: clean separation of `<think>...</think>` internal thoughts into `reasoning_content` (streaming chunks and non-streaming messages) without leaking raw tags into `content`.
|
||||||
|
- **OpenAI-compatible tool calling**:
|
||||||
|
- automatic tool definition formatting into system instructions.
|
||||||
|
- multi-turn tool execution response formatting (`role: "tool"` / `role: "function"`).
|
||||||
|
- real-time stream interceptor (`StreamToolInterceptor`) that catches `<tool_call>` blocks on the fly and emits standard OpenAI `delta.tool_calls` chunks with `finish_reason: "tool_calls"`.
|
||||||
|
- non-streaming tool call parsing with structured `tool_calls` and `finish_reason: "tool_calls"`.
|
||||||
|
- **Backend & model routing**: route between multiple upstream backends (`direct:together`, `direct:fireworks`, `hf:together`, `hf:fireworks-ai`, `hf:featherless-ai`, `hf:baseten`) dynamically or via model suffix (`kimi-k3:together`, `kimi-k3:fireworks`, etc.).
|
||||||
|
- **Reliability & resilience**: automatic Fibonacci exponential backoff retry mechanism (`DoWithFibonacciRetry`) on upstream network connections.
|
||||||
|
- **Full CORS support**: ready for direct browser integration, web frontends, and OpenAI-compatible client libraries.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
### Installation
|
||||||
|
|
||||||
|
Install directly with `go install`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go install code.luxferre.top/luxferre/k3r053n3@latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### Build from source
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make k3r053n3
|
||||||
|
```
|
||||||
|
|
||||||
|
Or build manually with Go:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go build -trimpath -ldflags="-s -w" -o bin/k3r053n3 .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./bin/k3r053n3
|
||||||
|
```
|
||||||
|
|
||||||
|
By default, the server starts on port `8080` pointing to `https://cw-105-kimi-k3-gguf-demo.hf.space` with default reasoning effort `"max"` and default backend `"direct:together"`.
|
||||||
|
|
||||||
|
## CLI options
|
||||||
|
|
||||||
|
| Flag | Default | Description |
|
||||||
|
||||
|
||||||
|
| `-port` | `8080` | Port to listen on |
|
||||||
|
| `-endpoint` | `https://cw-105-kimi-k3-gguf-demo.hf.space` | Root URL of the Kimi K3 Gradio space |
|
||||||
|
| `-model` | `kimi-k3` | Exposed default model name |
|
||||||
|
| `-backend` | `direct:together` | Default Space backend (`direct:together`, `direct:fireworks`, `hf:together`, `hf:fireworks-ai`, `hf:featherless-ai`, `hf:baseten`) |
|
||||||
|
| `-reasoning` | `max` | Default reasoning effort (`max`, `high`, `low`, `default`) |
|
||||||
|
| `-max-tokens` | `8192` | Default max completion tokens (256 - 8192) |
|
||||||
|
| `-temperature` | `0.7` | Default sampling temperature (0.0 - 1.5) |
|
||||||
|
| `-socks`, `-proxy` | `""` | SOCKS5 proxy URL (`socks5://127.0.0.1:1080` or `socks5://user:pass@host:port`, also checks `ALL_PROXY`/`all_proxy`/`SOCKS5_PROXY`/`socks5_proxy` env vars) |
|
||||||
|
| `-user-agent`, `-ua` | Firefox 153 on Linux | Custom `User-Agent` header for upstream requests |
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
- `GET /` - gateway health check and route overview
|
||||||
|
- `GET /v1/models` (or `GET /models`) - list of available models and backend mappings
|
||||||
|
- `POST /v1/chat/completions` (or `POST /chat/completions`) - OpenAI-compatible chat completions
|
||||||
|
|
||||||
|
## Usage examples
|
||||||
|
|
||||||
|
### 1. Models list
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://localhost:8080/v1/models
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Standard chat completion (non-streaming)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://localhost:8080/v1/chat/completions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "kimi-k3",
|
||||||
|
"messages": [
|
||||||
|
{"role": "user", "content": "What is 25 * 4? Show brief work."}
|
||||||
|
],
|
||||||
|
"stream": false
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Response includes separated `reasoning_content` and `content`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "chatcmpl-88054d2e-2084-4bcd-b9fa-8e99e9267523",
|
||||||
|
"object": "chat.completion",
|
||||||
|
"created": 1788348384,
|
||||||
|
"model": "kimi-k3",
|
||||||
|
"choices": [
|
||||||
|
{
|
||||||
|
"index": 0,
|
||||||
|
"message": {
|
||||||
|
"role": "assistant",
|
||||||
|
"content": "**25 × 4 = 100**\n\nQuick way: 25 × 4 = 25 × 2 × 2 = 50 × 2 = **100**\n\n(Think of it as 4 quarters = 1 dollar.)",
|
||||||
|
"reasoning_content": "The user is asking a simple arithmetic question: 25 * 4..."
|
||||||
|
},
|
||||||
|
"finish_reason": "stop"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"usage": {
|
||||||
|
"prompt_tokens": 0,
|
||||||
|
"completion_tokens": 0,
|
||||||
|
"total_tokens": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Real-time streaming with reasoning deltas
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -N http://localhost:8080/v1/chat/completions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "kimi-k3",
|
||||||
|
"messages": [
|
||||||
|
{"role": "user", "content": "Tell me a 1-sentence joke about computers."}
|
||||||
|
],
|
||||||
|
"stream": true
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Output delivers real-time `delta.reasoning_content` chunks during thinking, followed by `delta.content` chunks for the answer, ending with `[DONE]`:
|
||||||
|
|
||||||
|
```
|
||||||
|
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1788348397,"model":"kimi-k3","choices":[{"index":0,"delta":{"reasoning_content":"Thinking..."}}]}
|
||||||
|
...
|
||||||
|
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1788348397,"model":"kimi-k3","choices":[{"index":0,"delta":{"content":"There are only 10 types of people in the world: those who understand binary and those who don't."}}]}
|
||||||
|
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1788348397,"model":"kimi-k3","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
|
||||||
|
data: [DONE]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Reasoning effort control
|
||||||
|
|
||||||
|
Control reasoning effort via the `reasoning_effort` field (`"max"`, `"high"`, `"low"`, `"default"`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://localhost:8080/v1/chat/completions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "kimi-k3",
|
||||||
|
"reasoning_effort": "low",
|
||||||
|
"messages": [
|
||||||
|
{"role": "user", "content": "Hello!"}
|
||||||
|
]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Tool / function calling
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://localhost:8080/v1/chat/completions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "kimi-k3",
|
||||||
|
"messages": [
|
||||||
|
{"role": "user", "content": "What is the weather in Seattle right now?"}
|
||||||
|
],
|
||||||
|
"tools": [
|
||||||
|
{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "get_weather",
|
||||||
|
"description": "Get current weather for a given city",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"location": {"type": "string", "description": "City name"}
|
||||||
|
},
|
||||||
|
"required": ["location"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"stream": false
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Response emits standard OpenAI `tool_calls` with `finish_reason: "tool_calls"`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "chatcmpl-cbe006c5-1b5d-42d3-ab89-254c7f012061",
|
||||||
|
"object": "chat.completion",
|
||||||
|
"created": 1788348416,
|
||||||
|
"model": "kimi-k3",
|
||||||
|
"choices": [
|
||||||
|
{
|
||||||
|
"index": 0,
|
||||||
|
"message": {
|
||||||
|
"role": "assistant",
|
||||||
|
"content": null,
|
||||||
|
"reasoning_content": "The user is asking about the weather in Seattle...",
|
||||||
|
"tool_calls": [
|
||||||
|
{
|
||||||
|
"index": 0,
|
||||||
|
"id": "call_de6cdf1a_0",
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "get_weather",
|
||||||
|
"arguments": "{\"location\":\"Seattle\"}"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"finish_reason": "tool_calls"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Submitting tool results in follow-up turns
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://localhost:8080/v1/chat/completions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "kimi-k3",
|
||||||
|
"messages": [
|
||||||
|
{"role": "user", "content": "What is the weather in Seattle right now?"},
|
||||||
|
{
|
||||||
|
"role": "assistant",
|
||||||
|
"tool_calls": [
|
||||||
|
{
|
||||||
|
"id": "call_de6cdf1a_0",
|
||||||
|
"type": "function",
|
||||||
|
"function": {"name": "get_weather", "arguments": "{\"location\":\"Seattle\"}"}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"role": "tool",
|
||||||
|
"tool_call_id": "call_de6cdf1a_0",
|
||||||
|
"name": "get_weather",
|
||||||
|
"content": "{\"temperature\": \"16C\", \"conditions\": \"Partly cloudy with gentle breeze\"}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Python OpenAI client integration
|
||||||
|
|
||||||
|
```python
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(
|
||||||
|
base_url="http://localhost:8080/v1",
|
||||||
|
api_key="not-needed",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Streaming with reasoning
|
||||||
|
response = client.chat.completions.create(
|
||||||
|
model="kimi-k3",
|
||||||
|
messages=[
|
||||||
|
{"role": "user", "content": "Explain quantum superposition in 2 sentences."}
|
||||||
|
],
|
||||||
|
stream=True,
|
||||||
|
extra_body={"reasoning_effort": "max"},
|
||||||
|
)
|
||||||
|
|
||||||
|
for chunk in response:
|
||||||
|
delta = chunk.choices[0].delta
|
||||||
|
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||||
|
print(delta.reasoning_content, end="", flush=True)
|
||||||
|
if delta.content:
|
||||||
|
print(delta.content, end="", flush=True)
|
||||||
|
print()
|
||||||
|
```
|
||||||
|
|
||||||
|
## SOCKS5 proxy usage
|
||||||
|
|
||||||
|
Run the gateway through a SOCKS5 proxy (e.g. Tor or local tunnel):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Using CLI flag
|
||||||
|
./bin/k3r053n3 -socks socks5://127.0.0.1:9050
|
||||||
|
|
||||||
|
# With authentication
|
||||||
|
./bin/k3r053n3 -socks socks5://user:pass@127.0.0.1:1080
|
||||||
|
|
||||||
|
# Using environment variable
|
||||||
|
export ALL_PROXY=socks5://127.0.0.1:1080
|
||||||
|
./bin/k3r053n3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Run unit and integration tests:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Credits
|
||||||
|
|
||||||
|
Created by Luxferre in 2026, released into the public domain with no warranties.
|
||||||
@@ -0,0 +1,396 @@
|
|||||||
|
// k3r053n3_test: Test suite for Kimi K3 Gateway
|
||||||
|
// Created by Luxferre in 2026, released into the public domain
|
||||||
|
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestModelsEndpoint(t *testing.T) {
|
||||||
|
svc := NewKimiService("https://test.hf.space", "kimi-k3", "direct:together", "max", "", 8192, 0.7)
|
||||||
|
req, w := httptest.NewRequest("GET", "/v1/models", nil), httptest.NewRecorder()
|
||||||
|
NewMux(svc).ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("expected status 200, got %d", w.Code)
|
||||||
|
}
|
||||||
|
var res ModelsResponse
|
||||||
|
if err := json.NewDecoder(w.Body).Decode(&res); err != nil || len(res.Data) == 0 {
|
||||||
|
t.Fatalf("invalid models response: %v", err)
|
||||||
|
}
|
||||||
|
for _, m := range res.Data {
|
||||||
|
if m.ID == "kimi-k3" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.Fatalf("expected kimi-k3 in models list")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveReasoningEffort(t *testing.T) {
|
||||||
|
tests := []struct{ in, def, exp string }{
|
||||||
|
{"", "max", "max"}, {"max", "default", "max"}, {"HIGH", "low", "high"},
|
||||||
|
{"low", "max", "low"}, {"medium", "max", "default"}, {"default", "max", "default"}, {"unknown", "max", "max"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
if res := ResolveReasoningEffort(ChatCompletionRequest{ReasoningEffort: tt.in}, tt.def); res != tt.exp {
|
||||||
|
t.Errorf("ResolveReasoningEffort(%q, %q) = %q, want %q", tt.in, tt.def, res, tt.exp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveBackend(t *testing.T) {
|
||||||
|
svc := NewKimiService("https://test.hf.space", "kimi-k3", "direct:together", "max", "", 8192, 0.7)
|
||||||
|
tests := []struct{ model, exp string }{
|
||||||
|
{"kimi-k3", "direct:together"}, {"kimi-k3:fireworks", "direct:fireworks"},
|
||||||
|
{"kimi-k3:together", "direct:together"}, {"kimi-k3:hf-together", "hf:together"},
|
||||||
|
{"kimi-k3:hf-fireworks", "hf:fireworks-ai"}, {"kimi-k3:hf-featherless", "hf:featherless-ai"},
|
||||||
|
{"kimi-k3:hf-baseten", "hf:baseten"}, {"custom-model", "direct:together"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
if b := svc.ResolveBackend(tt.model); b != tt.exp {
|
||||||
|
t.Errorf("ResolveBackend(%q) = %q, want %q", tt.model, b, tt.exp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseSOCKS5URL(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
in string
|
||||||
|
exp *SOCKS5Config
|
||||||
|
}{
|
||||||
|
{"", nil},
|
||||||
|
{"127.0.0.1:1080", &SOCKS5Config{Address: "127.0.0.1:1080"}},
|
||||||
|
{"socks5://127.0.0.1:9050", &SOCKS5Config{Address: "127.0.0.1:9050"}},
|
||||||
|
{"socks5h://user:pass@proxy.example.com:1080", &SOCKS5Config{Address: "proxy.example.com:1080", Username: "user", Password: "pass"}},
|
||||||
|
{"socks5://myuser@10.0.0.1:1080", &SOCKS5Config{Address: "10.0.0.1:1080", Username: "myuser"}},
|
||||||
|
{"127.0.0.1", &SOCKS5Config{Address: "127.0.0.1:1080"}},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
cfg, err := ParseSOCKS5URL(tt.in)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("ParseSOCKS5URL(%q) err: %v", tt.in, err)
|
||||||
|
} else if tt.exp == nil && cfg != nil {
|
||||||
|
t.Errorf("ParseSOCKS5URL(%q) expected nil, got %+v", tt.in, cfg)
|
||||||
|
} else if tt.exp != nil && (cfg.Address != tt.exp.Address || cfg.Username != tt.exp.Username || cfg.Password != tt.exp.Password) {
|
||||||
|
t.Errorf("ParseSOCKS5URL(%q) = %+v, want %+v", tt.in, cfg, tt.exp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func startMockSOCKS5Server(t *testing.T, expectedUser, expectedPass string) (string, func()) {
|
||||||
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to start mock socks5 listener: %v", err)
|
||||||
|
}
|
||||||
|
stopCh := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
for {
|
||||||
|
conn, err := ln.Accept()
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
go func(c net.Conn) {
|
||||||
|
defer c.Close()
|
||||||
|
hdr := make([]byte, 2)
|
||||||
|
if _, err := io.ReadFull(c, hdr); err != nil || hdr[0] != 0x05 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
methods := make([]byte, int(hdr[1]))
|
||||||
|
if _, err := io.ReadFull(c, methods); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if expectedUser != "" {
|
||||||
|
c.Write([]byte{0x05, 0x02})
|
||||||
|
authHdr := make([]byte, 2)
|
||||||
|
if _, err := io.ReadFull(c, authHdr); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
userBuf := make([]byte, int(authHdr[1]))
|
||||||
|
io.ReadFull(c, userBuf)
|
||||||
|
pLenBuf := make([]byte, 1)
|
||||||
|
io.ReadFull(c, pLenBuf)
|
||||||
|
passBuf := make([]byte, int(pLenBuf[0]))
|
||||||
|
io.ReadFull(c, passBuf)
|
||||||
|
|
||||||
|
if string(userBuf) != expectedUser || string(passBuf) != expectedPass {
|
||||||
|
c.Write([]byte{0x01, 0x01})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
c.Write([]byte{0x01, 0x00})
|
||||||
|
} else {
|
||||||
|
c.Write([]byte{0x05, 0x00})
|
||||||
|
}
|
||||||
|
|
||||||
|
reqHdr := make([]byte, 4)
|
||||||
|
if _, err := io.ReadFull(c, reqHdr); err != nil || reqHdr[0] != 0x05 || reqHdr[1] != 0x01 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var targetAddr string
|
||||||
|
switch reqHdr[3] {
|
||||||
|
case 0x01:
|
||||||
|
ipBuf := make([]byte, 4)
|
||||||
|
io.ReadFull(c, ipBuf)
|
||||||
|
targetAddr = net.IP(ipBuf).String()
|
||||||
|
case 0x03:
|
||||||
|
lBuf := make([]byte, 1)
|
||||||
|
io.ReadFull(c, lBuf)
|
||||||
|
dBuf := make([]byte, int(lBuf[0]))
|
||||||
|
io.ReadFull(c, dBuf)
|
||||||
|
targetAddr = string(dBuf)
|
||||||
|
}
|
||||||
|
portBuf := make([]byte, 2)
|
||||||
|
io.ReadFull(c, portBuf)
|
||||||
|
port := (int(portBuf[0]) << 8) | int(portBuf[1])
|
||||||
|
|
||||||
|
remote, err := net.DialTimeout("tcp", fmt.Sprintf("%s:%d", targetAddr, port), 5*time.Second)
|
||||||
|
if err != nil {
|
||||||
|
c.Write([]byte{0x05, 0x05, 0x00, 0x01, 0, 0, 0, 0, 0, 0})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer remote.Close()
|
||||||
|
c.Write([]byte{0x05, 0x00, 0x00, 0x01, 127, 0, 0, 1, 0x1f, 0x90})
|
||||||
|
go io.Copy(remote, c)
|
||||||
|
io.Copy(c, remote)
|
||||||
|
}(conn)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
return ln.Addr().String(), func() { close(stopCh); ln.Close() }
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSOCKS5EndToEnd(t *testing.T) {
|
||||||
|
mockTarget := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.URL.Path == "/test" {
|
||||||
|
w.Write([]byte("socks5-ok"))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
http.NotFound(w, r)
|
||||||
|
}))
|
||||||
|
defer mockTarget.Close()
|
||||||
|
|
||||||
|
targetHostPort := strings.TrimPrefix(mockTarget.URL, "http://")
|
||||||
|
proxyAddr, cleanupProxy := startMockSOCKS5Server(t, "alice", "secret123")
|
||||||
|
defer cleanupProxy()
|
||||||
|
|
||||||
|
proxyURL := fmt.Sprintf("socks5://alice:secret123@%s", proxyAddr)
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
conn, err := DialSOCKS5(ctx, proxyURL, targetHostPort)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DialSOCKS5 failed: %v", err)
|
||||||
|
}
|
||||||
|
defer conn.Close()
|
||||||
|
|
||||||
|
fmt.Fprintf(conn, "GET /test HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n\r\n", targetHostPort)
|
||||||
|
respBody, err := io.ReadAll(conn)
|
||||||
|
if err != nil || !strings.Contains(string(respBody), "socks5-ok") {
|
||||||
|
t.Fatalf("socks5 direct dial failed: %v, got: %s", err, string(respBody))
|
||||||
|
}
|
||||||
|
|
||||||
|
svc := NewKimiService(mockTarget.URL, "kimi-k3", "direct:together", "max", proxyURL, 8192, 0.7)
|
||||||
|
req, _ := http.NewRequest("GET", mockTarget.URL+"/test", nil)
|
||||||
|
resp, err := svc.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("svc.client via SOCKS5 failed: %v", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
body, _ := io.ReadAll(resp.Body)
|
||||||
|
if string(body) != "socks5-ok" {
|
||||||
|
t.Fatalf("expected 'socks5-ok', got: %s", string(body))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseGradioSnapshot(t *testing.T) {
|
||||||
|
// Test 1: Standard reasoning done and answer present with internal </div> tag in reasoning
|
||||||
|
sample1 := `[[
|
||||||
|
{"role":"user","metadata":null,"content":[{"text":"Hi","type":"text"}],"options":null},
|
||||||
|
{"role":"assistant","metadata":null,"content":[{"text":"<","type":"text"}],"options":null},
|
||||||
|
{"role":"assistant","metadata":{"title":"Reasoning","status":"done"},"content":[{"text":">I am thinking about </div> and </tag> tags.</","type":"text"}],"options":null},
|
||||||
|
{"role":"assistant","metadata":null,"content":[{"text":">\n\nHello there!","type":"text"}],"options":null}
|
||||||
|
], {"value":{"text":"","files":[]},"__type__":"update"}]`
|
||||||
|
|
||||||
|
content, reasoning, ok := parseGradioSnapshot(sample1)
|
||||||
|
if !ok || content != "Hello there!" || reasoning != "I am thinking about </div> and </tag> tags." {
|
||||||
|
t.Fatalf("parseGradioSnapshot sample1 failed: content=%q, reasoning=%q", content, reasoning)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test 2: Reasoning still pending
|
||||||
|
sample2 := `[[
|
||||||
|
{"role":"user","metadata":null,"content":[{"text":"Hi","type":"text"}],"options":null},
|
||||||
|
{"role":"assistant","metadata":null,"content":[{"text":"<","type":"text"}],"options":null},
|
||||||
|
{"role":"assistant","metadata":{"title":"Reasoning","status":"pending"},"content":[{"text":">Still reasoning","type":"text"}],"options":null}
|
||||||
|
], {"value":{"text":"","files":[]},"__type__":"update"}]`
|
||||||
|
|
||||||
|
content2, reasoning2, ok2 := parseGradioSnapshot(sample2)
|
||||||
|
if !ok2 || content2 != "" || reasoning2 != "Still reasoning" {
|
||||||
|
t.Fatalf("parseGradioSnapshot sample2 failed: content=%q, reasoning=%q", content2, reasoning2)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test 3: Interleaved messages
|
||||||
|
sample3 := `[[
|
||||||
|
{"role":"user","metadata":null,"content":[{"text":"Hi","type":"text"}]},
|
||||||
|
{"role":"assistant","metadata":null,"content":[{"text":"<"}]},
|
||||||
|
{"role":"assistant","metadata":{"title":"Reasoning","status":"done"},"content":[{"text":">Part 1"}]},
|
||||||
|
{"role":"assistant","metadata":null,"content":[{"text":"intro text: "}]},
|
||||||
|
{"role":"assistant","metadata":{"title":"Reasoning","status":"done"},"content":[{"text":". Part 2</"}]},
|
||||||
|
{"role":"assistant","metadata":null,"content":[{"text":">\n\n<tool_call>{\"name\":\"fn\"}</tool_call>"}]}
|
||||||
|
]]`
|
||||||
|
content3, reasoning3, ok3 := parseGradioSnapshot(sample3)
|
||||||
|
if !ok3 || !strings.Contains(content3, "<tool_call>") || !strings.Contains(reasoning3, "Part 1. Part 2") {
|
||||||
|
t.Fatalf("parseGradioSnapshot sample3 failed: content=%q, reasoning=%q", content3, reasoning3)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDetectToolCalls(t *testing.T) {
|
||||||
|
// 1. Standard XML tag with conversational text
|
||||||
|
text1 := "Here is the weather:\n<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"location\": \"Tokyo\"}}\n</tool_call>\nDone."
|
||||||
|
tcs1, cleaned1, ok1 := DetectToolCalls(text1)
|
||||||
|
if !ok1 || len(tcs1) != 1 || tcs1[0].Function.Name != "get_weather" || !strings.Contains(tcs1[0].Function.Arguments, "Tokyo") {
|
||||||
|
t.Fatalf("DetectToolCalls 1 failed: ok=%v, tcs=%+v", ok1, tcs1)
|
||||||
|
}
|
||||||
|
if strings.Contains(cleaned1, "<tool_call>") || !strings.Contains(cleaned1, "Here is the weather:") {
|
||||||
|
t.Fatalf("DetectToolCalls cleaned 1 invalid: %q", cleaned1)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. tool-call variant
|
||||||
|
text2 := "<tool-call>\n{\"name\": \"calc\", \"arguments\": \"{\\\"x\\\": 1}\"}\n</tool-call>"
|
||||||
|
tcs2, _, ok2 := DetectToolCalls(text2)
|
||||||
|
if !ok2 || len(tcs2) != 1 || tcs2[0].Function.Name != "calc" {
|
||||||
|
t.Fatalf("DetectToolCalls 2 failed: ok=%v, tcs=%+v", ok2, tcs2)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. markdown codeblock format
|
||||||
|
text3 := "```tool_call\n{\"name\": \"search\", \"query\": \"golang\"}\n```"
|
||||||
|
tcs3, _, ok3 := DetectToolCalls(text3)
|
||||||
|
if !ok3 || len(tcs3) != 1 || tcs3[0].Function.Name != "search" {
|
||||||
|
t.Fatalf("DetectToolCalls 3 failed: ok=%v, tcs=%+v", ok3, tcs3)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. unclosed tag at end of response
|
||||||
|
text4 := "I will query:\n<tool_call>\n{\"name\": \"fetch_data\", \"id\": 123}"
|
||||||
|
tcs4, cleaned4, ok4 := DetectToolCalls(text4)
|
||||||
|
if !ok4 || len(tcs4) != 1 || tcs4[0].Function.Name != "fetch_data" || cleaned4 != "I will query:" {
|
||||||
|
t.Fatalf("DetectToolCalls 4 unclosed failed: ok=%v, tcs=%+v, cleaned=%q", ok4, tcs4, cleaned4)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 5. function call syntax: get_weather(location="Tokyo")
|
||||||
|
text5 := "<tool_call>get_weather(location=\"Tokyo\")</tool_call>"
|
||||||
|
tcs5, _, ok5 := DetectToolCalls(text5)
|
||||||
|
if !ok5 || len(tcs5) != 1 || tcs5[0].Function.Name != "get_weather" || !strings.Contains(tcs5[0].Function.Arguments, "Tokyo") {
|
||||||
|
t.Fatalf("DetectToolCalls 5 function syntax failed: ok=%v, tcs=%+v", ok5, tcs5)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 6. markdown ```json with tool payload
|
||||||
|
text6 := "```json\n{\"name\": \"lookup\", \"arguments\": {\"id\": 42}}\n```"
|
||||||
|
tcs6, _, ok6 := DetectToolCalls(text6)
|
||||||
|
if !ok6 || len(tcs6) != 1 || tcs6[0].Function.Name != "lookup" {
|
||||||
|
t.Fatalf("DetectToolCalls 6 json block failed: ok=%v, tcs=%+v", ok6, tcs6)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStreamToolInterceptor(t *testing.T) {
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
streamer := NewStreamer(w, nil, "test-id", 123456789, "kimi-k3")
|
||||||
|
interceptor := NewStreamToolInterceptor(streamer)
|
||||||
|
|
||||||
|
interceptor.ProcessContentDelta("Checking weather for you...\n<tool_")
|
||||||
|
interceptor.ProcessContentDelta("call>\n{\"name\": \"get_weather\", \"arguments\": {\"city\": \"SF\"}}\n</tool_call>\nHave a nice day!")
|
||||||
|
interceptor.FlushRemaining()
|
||||||
|
|
||||||
|
if !interceptor.HasToolCalls() {
|
||||||
|
t.Fatalf("expected tool call registered in interceptor")
|
||||||
|
}
|
||||||
|
body := w.Body.String()
|
||||||
|
if !strings.Contains(body, "Checking weather for you...") || !strings.Contains(body, "get_weather") || strings.Contains(body, "<tool_call>") {
|
||||||
|
t.Fatalf("unexpected stream body: %s", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEndToEndMockChatCompletion(t *testing.T) {
|
||||||
|
mockGradio := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.URL.Path == "/gradio_api/call/on_submit" && r.Method == "POST" {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
json.NewEncoder(w).Encode(GradioJoinResponse{EventID: "evt_12345"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if r.URL.Path == "/gradio_api/call/on_submit/evt_12345" && r.Method == "GET" {
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
flusher, _ := w.(http.Flusher)
|
||||||
|
fmt.Fprintf(w, "event: generating\ndata: [[{\"role\":\"user\",\"metadata\":null,\"content\":[{\"text\":\"Hi\"}],\"options\":null},{\"role\":\"assistant\",\"metadata\":null,\"content\":[{\"text\":\"<\"}],\"options\":null},{\"role\":\"assistant\",\"metadata\":{\"title\":\"Reasoning\",\"status\":\"done\"},\"content\":[{\"text\":\">Thinking\"}],\"options\":null}]]\n\n")
|
||||||
|
if flusher != nil {
|
||||||
|
flusher.Flush()
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "event: complete\ndata: [[{\"role\":\"user\",\"metadata\":null,\"content\":[{\"text\":\"Hi\"}],\"options\":null},{\"role\":\"assistant\",\"metadata\":null,\"content\":[{\"text\":\"<\"}],\"options\":null},{\"role\":\"assistant\",\"metadata\":{\"title\":\"Reasoning\",\"status\":\"done\"},\"content\":[{\"text\":\">Thinking</\"}],\"options\":null},{\"role\":\"assistant\",\"metadata\":null,\"content\":[{\"text\":\">\\n\\nHello!\"}],\"options\":null}]]\n\n")
|
||||||
|
if flusher != nil {
|
||||||
|
flusher.Flush()
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
http.NotFound(w, r)
|
||||||
|
}))
|
||||||
|
defer mockGradio.Close()
|
||||||
|
|
||||||
|
svc := NewKimiService(mockGradio.URL, "kimi-k3", "direct:together", "max", "", 8192, 0.7)
|
||||||
|
mux := NewMux(svc)
|
||||||
|
|
||||||
|
// 1. Non-Streaming
|
||||||
|
reqB, _ := json.Marshal(ChatCompletionRequest{Model: "kimi-k3", Messages: []ChatMessage{{Role: "user", Content: "Hi"}}, Stream: false})
|
||||||
|
req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBuffer(reqB))
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
mux.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
var resp ChatCompletionResponse
|
||||||
|
if err := json.NewDecoder(w.Body).Decode(&resp); err != nil || len(resp.Choices) == 0 {
|
||||||
|
t.Fatalf("non-streaming chat failed: %v", err)
|
||||||
|
}
|
||||||
|
if resp.Choices[0].Message.ReasoningContent != "Thinking" || resp.Choices[0].Message.GetContentString() != "Hello!" {
|
||||||
|
t.Errorf("unexpected choice response: %+v", resp.Choices[0])
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Streaming
|
||||||
|
reqBStream, _ := json.Marshal(ChatCompletionRequest{Model: "kimi-k3", Messages: []ChatMessage{{Role: "user", Content: "Hi"}}, Stream: true})
|
||||||
|
reqStream := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBuffer(reqBStream))
|
||||||
|
reqStream.Header.Set("Content-Type", "application/json")
|
||||||
|
wStream := httptest.NewRecorder()
|
||||||
|
mux.ServeHTTP(wStream, reqStream)
|
||||||
|
|
||||||
|
scanner := bufio.NewScanner(wStream.Body)
|
||||||
|
hasReasoning, hasContent, hasDone := false, false, false
|
||||||
|
for scanner.Scan() {
|
||||||
|
line := scanner.Text()
|
||||||
|
if strings.HasPrefix(line, "data: ") {
|
||||||
|
data := strings.TrimPrefix(line, "data: ")
|
||||||
|
if data == "[DONE]" {
|
||||||
|
hasDone = true
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
var chunk StreamResponse
|
||||||
|
if err := json.Unmarshal([]byte(data), &chunk); err == nil && len(chunk.Choices) > 0 {
|
||||||
|
if chunk.Choices[0].Delta.ReasoningContent != "" {
|
||||||
|
hasReasoning = true
|
||||||
|
}
|
||||||
|
if chunk.Choices[0].Delta.Content != "" {
|
||||||
|
hasContent = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !hasReasoning || !hasContent || !hasDone {
|
||||||
|
t.Errorf("streaming test failed: reasoning=%v, content=%v, done=%v", hasReasoning, hasContent, hasDone)
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user