Switch embedded tokenizer to Tencent Hy3 120k ByteLevel BPE

README.md

@@ -70,9 +70,11 @@ CORS is enabled for all origins (`Access-Control-Allow-Origin: *`).
 
 Responses include an OpenAI-style `usage` object with `prompt_tokens`,
 `completion_tokens`, `total_tokens`, and `completion_tokens_details`
-(`reasoning_tokens`). These are computed locally by an **embedded cl100k_base
-BPE tokenizer** (vocabulary data in `tokenizer_data.c`, logic in `tokenizer.c`)
-— no upstream token counts are available from the Gradio backend.
+(`reasoning_tokens`). These are computed locally by an **embedded Tencent Hy3
+ByteLevel BPE tokenizer** (`tencent/Hy3`, 120,000 BPE tokens + 818 added
+special/control tokens; vocabulary data in `tokenizer_data.c`, 4-stage
+pre-tokenizer and BPE logic in `tokenizer.c`) — no upstream token counts are
+available from the Gradio backend.
 
 - `prompt_tokens` sums the active `system` prompt, each message's `content`
   (or individual multi-part content parts), `reasoning_content`, `tool_calls`
@@ -83,11 +85,12 @@ BPE tokenizer** (vocabulary data in `tokenizer_data.c`, logic in `tokenizer.c`)
   broken out under `completion_tokens_details`.
 - `total_tokens = prompt_tokens + completion_tokens`.
 
-cl100k_base is OpenAI's canonical multilingual tokenizer and is used as the
-best available approximation for Hunyuan; the resulting counts are
-deterministic subword counts and may differ slightly from the exact Hunyuan
-vocabulary. No per-message formatting overhead (the `+3`/priming tokens
-OpenAI applies) is added.
+The embedded tokenizer implements `tencent/Hy3`'s exact 4-stage `Sequence`
+pre-tokenizer (`\p{N}{1,3}` digit splitting, CJK/Hiragana/Katakana isolation,
+word/punctuation/whitespace regex splitting, and `ByteLevel` byte mapping),
+added special-token matching (`<｜hy_...｜>`, `<think:opensource>`, tool tags,
+etc.), and rank-ordered BPE merges. No per-message chat-template wrapper
+overhead is added.
 
 ## Security considerations
 
@@ -104,7 +107,9 @@ OpenAI applies) is added.
 |------|------|
 | `main.c` | Entry point, flag parsing, TCP listen, thread-per-connection with concurrency cap. |
 | `server.c` | HTTP parsing, routing, SSE emitters, libcurl upstream calls, streaming state machine. |
-| `jsonapi.c` | OpenAI <-> Gradio payload translation and stream parsing. |
+| `jsonapi.c` | OpenAI <-> Gradio payload translation, stream parsing, and message token counting. |
+| `tokenizer.c` | Embedded Tencent Hy3 4-stage pre-tokenizer, added-token matcher, and BPE engine. |
+| `tokenizer_data.c` | Generated Tencent Hy3 120k BPE vocabulary tables. |
 | `util.c` | Growable string buffer (`strbuf`), UUID, backoff, helpers. |
 | `hygate.h` | Shared types and declarations. |