Tokenizer implemented

README.md

@@ -66,8 +66,30 @@ CORS is enabled for all origins (`Access-Control-Allow-Origin: *`).
 - Upstream calls use Fibonacci-backoff retries (up to `MAX_RETRIES = 5`,
   300 s timeout).
 
+## Token usage counting
+
+Responses include an OpenAI-style `usage` object with `prompt_tokens`,
+`completion_tokens`, and `total_tokens`. These are computed locally by an
+**embedded cl100k_base BPE tokenizer** (vocabulary data in
+`tokenizer_data.c`, logic in `tokenizer.c`) — no upstream token counts are
+available from the Gradio backend.
+
+- `prompt_tokens` sums each message's `content` (or multi-part content
+  parts), `reasoning_content`, and any `tool_calls` function `name` /
+  `arguments`.
+- `completion_tokens` sums the assistant `content`, `reasoning_content`, and
+  any emitted `tool_calls`.
+- `total_tokens = prompt_tokens + completion_tokens`.
+
+cl100k_base is OpenAI's canonical multilingual tokenizer and is used as the
+best available approximation for Hunyuan; the resulting counts are
+deterministic subword counts and may differ slightly from the exact Hunyuan
+vocabulary. No per-message formatting overhead (the `+3`/priming tokens
+OpenAI applies) is added.
+
 ## Security considerations
 
+
 - The server binds all interfaces (`0.0.0.0`) and currently enforces **no
   authentication** — it is an open relay to the configured upstream. Do not
   expose it to untrusted networks without a fronting proxy or auth layer.