luxferre/hygate-c
1
0
Fork 0
Code Actions Activity

tok/mkf upd

Browse Source
This commit is contained in:
Luxferre
2026-10-15 23:13:11 +03:00
parent 561d90629f
commit 18ebe25d34
4 changed files with 92 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+7 -5
View File
@@ -77,12 +77,14 @@ pre-tokenizer and BPE logic in `tokenizer.c`) — no upstream token counts are
available from the Gradio backend.
- `prompt_tokens` sums the active `system` prompt, each message's `content`
(or individual multi-part content parts), `reasoning_content`, `tool_calls`
/ `function_call` (`name` and `arguments`), `tool` result wrappers, and any
request `tools` function definitions (`name`, `description`, `parameters`).
(or individual multi-part content parts), `reasoning_content` when
`preserved_thinking` is active (enabled by default when `tools` are
present, matching `tencent/Hy3`'s `chat_template.jinja`), `tool_calls` /
`function_call` (`name` and unpacked `arguments` key/value pairs), the
final `tool` result wrapper, and any request `tools` function definitions.
- `completion_tokens` sums the assistant `content`, `reasoning_content`, and
any emitted `tool_calls` (`name` and `arguments`), with `reasoning_tokens`
broken out under `completion_tokens_details`.
any emitted `tool_calls` (`name` and unpacked `arguments` key/value pairs),
with `reasoning_tokens` broken out under `completion_tokens_details`.
- `total_tokens = prompt_tokens + completion_tokens`.
The embedded tokenizer implements `tencent/Hy3`'s exact 4-stage `Sequence`