luxferre/hygate-c
1
0
Fork 0
Code Actions Activity

Switch embedded tokenizer to Tencent Hy3 120k ByteLevel BPE

Browse Source
This commit is contained in:
Luxferre
2026-10-14 14:15:28 +03:00
parent 80ad504d16
commit 561d90629f
5 changed files with 17110 additions and 19636 deletions
Download Patch File Download Diff File Expand all files Collapse all files
jsonapi.c
+1 -1
View File
@@ -40,7 +40,7 @@ static int count_content_tokens(json_object *content) {
}
/* ------------------------------------------------------------------ */
/* Token counting (embedded cl100k_base BPE) */
/* Token counting (embedded Tencent Hy3 BPE) */
/* ------------------------------------------------------------------ */
int count_messages_tokens(void *req_obj) {
json_object *req = (json_object *)req_obj;