Switch embedded tokenizer to Tencent Hy3 120k ByteLevel BPE

Browse Source

...

This commit is contained in:

Luxferre

2026-10-14 14:15:28 +03:00

parent 80ad504d16

commit 561d90629f

5 changed files with 17110 additions and 19636 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

tokenizer_data.c

+16545 -19454

File diff suppressed because it is too large Load Diff