2 Commits
Author SHA1 Message Date
Luxferre 561d90629f Switch embedded tokenizer to Tencent Hy3 120k ByteLevel BPE 2026-10-14 14:15:28 +03:00
Luxferre 8a0c1d0ec0 Tokenizer implemented 2026-10-13 20:37:36 +03:00