ended7월 23일· 1 sources

GigaToken: ~1000x faster Language model tokenization

Why it matters

~1000x faster than HuggingFace's tokenizers, drop-in replacement. Tokenize your text data at GB/s! Note that both HF tokenizers and tiktoken are already running multithreaded Rust! Gigatoken is the fa...

1
Sources
+0
24h
Growth
18d
Active

Sources

Related Issues