ended7월 23일· 1 sources
GigaToken: ~1000x faster Language model tokenization
Why it matters
~1000x faster than HuggingFace's tokenizers, drop-in replacement. Tokenize your text data at GB/s! Note that both HF tokenizers and tiktoken are already running multithreaded Rust! Gigatoken is the fa...
1
Sources
+0
24h
—
Growth
18d
Active