Performance¶
What runs where in the 0.1 series¶
Work |
Execution path |
|---|---|
Canonical SHA-256 in binary wheels |
Rust |
|
Rust, parallel, without holding the GIL during native work |
Hugging Face tokenizer-JSON batch counting |
Rust |
Frame planning and transformations |
Python immutable planner |
Python tokenizer plugin |
Batched Python callback; may hold the GIL |
DataFusion |
Rust session is exposed, but it does not yet execute Python frame plans |
The architecture is intentionally prepared for more native execution, but documentation and benchmarks must describe the current path rather than a future one.
Token cache¶
Token counts are cached in process by token-profile fingerprint and content hash. The cache is an in-memory, process-local LRU with a 100,000-entry limit in the 0.1 series. It is not shared between workers and is not persisted.
Changing tokenizer vocabulary, envelope rules, or plugin configuration must change the fingerprint to avoid stale counts.
Benchmark policy¶
The repository benchmarks hashing and native tokenization with Criterion. CI compares base and candidate revisions on the same runner and rejects a regression greater than ten percent for the configured benchmark set.
That gate is a regression signal, not a universal throughput claim. Production performance depends on payload size, selected tokenizer, cache hit rate, Python callbacks, and machine architecture. No headline throughput number is published for the 0.1 series because the repository does not yet maintain a versioned benchmark report with hardware and corpus details.
Use native profiles for high-throughput paths, batch plugin work, avoid per-item network policies, and measure the complete application pipeline rather than tokenization alone.