Performance
Binary encode:index_of_token uses a length-bucketed tiny_map instead of PTHash (#873)
index_of_token runs on the encode hot path: classify_string_hint probes every string in every outgoing stanza. The production send path (marshal_auto → Encoder::new_vec) has no StringHintCache, so the lookup runs uncached per string.
Previously, the lookup used hashify::map! — a compile-time PTHash map with FNV-1a key hashing. On the miss-heavy encode path (JIDs, message IDs, timestamps, and content strings all fall within the token length range but are not tokens), this folded every byte of the key before probing, regardless of outcome.
Now, the codegen switches to hashify::tiny_map!, which dispatches on key.len() first and compares only discriminator bytes within each length bucket — no full-key hash on any code path. This is the same technique used in Bun PR #31875. Multi-byte leaves still get a full key comparison at build time, so the map cannot silently alias a non-token onto a token — build-time resolution fails loud if any bucket is unresolvable.
Benchmark results (divan A/B, median, main vs. branch):
| Benchmark | Before | After | Change |
|---|---|---|---|
marshal_auto_small (dominant shape: message/receipt/ack) | 168.7 ns | 111.8 ns | −34% |
marshal_plain_large | 1.241 µs | 911 ns | −27% |
marshal_auto_large | 1.116 µs | 905 ns | −19% |
marshal_to_reused_buffer_large | 1.104 µs | 916 ns | −17% |
marshal_auto_many_children | 184.5 µs | 165.4 µs | −10% |
tokens.json source, same byte-array keys, no new dependency. A new exhaustive test (lookup_matches_reference_under_byte_mutation) builds a reference HashMap from the full token tables and verifies that the tiny_map lookup agrees for every one-byte input and every single-byte mutation (all 256 values) of every token string.
No breaking changes. index_of_token’s signature and return type are unchanged.