> ## Documentation Index
> Fetch the complete documentation index at: https://whatsapp-rust.jlucaso.com/llms.txt
> Use this file to discover all available pages before exploring further.

# June 15, 2026 — Binary encode: length-bucketed token lookup

> index_of_token switches from a PTHash perfect-hash map to hashify::tiny_map!, cutting marshal_auto_small latency by 34% on the miss-heavy uncached encode path.

## Performance

**Binary encode: `index_of_token` uses a length-bucketed `tiny_map` instead of PTHash ([#873](https://github.com/oxidezap/whatsapp-rust/pull/873))**

`index_of_token` runs on the encode hot path: `classify_string_hint` probes every string in every outgoing stanza. The production send path (`marshal_auto` → `Encoder::new_vec`) has no `StringHintCache`, so the lookup runs uncached per string.

**Previously**, the lookup used `hashify::map!` — a compile-time PTHash map with FNV-1a key hashing. On the miss-heavy encode path (JIDs, message IDs, timestamps, and content strings all fall within the token length range but are not tokens), this folded every byte of the key before probing, regardless of outcome.

**Now**, the codegen switches to `hashify::tiny_map!`, which dispatches on `key.len()` first and compares only discriminator bytes within each length bucket — no full-key hash on any code path. This is the same technique used in [Bun PR #31875](https://github.com/oven-sh/bun/pull/31875). Multi-byte leaves still get a full key comparison at build time, so the map cannot silently alias a non-token onto a token — build-time resolution fails loud if any bucket is unresolvable.

**Benchmark results (divan A/B, median, main vs. branch):**

| Benchmark                                                  | Before   | After    | Change   |
| ---------------------------------------------------------- | -------- | -------- | -------- |
| `marshal_auto_small` (dominant shape: message/receipt/ack) | 168.7 ns | 111.8 ns | **−34%** |
| `marshal_plain_large`                                      | 1.241 µs | 911 ns   | −27%     |
| `marshal_auto_large`                                       | 1.116 µs | 905 ns   | −19%     |
| `marshal_to_reused_buffer_large`                           | 1.104 µs | 916 ns   | −17%     |
| `marshal_auto_many_children`                               | 184.5 µs | 165.4 µs | −10%     |

These benchmarks are tracked by CodSpeed CI, so the gain is regression-guarded going forward.

**Correctness.** This is a pure codegen change — same `tokens.json` source, same byte-array keys, no new dependency. A new exhaustive test (`lookup_matches_reference_under_byte_mutation`) builds a reference `HashMap` from the full token tables and verifies that the `tiny_map` lookup agrees for every one-byte input and every single-byte mutation (all 256 values) of every token string.

**No breaking changes.** `index_of_token`'s signature and return type are unchanged.