Hashing 100,000 JSON Lines Found No Duplicates, Hashing Them With Canon Found 9,900

Thu, 22 Oct 2026 00:00:00 +0000

Event pipelines get the same event twice. A producer times out, retries, and the retry goes through a different service that writes JSON its own way. The usual dedupe is a hash of each line, and it quietly misses all of these.

We generated a stream of 100,000 JSON lines from a fixed seed, 13.1 MB in all. 90,000 are distinct purchase events. The other 10,000 are resends of earlier events, written by two other producers. One sorts the keys and escapes non-ASCII, so “Zürich” arrives as Z\u00fcrich. The other reverses the key order and adds spaces. For a twist, 100 of the resends come from a producer that writes the quantity as a float: 2.0 where the original said 2.

Data on BXM.net

Hashing 100,000 JSON Lines Found No Duplicates, Hashing Them With Canon Found 9,900