Replaying 13 Commits Through Memo Cut Test Time From 228 to 79 Seconds and Turned Up Two Flaky Tests

BXM’s repository has two packages with their own test suites: Canon, tested with cargo test, and Memo, tested with go test. Most commits touch one of them, or neither. A docs change or a new JavaScript build script doesn’t change either suite’s result, yet a plain CI job runs both every time.

So we replayed the repository’s own history. All 13 commits from 0.1.0 to the end of 0.3.0 were checked out one after another, and both suites ran at each one. That happened twice: once plain, and once with each suite behind Memo. Canon’s suite takes canon/** and the spec’s test vectors as inputs. Memo’s takes memo/**. Memo needs a declared output, so each suite writes its log to a file and that file is the output.

We ran it in two settings on a 2-CPU Linux machine. In the first, the Rust target folder and the Go build cache were wiped before every commit, like a CI runner that keeps nothing but Memo’s small cache folder. In the second, they were kept, like a laptop or a runner that caches them. Go’s own test cache was off throughout (-count=1), so the numbers show what running the suites costs.

Through Memo, both suites were skipped at 7 of the 13 commits. Canon’s suite ran at 6, the ones that changed the Rust code, its Cargo.toml or the vectors. Memo’s ran at 4. In the fresh setting that’s about a third of the time. In the warm setting the build caches already save most of the compiling, so Memo’s share is smaller, from 51 to 33 seconds. Either way, a skipped suite costs nothing, since Memo restores its log and replays its output.

The flaky tests

The plain runs failed three times. Twice it was TestManyProcessesShareOneCache, which said the result it restored was empty. Once it was TestConcurrentRunsOfTheSameJob. CI had been green for all three releases.

Both tests run many jobs at once, 16 in one and 8 in the other, and in both, every job wrote the same output file in the same folder. A shell > empties a file before writing it. Now and then one job stored the file just as another had emptied it. That’s a race between the commands themselves, and Memo made it worse. Without Memo, the folder is right once every job has finished. With Memo, the empty copy went into the cache, and every later hit restored it. Memo stores the output as it finds it, and it can’t tell a file another process is rewriting from a finished one.

The fixes are in the repository. The 16-process test now gives every process its own checkout, as 16 CI runners would have. The 8-job test writes its output to a temporary name and renames it into place. Each ran 40 times in a row without failing. The README and the Memo page now say it plainly: give each concurrent run its own working directory, as separate CI runners and checkouts already have.

There’s a lesson for anyone putting tests behind Memo, too. A flaky test that passes gets cached, and then it doesn’t run again until its inputs change. Memo doesn’t make a test flaky, but it can keep you from seeing it. Run the full suite without Memo every so often, for example nightly. That’s how this replay caught these two.

Run it yourself

The script is examples/case-studies/05-tests-across-commits/run.sh in the repository, and the recorded output, with every commit and every failure, is test/results/case-studies/05-tests-across-commits.txt. The four passes took about six and a half minutes on two CPUs, plus the clone and the dependency download.

More examples are on the use cases page.