A Rule You Can Recite Is Not a Rule That Runs

My outbound system reported "17 letters out, no bounces" this morning. One of them had bounced seven seconds after it left.

It was a batch of freedom of information requests to local councils. After every letter the system waits 150 seconds, asks the mailbox whether anything bounced, and stops the whole batch if something did. That rule sits in the first lines of the script. I could recite it.

For at least twelve days it had not been able to read.

Two guards, each correct on its own

The bounce check asked the mailbox for up to 500 messages. Two weeks earlier I had built a second safety guard, one that refuses any mailbox read bigger than the provider's per-minute quota, so the system can never lock itself out of sending. 500 messages cost about 10,000 quota units. The limit is 6,000 per minute. So the new guard refused the bounce check. Every single time.

The refusal went to the error stream. The bounce check read only the normal output, found it empty, and treated empty as "no bounce". Two guards, each correct on its own, and one had quietly blinded the other.

What it cost

On 8 September it had already missed a bounce. That one was caught the next day by a person reading the inbox, not by the check. Today the bounce landed at 10:28, and the batch sent the next letter 150 seconds later as if nothing had happened. I stopped it by hand.

Then the part I like least. With the check fixed and the batch finished, the summary still said "no bounces". It counted only the run it was in, and the bounce belonged to the run I had stopped.

What changed

• ✓The check now has three answers, not two: bounced, clear, could not read. The third one stops the batch.
• ✓It asks for 50 messages, which fits the quota.
• ✓It is tested on the real cases, with the real function, not a copy of it: the bounced address says bounced, a delivered one says clear, a broken read says stop. The video above is that test, replayed.
• ✓The summary counts bounces from the batch's own record, not from the current run.

The question to ask of every check

A newsletter I read today put it better than I can: a standard with no enforcement becomes an opinion you hold about yourself. Mine had enforcement. It just could not see.

Most checks are written for two outcomes, pass and fail. The dangerous case is the third one, when the check could not look at all, because a failed read and a clean result usually print the same thing: nothing. So the question is not "does this check pass?" but "what does this check say when it cannot read?" If the answer is "the same as when everything is fine", it is not a check yet.

When we build a pipeline for a client, that is one of the questions we put to every step that reads from somewhere else: a mailbox, an API, a scanned document, a queue. A read that fails has to stop the line, visibly, instead of passing as a clean result.

Which of your safety checks would tell you if it had stopped being able to read?