Essay · From the shop

Twenty Perfect Deliveries

A webhook that returned 200 twenty times, delivered nothing, and produced the healthiest log I have ever seen

By Kitonga Kiiva · 1,093 words

For most of August, the delivery log for my bookshop read twenty for twenty. Every webhook accepted. Every one on the first attempt. No retries, no errors, no warnings. It is the healthiest-looking log I have ever produced.

Not one of those deliveries delivered anything.

I want to describe this properly, because the failure is not interesting as a bug. It is interesting as a shape — and it is a shape I keep finding in systems that are supposed to be watching themselves.

The setup

I sell five books directly from my own site. No storefront platform, no marketplace. A payment processor handles the money and, when a purchase completes, sends a signed notification to a small PHP file on my server. That file checks the signature, works out which book was bought, attaches the files, and emails them to the buyer.

It is about four hundred lines. I wrote all of it. There is no framework involved and nothing about it is clever.

What I was actually doing

I was not debugging. I had no reason to think anything was wrong.

I was trying to test the delivery path without spending money, because an earlier attempt to test it with a real card had ended with a mistyped one-time code and a temporarily locked card. So I set up a simulation: the processor would send my server a genuine, correctly signed purchase event, and I would watch what happened.

The first thing I checked was the log of everything my server had ever received. Twenty notifications. All delivered, all first attempt, all 200 OK. Account created, address created, transaction created, transaction ready, transaction updated, payment failed, payment failed again. A complete, healthy record of a shop functioning normally.

Then I noticed what was not in it.

There was no transaction.completed. Not once. That is the only event that triggers a delivery — and it had never fired, because nobody had ever completed a purchase. Every other event type had exercised the endpoint hundreds of times. The one path that mattered had never run.

So the endpoint had been proving its health, continuously, on the paths that did not matter.

The first fault

When I finally made the real event fire, the server answered 200 OK and sent nothing.

The reason is three lines of code:

$customerEmail = $tx['customer']['email'] ?? $tx['billing_details']['email'] ?? null;
if (!$customerEmail) {
    error_log('No customer email'); http_response_code(200); exit('No email');
}

The payload has no customer object. It never has. It carries a customer_id — a string — and nothing else that identifies a human being. billing_details is null on anything bought through a checkout. I checked a real payload the processor had sent my server weeks earlier: three thousand seven hundred and forty-six characters, and not a single @ anywhere in it.

So the lookup failed, every time, for every purchase. The file logged a line, returned success, and stopped.

The second fault

Underneath that was a second, entirely independent failure, which I only found because the first one had been hiding it.

My catalogue maps a price identifier to a set of files. In August I replaced the price of the five-book set. New identifier, new price. I updated the shop. I did not update the catalogue, which still pointed at the retired identifier.

So even with the email problem solved, the flagship product would have matched nothing, logged "unknown price", and sent nothing. Two independent faults, either one sufficient, both invisible.

Why the 200 is the villain

Every one of these failures ended the same way: http_response_code(200).

That is not a mistake in the ordinary sense. Returning 200 to a webhook is correct behaviour — it tells the sender "I have received this, do not retry." A webhook that returns an error gets retried, and a webhook that returns an error for something unretryable gets retried forever. The advice to return 200 is good advice.

But look at what the 200 actually asserts. It says: I received your message. It does not say I did the job. Those are different claims, and my system had quietly collapsed them into one.

Everything downstream then inherited the confusion. The dashboard counts delivered notifications, so it counted twenty. The retry logic sees success, so it never retried. And I read the dashboard and concluded the integration was fine, because that is what the record said.

The record was not lying. It was answering a different question from the one I thought I was asking.

The shape

This is the part I keep meeting.

A system records the thing that is easy to record. Receipt is easy: it happens at a single point, it is unambiguous, it needs no knowledge of what the work was for. Delivery is hard: it spans a mail server, a file system, an address you have to look up, and eleven attachments that have to survive the journey.

So the register fills up with receipts, everyone downstream treats the receipt as evidence of delivery, and the gap between the two becomes the place where failure lives — undetected, sometimes for a very long time, and with a clean audit trail the whole while.

The dangerous version is not the system that fails loudly. It is the one whose success signal measures something adjacent to what you care about. You cannot catch that by watching harder. Watching harder just gives you more of the wrong number.

I got lucky. Nobody had bought the set yet, so nobody paid me forty-four dollars and received silence. But the only reason I found it is that I went looking for something else entirely, on a path nothing had ever exercised. If a customer had arrived first, they would have paid, received nothing, emailed me — and I would have opened a dashboard showing twenty perfect deliveries and told them, in good faith, that everything was working.

What I changed

The email now comes from an authenticated lookup against the processor's API using the customer identifier, which is the only thing the payload actually contains. The catalogue is keyed on the live price with the retired one aliased to it, so an old link still works and a future tidy-up cannot silently break the current product.

And the test is now cardless and repeatable. I can fire a genuine signed purchase event at the live server whenever I want, with my own customer record attached, and watch eleven files arrive in my inbox. Not a receipt for eleven files. The files.

That last distinction is the whole of it.