Working on live news infrastructure taught me that failures are not exceptional — they are scheduled. When a Fed rate decision drops at 2pm, every monitoring system, every analytics pipeline, every content update fires simultaneously. Network partitions happen. Acknowledgements get dropped. The question is never "will this request fail?" — it's "what happens when it does?"

Idempotency keys are the answer. The design rule I now apply to every API: the client should always be able to safely retry. If handing you the same request twice produces different side effects, the design isn't finished. This matters even more in event-driven systems where the cost of double-processing — a duplicate trade entry, a duplicate webhook delivery, a duplicate analytics event — compounds faster than the failure that triggered the retry.