What resilience actually means
Resilience is usually sold as redundancy, and redundancy is only one part of it. A system is resilient when a failure in one part does not take down the rest, when the failure is noticed by a monitor rather than a customer, and when getting back to normal is a documented procedure instead of an improvisation.
It is also a business decision before it is a technical one. Not everything needs the same guarantee. Deciding which parts of the system must never lose data, which may be briefly unavailable, and which can be down for a day tells the engineering exactly where to spend - and, just as usefully, where not to. It is usually the first question in an architecture review.
Know your failure modes
Write down what can break before designing around it. The list is shorter than it feels: a dependency is slow, a dependency is down, a disk fills, a deployment is bad, a credential expires, traffic spikes, or someone deletes the wrong thing.
For each one, the useful questions are what the visitor sees, what the on-call person sees, and how the system behaves while it is happening. A dependency that is slow is often worse than one that is down, because without a timeout it consumes every connection the application has and turns one broken feature into an outage.
- Set explicit timeouts on every outbound call, and a retry policy that will not amplify an outage
- Fail one feature closed rather than letting it exhaust shared resources
- Make expiry dates - certificates, tokens, credentials - something a monitor watches, not a diary
- Rehearse a bad deployment: the rollback is only real if someone has run it
You cannot fix what you cannot see
The minimum useful setup is smaller than the tooling market suggests: structured logs that can be searched, a handful of metrics tied to what the business cares about, and an alert on each one that a human would actually act on.
The discipline is in the last part. Alerts that fire routinely and are routinely ignored are worse than no alerts, because they train the team to dismiss the one that matters. If an alert does not have an action attached to it, it belongs on a dashboard, not in a pager.
A backup is a restore you have practised
Backups get configured and then trusted for years without anyone testing whether they can be read. The only meaningful measure is a restore: how long it takes, who can perform it, and how much data was lost between the last backup and the failure.
Run that restore into a scratch environment on a schedule. It will surface the things a backup policy never does - a dependency that was never included, a credential nobody has, a procedure only one person knows. Migrations deserve the same discipline: when we rebuilt a legacy ERP on a cloud platform, data moved in validated batches with a reconciliation at each step.
Resilience you can afford
Multi-region, active-active infrastructure is right for a small number of businesses and ruinous for the rest. Most operations get far more from unglamorous work: tested restores, health checks, sensible timeouts, a rollback that runs in one command, and a documented on-call procedure.
Spend on high availability where downtime has a direct cost you can name. Elsewhere, spend on recovery speed instead. The second is dramatically cheaper and covers a far wider range of things that actually go wrong.
