Building Resilient Software: Planning for the Day Things Break

Picture a Tuesday night. A product team ships a small update, closes their laptops, and goes home. At 2:14 AM a pager goes off. The primary database is unreachable, and the dashboard is filling up with errors.

Nobody on this team was careless. They had backups. They had monitoring. What they did not have was a clear answer to a simple question: what exactly do we do now?

That question is the subject of this article. Most teams plan carefully for the day a product succeeds and very little for the day it fails. Yet servers fail, cloud regions go offline, and AI providers stop responding. These events are not rare, and they are rarely anyone's fault.

The ideas here are drawn from Sam Newman's Building Resilient Distributed Systems. I have organized them as a story: from the worst day, to the numbers that define it, to the choices that make it survivable.

01Why this matters more now

Teams ship faster than they ever have, and AI coding tools are a large part of the reason. More code means more deployments, and more deployments mean more change reaching production. Every change is a small chance for something to go wrong.

Writing software quickly has become the easy part. Keeping it running is the part that now separates good teams from the rest.

02The worst day, planned in advance

Disaster recovery is the set of steps a team follows when a large part of the system disappears, such as a full data center or an entire cloud region. A good plan answers four plain questions:

1. Where will the data come from?
2. Where will the new servers run?
3. How will the data be restored?
4. How will user traffic find its way to the new system?

Back to our team at 2:14 AM. If they have written answers to those four questions, the night is stressful but short. If they do not, the same incident turns into hours of improvised decisions made by tired people.

There is one point that teams overlook more often than any other: a backup is not a recovery plan. A backup is only a copy of your data. It becomes useful when someone has restored it successfully, and that has to happen before the real incident, not during it. A backup that has never been restored is a hope, not a safeguard.

03Two numbers that decide how bad it gets

Once a team accepts that failure will happen, two questions shape every decision that follows.

RPO, the Recovery Point Objective, asks how much data the business can afford to lose. If backups run once an hour, up to an hour of data can disappear. The RPO is one hour.

RTO, the Recovery Time Objective, asks how long the system can stay offline. If it takes two hours to get back online, the RTO is two hours.

A simple way to hold the difference: RPO is about data loss, and RTO is about downtime.

Take our team. The last backup finished at 1:00 AM. The server failed at 2:00 AM. The application was restored at 4:00 AM. Everything written between 1:00 and 2:00 is gone, and users were without service from 2:00 to 4:00.

Zero data loss and zero downtime sound ideal, but they are expensive and complex to achieve. They also do not apply equally to every part of a product. In a banking app, payments need very low targets, while new account sign-ups can tolerate more.

These targets should be agreed with the business, not set by engineers alone. Lower targets cost more, so the right answer depends on what users need, what regulators require, and what the budget allows.

04Choosing how fast you want to recover

There are four common ways to prepare a second location for a bad day. They are ordered here from the cheapest and slowest to the most expensive and fastest.

The trade is always the same. Each step toward faster recovery buys you a lower RPO and RTO, and charges you in cost and complexity.

Backup and Restore

This is where most teams begin. Nothing runs at the second location until a disaster happens. When it does, the team builds new servers and restores the latest backup into them. The RPO depends on how often backups are taken.

Pilot Light

Here the data is copied to the second site continuously, so it is always current, like a pilot light that stays lit. The servers stay off until they are needed, which keeps the cost low and shortens recovery.

Warm Failover

The most important services are already running at the second site, at small capacity. When disaster strikes, they take over first and the rest scales up behind them.

Multisite Active-Active

Two or more sites serve users all the time. If one fails, a router sends everyone to the others, so there is no real failover step. It is the fastest option and the most expensive one.

Most teams do not need the top option on day one. Start with backup and restore, and move up as the needs of the business grow. Different parts of the same system can also use different patterns.

05Will the servers be there when you need them?

Many recovery plans quietly assume that new servers can be created on demand. In a major outage, that assumption can fail. When a whole region goes down, many customers rush to claim capacity elsewhere at the same moment, and the supply may simply run out.

A plan that depends on capacity that is not there is not a plan. There are three ways to prepare:

• Reserve capacity ahead of time, where the provider offers it. You pay for it even when it sits unused.
• Prioritize critical services, so that limited capacity goes to what matters most.
• Keep redundant infrastructure ready. This is the most expensive option, though the spare resources can earn their keep in the meantime by running load tests.

06When your product depends on AI

Many products now lean on external AI models. These bring risks that behave differently from a database or a standard API, and they deserve their own planning.

The provider can disappear

AI services go down, hit rate limits, and run out of quota. The product should keep working when that happens. This is called graceful degradation. If an AI summary feature fails, users should still be able to open, edit, and save their documents. The feature fails quietly, and the work continues.

The model can change without warning

A model update can change the quality or format of the output, even when nothing was announced. The goal is for the team to notice before users do. That means automated tests for the AI parts of the product, evaluations that measure output quality, and a validation step before any switch to a new model.

The response time can vary enormously

One request may take two seconds and the next may take thirty. AI calls should not be treated like predictable database queries. Set sensible timeouts, stream output so users see progress early, handle rate limits and token limits properly, and monitor latency and error rates so that a slow decline is noticed early.

07Living with uncertainty in the AI space

Avoid depending on a single vendor

Design the product so that switching models or providers does not require a rewrite. If a provider raises prices, changes behavior, or goes offline, you want options.

Two cautions apply. A backup provider only helps if it can actually replace the primary one for the task at hand. And a prompt tuned for one model may behave differently on another, so it must be tested again after any switch.

More vendors do not automatically mean more safety. If a single workflow needs several providers working at once, a failure in any one of them breaks the whole flow. A backup should replace the primary, not extend the chain.

Make the cost visible

Pay-as-you-go pricing can grow faster than anyone expects. Track spending by provider, feature, and workflow, and set budget alerts. High spending is not always a problem. If it produces real productivity or revenue, it may be money well spent. The danger is spending that nobody can see.

Review your vendors on a schedule

A vendor you trusted last year may not deserve the same trust today. Review them regularly. Is the service reliable? Is the pricing changing? Is the quality holding up? Does the vendor itself depend heavily on one other company? And what would you do if it became unavailable tomorrow? Keep your security threat model current as well, especially when new AI features are added.

08Where a small team should start

A complex multi-site setup is usually too much, too early, for a small team or a new product. This order is more realistic:

1. Automated database backups, with restores that are actually tested.
2. Monitoring and alerts for errors, downtime, latency, and resource usage.
3. A written recovery plan that says what to do if a server or database fails.
4. AI error handling that covers timeouts, rate limits, and provider failures.
5. Cost controls with usage tracking and budget alerts.
6. Regular recovery practice, to confirm the system can really be restored from a backup.

09The morning after

Return one last time to the team at 2:14 AM. The difference between a bad night and a bad week is rarely talent. It is whether someone, weeks earlier, wrote the plan, tested the restore, and agreed on how much loss and downtime the business could accept.

Great software is not software that never fails. It is software that fails safely, recovers quickly, and keeps the trust of the people who use it. Backups, monitoring, and recovery plans only create value once they have been tested, and the cheapest time to find a gap is during practice, not during an outage.

Source and credit: This article is a summary and commentary inspired by Building Resilient Distributed Systems by Sam Newman (O'Reilly, 2026). For the full detail, please read the book.