Designing for Failure on a Cloud Computing Platform Moving to a Cloud Computing Platform did not make our systems more reliable. For the first eighteen months it made them less reliable, because we had rebuilt the same single points of failure on infrastructure that fails more visibly. Here is what we learned about designing for the failures a Cloud Computing Platform actually produces. The common failures are small and frequent Not the regional outage everybody plans for. An instance disappearing. A managed service being briefly unavailable. A network path slowing. A dependency rate limiting you. A deployment rolling out badly. Those happen weekly. Designing for a regional failure while being unable to tolerate a single instance restarting is the wrong order, and it is the order most architecture reviews follow. Retry carefully, and give up Retries fix most transient failures and cause most cascading ones. Ours retry with increasing waits and random jitter, up to a stated limit, and then stop. Without jitter, every client retries at the same moment and the recovering service is immediately overwhelmed again. We also stop calling a dependency that is clearly failing, rather than queuing requests against it. A service that fails fast is far easier to recover than one holding thousands of pending calls. Decide what degraded looks like The design question people skip. When the recommendation service is unavailable, our product pages show the catalogue without recommendations. When the loyalty service is down, checkout completes and points are credited later. Each of those is a decision somebody made about what the customer sees. The alternative, which we had, is that any dependency failing returns an error page for everything. A Cloud Computing Platform makes individual components more likely to wobble and makes graceful degradation more valuable, not less. Set timeouts everywhere, deliberately The most common omission we find. A call without a timeout waits forever, holds a connection, and turns one slow dependency into an exhausted connection pool and a dead service. Every call has a timeout, set shorter than the caller's own, so failures surface at the right layer rather than propagating as timeouts all the way up. Spread across failure domains, and check Three instances on one host is one instance of resilience. Our scheduling spreads instances across hosts and zones, and we verify it rather than assuming. Under capacity pressure, platforms will quietly place things together. We check placement as part of service review. Twice it had drifted without anybody noticing. Practise the failure you fear Documents are not capabilities. We run an exercise each quarter. Remove a zone, fail a managed service, exhaust a connection pool, expire a certificate, in a test environment, with whoever is on call working it through using the real tooling. The first exercise found a runbook referencing a dashboard that had been replaced and two engineers without the access they would have needed. Both would have cost an hour during something real. Protect the data paths too Resilience is usually discussed as a service concern. Our batch and analytical paths needed the same thinking. During our data warehouse migration to cloud we found jobs that could not be safely rerun, which meant a transient failure required somebody to clean up by hand before retrying. Every job now tracks progress, resumes from the last good point, and writes idempotently. That turned overnight incidents into automatic retries. Verify the recovery path works Backups nobody restores are a belief rather than a control. We restore from backup quarterly, in a test environment, timed. The first attempt took nine hours and the documented expectation was two, because nobody had done it since the system was first built. Our Software Testing and Quality Assurance team owns that exercise now, which is the right home for it. Verifying that something works when it is supposed to is exactly what testing does. Alert on the deadline, not the failure A job failing and succeeding on retry is not an incident. We alert when data has not arrived by the time it is needed and when a journey is failing for customers, rather than on individual component errors. That reduced our out of hours alerts by around eighty per cent, and we have not missed a real problem since. Our resilience standard ● Timeouts on every call, shorter than the caller's own. ● Retries with backoff, jitter and a stated limit. ● A defined degraded behaviour for every dependency. ● Spreading across hosts and zones, verified rather than assumed. ● Jobs that resume and write idempotently. ● Alerts on customer impact and deadlines, not component errors. ● A quarterly failure exercise with real tooling. Common questions Is multi region worth it? For most workloads, no. Getting single region resilience right is cheaper and addresses the failures that actually occur. Who should own the failure exercises? Whoever owns verification. Our Software Testing and Quality Assurance team runs ours, alongside the engineers who built the systems. How much headroom should we keep? Enough to lose a host at peak without disruption. Autoscaling is not instant and a traffic spike does not wait. Does this change on a managed platform? The mechanisms change and the decisions do not. A Cloud Computing Platform supplies failover and says nothing about what your product shows when a dependency is unavailable. What causes most incidents? Missing timeouts and retry storms, in our experience. Neither is sophisticated and both are easy to leave out. Does the platform make this easier? A Cloud Computing Platform gives you the mechanisms and none of the decisions. Nothing in it decides what your product does when a dependency is unavailable. What changed Our customer facing incidents fell substantially in the year after we worked through this, and none of it required new infrastructure. Timeouts, backoff with jitter, defined degraded behaviour, verified spreading, and jobs that can be rerun. Five unremarkable things, each easy to postpone, and together the difference between a platform that absorbs small failures and one that amplifies them.