AWS S3: One Wrong Input, Four Hours Down
On February 28, 2017, an authorized member of the Amazon S3 team was debugging a slow billing system, following an established playbook. One input to a routine capacity-removal command was entered incorrectly, and a far larger set of servers was removed than intended — including the servers behind two subsystems every S3 request depends on. Both subsystems had to restart completely, something neither had done in the larger regions for many years, and while they restarted S3 could not answer requests at all. Other AWS services that rely on S3 went down with it, and for the first two hours Amazon could not even update its own status dashboard, because the dashboard's admin console depended on S3 too. Full recovery took until 1:54 PM PST. This is the documented story, from Amazon's own post-event summary, of how one wrong input became a four-hour outage — and the safeguards it forced.
The short answer
At 9:37 AM PST on February 28, 2017, an authorized member of the Amazon S3 team, using an established playbook (a written, step-by-step operating procedure) while debugging a billing system that was progressing more slowly than expected, executed a command meant to remove a small number of servers for one subsystem used by that billing process. One input to the command was entered incorrectly, and a much larger set of servers was removed than intended. The removed servers supported two other subsystems that every S3 request needs: the index subsystem, which manages the metadata and location information for all S3 objects in the region, and the placement subsystem, which manages allocation of new storage. Removing that much capacity forced each subsystem into a full restart, and while they restarted, S3 could not service requests. The index subsystem began servicing some requests at 12:26 PM PST and was fully recovered at 1:18 PM PST. The placement subsystem finished recovery at 1:54 PM PST, when S3 was operating normally — about four hours and seventeen minutes after the command. Amazon's summary is the source for every fact and time on this page.
The company: what Amazon did
Amazon Web Services, or AWS, is Amazon's cloud business: it rents computing power, storage, and networking to other companies by the minute, so a startup or a large enterprise can run its systems on Amazon's machines instead of buying its own.
The service at the center of this story is Amazon Simple Storage Service, shortened to S3. S3 is object storage — in plain English, a vast filing cabinet on the internet. You put a file in, called an object (a photo, a backup, a web page asset, a data file), and later you ask for it back. The asking uses a small set of operations whose names appear throughout this story, so here they are in plain English: GET means fetch an object, LIST means show what is in a container, PUT means store a new object, and DELETE means remove one. Together those four operations are, for most customers, the whole service.
The outage happened in the US-EAST-1 Region — Amazon's name for its cluster of data centers in Northern Virginia. A region is a geographic grouping of data centers that customers choose when they deploy. Why this region matters: S3 in US-EAST-1 did not only serve customers who stored files there directly. Other AWS services in the same region relied on S3 for their own storage, so when S3 stopped, they stopped too — a chain this case follows step by step.
The problem
Nothing about the morning started as reckless. That is why the case is worth studying.
First, removing capacity was normal work. Amazon's summary stresses this: S3 subsystems are designed to support the removal or failure of significant capacity with little or no customer impact. The systems are built on the assumption that things will occasionally fail, and the ability to remove and replace capacity is, in Amazon's words, one of its core operational processes — an operation it has relied on to maintain S3 since the service launched. In other words, the command that caused the outage was not an exotic experiment. It was routine maintenance, run by an authorized person, from an established playbook.
Second, the tool had no floor. The summary says it plainly in its fix section: in this instance, the tool used allowed too much capacity to be removed too quickly. There was no safeguard in the tool that said, in effect, "stop — this removal would take a subsystem below the minimum capacity it needs to keep running." So a single incorrect input was not caught by the tool, softened by the tool, or slowed down by the tool. It was executed, at full size, immediately.
Third, nobody knew how long a full restart would really take — because it had almost never happened. Amazon had not completely restarted the index subsystem or the placement subsystem in its larger regions for many years. In the years since, S3 had experienced, in the summary's words, massive growth. Restarting those services now meant bringing up far larger systems than the last time anyone had tried, and then running the necessary safety checks to validate the integrity of the metadata — the index that says where every object lives. You cannot skip those checks: an index you do not trust is not an index you can serve customers from. The restart plus the checks took longer than expected, and that gap between expected and actual restart time is where a short incident became a four-hour one.
An analogy: imagine a library where taking a few shelves offline for repair is routine and harmless. One morning, a work order with one wrong number takes not a few shelves but the entire card catalogue offline. The books are all still there. Nobody can find any of them — and rebuilding and double-checking the catalogue, which has grown for years since anyone last rebuilt it from scratch, takes hours.
What this shows: the command did not break S3's storage itself. It removed the servers behind the two subsystems that find objects and allocate space for new ones — so every request type stopped, and every service in the region that leans on S3 felt it.
What happened on February 28, 2017
All times are Pacific Standard Time (PST), exactly as Amazon gives them in its summary. Do not let the clock times hide the order of events: the index subsystem had to come back first, because the placement subsystem needs it in order to operate.
What this shows: four hours and seventeen minutes from one command to a fully normal S3 — with the telling detail in the middle: for the first two hours, the status page could not be updated service by service, because it depended on the service that was down.
Before 9:37 AM — The S3 team is debugging an issue that is causing the S3 billing system to progress more slowly than expected. Billing sounds like back-office work, far from customer traffic. Keep that in mind: the subsystem being debugged was not the one that failed. The servers removed by the command supported other subsystems as well.
9:37 AM — An authorized S3 team member, using an established playbook, executes the command. It is intended to remove a small number of servers for one of the S3 subsystems used by the billing process. One input is entered incorrectly. A larger set of servers is removed than intended.
What the removal actually hit — The inadvertently removed servers supported two other S3 subsystems. The index subsystem manages the metadata and location information of all S3 objects in the region, and it is necessary to serve all GET, LIST, PUT, and DELETE requests. Without it, S3 cannot find anything. The placement subsystem manages allocation of new storage, is used during PUT requests to allocate space for new objects, and requires the index subsystem to be functioning properly in order to operate correctly. Removing a significant portion of the capacity caused each of these systems to require a full restart. While the subsystems were being restarted, S3 was unable to service requests.
The knock-on impact — Other AWS services in US-EAST-1 that rely on S3 for storage were also impacted while the S3 APIs (the interfaces other software uses to talk to S3) were unavailable. Amazon names them: the S3 console itself, Amazon EC2 new instance launches (starting new virtual servers), Amazon EBS volumes when data was needed from an S3 snapshot (a saved copy of a disk), and AWS Lambda (a service that runs code on demand). Notice the pattern: none of those services had to be broken themselves to be down. They only had to depend on S3.
The blind status page — From the beginning of the event until 11:37 AM PST, Amazon was unable to update the individual services' status on the AWS Service Health Dashboard (SHD) — the public page customers check during an outage — because of a dependency the SHD administration console has on Amazon S3. The tool for telling customers what was wrong depended on the thing that was wrong. Instead, Amazon used the AWS Twitter feed (@AWSCloud) and SHD banner text to communicate status until it was able to update individual services' status on the dashboard.
Why the restart took so long — The index subsystem was the first of the two affected subsystems that needed to be restarted — placement could not finish without it. But S3 had grown massively since the last complete restart of these subsystems in a large region, years earlier. Restarting them and running the necessary safety checks to validate the integrity of the metadata took longer than expected. There is no shortcut in that sentence: the checks are what make the recovered index trustworthy.
What this shows: recovery was a queue, not a race. The index had to be trusted first; only then could placement finish and PUT — storing new objects — come back. That ordering is why the last step landed at 1:54 PM, not earlier.
The damage
Amazon's summary does not publish a dollar figure, a count of affected customers, or a list of websites that went down — so this page invents none of those. The documented damage is duration, breadth, and a communications failure layered on top of a service failure.
Duration: from the 9:37 AM command to S3 operating normally at 1:54 PM is about four hours and seventeen minutes during which, for much of the time, S3 in US-EAST-1 could not service requests at all. Partial relief came at 12:26 PM, when GET, LIST, and DELETE began to be serviced — but PUT, the operation that stores anything new, needed placement as well, and did not return to normal until the end.
Breadth: the impact was not confined to people storing files. Because the S3 console, EC2 instance launches, EBS volumes that needed data from an S3 snapshot, and Lambda all rely on S3 in the region, a storage outage became a compute outage and a management outage at the same time. A customer could have every one of their own systems healthy and still be unable to launch a server, restore a volume, or open the console.
Visibility: for roughly the first two hours, customers could not get per-service status from the dashboard built for exactly that purpose. Amazon communicated by Twitter feed and a banner instead — workable, but an admission that the reporting path and the failing path were the same path. Amazon closes its summary with an apology, acknowledging how critical S3 is to its customers, their applications and end users, and their businesses, and saying it would do everything it could to learn from the event. That apology is part of the record; the fixes below are the concrete part.
How they recovered
Step one: restart the index, and check it properly. The index subsystem was restarted first, because everything else depends on it. Recovery meant not only bringing capacity back, but running the safety checks needed to validate the integrity of the metadata. By 12:26 PM PST, the index subsystem had activated enough capacity to begin servicing GET, LIST, and DELETE requests. By 1:18 PM PST, it was fully recovered and those APIs were functioning normally.
Step two: then recover placement. The placement subsystem began recovery when the index subsystem was functional — it cannot operate correctly without it — and finished recovery at 1:54 PM PST. At that point S3, including the PUT API, was operating normally.
Step three: let the dependents drain their backlogs. Other impacted services began recovering once S3 was back. Some of them had accumulated a backlog of work during the disruption and required additional time to fully recover. That tail is normal in dependency outages, and worth planning for: "the core service is back" and "everything is back" are two different times.
Step four: communicate around the broken tool. Throughout, until 11:37 AM, status communication ran through the @AWSCloud Twitter feed and the SHD banner text, because the dashboard's own administration console could not be used to update individual services. Recovery of the message mattered almost as much as recovery of the service — customers making decisions in the dark make worse decisions.
The fix: technology and controls that came out of it
Amazon lists its changes in the summary itself. Each one maps to a specific failure in the story above — none of them is "tell operators to be more careful."
Put a floor and a speed limit on the removal tool. Amazon modified the tool to remove capacity more slowly, and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level. Read that carefully: the fix does not try to prevent a human from typing a wrong input. It makes a wrong input unable to trigger a similar event. Slow removal buys time to notice; a minimum-capacity floor makes the worst case bounded.
Audit every other tool for the same gap. Amazon said it was also auditing its other operational tools to ensure they have similar safety checks. One tool with no floor is a bug. A fleet of operational tools, some with floors and some without, is a system waiting for the same morning on a different service.
Make recovery faster by making pieces smaller — cells. To improve recovery time, Amazon pointed to one of its most important techniques: breaking services into small partitions it calls cells. By factoring services into cells, engineering teams can assess and thoroughly test the recovery processes of even the largest service or subsystem, and smaller pieces mean a smaller blast radius (the amount of the system one failure can touch). S3 had already done considerable work to refactor parts of the service into smaller cells. During this event, however, recovery of the index subsystem still took longer than expected. The S3 team had planned further partitioning of the index subsystem later that year; Amazon reprioritized that work to begin immediately.
Move the status dashboard off the service it reports on. Amazon changed the SHD administration console to run across multiple AWS regions. After this change, an outage in one region — including an S3 outage in that region — should no longer blind the console used to tell customers about it.
What this means for you
Hyperscaler engineers
This is your canonical "operator error is a system design" case. The operator was authorized, trained enough to be on the playbook, and debugging a real problem — and the system still let one input remove unbounded capacity instantly. Your checks follow directly. Do your destructive operational tools enforce a minimum-capacity floor and a maximum rate of change in code, not in a wiki page? Have you actually restarted your largest subsystems end-to-end recently enough to know the time — with integrity checks included — or is your recovery time an inherited guess from when the system was a tenth of its current size? And is your fleet partitioned into cells small enough that a full restart of one cell is a rehearsed, timed event rather than a first-time discovery? Amazon reprioritized exactly that partitioning after this event; the lesson is to do it before yours.
Enterprise network engineers
Your version of this command is the change that removes "a few" routes, VLANs, firewall rules, or cluster members — typed by a good engineer, in a maintenance window, from a runbook. The transferable controls are identical: rate-limit and floor your bulk changes (never let one command take a service below its redundant minimum), rehearse the restart you assume will save you (a failover or rebuild that has not run in years, at current scale, is a hope, not a plan), and audit your out-of-band path. If your monitoring dashboard, jump host, or status page authenticates through, or runs on, the very system it is meant to report on, you have Amazon's 11:37 AM problem in miniature. Find that dependency on a quiet day.
HFT / low-latency engineers
Your blast radius is measured in seconds and dollars, which makes the floor lesson sharper, not different. A tool that can drain liquidity, widen a kill-switch scope, or remove market-data capacity should have the same two properties Amazon added: it acts slowly enough to be stopped, and it refuses to go below a hard minimum no matter what input it is given. And take the restart lesson literally: a matching engine, feed handler, or risk check that has never been cold-restarted at current message rates has an unknown recovery time. In your world, unknown recovery time during a live session is the outage.
Try it yourself
Two habits from this case are worth practising with free tools. First, never assume a fleet matches its intended state — the habit behind capacity floors and drift checks. Packet Path's free Network Config Verifier compares device configurations side by side and flags mismatches, so you can build the compare-before-you-trust reflex. Second, think in capacity floors and headroom before you need them: the free Data Center Capacity Planner helps you model how much capacity a design really has, and how much is left when part of it is removed — the exact arithmetic the S3 tool was missing a safeguard for.
Sources
Primary source — Amazon Web Services, "Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region" (post-event summary for February 28, 2017): the 9:37 AM PST command and the incorrect input, the index and placement subsystems and what each does, the full-restart requirement, the impacted services (S3 console, EC2 new instance launches, EBS volumes when data was needed from an S3 snapshot, and AWS Lambda), the recovery times (index beginning GET, LIST and DELETE at 12:26 PM PST, index fully recovered at 1:18 PM PST, placement finished at 1:54 PM PST with S3 operating normally), the Service Health Dashboard dependency and the 11:37 AM PST limit with Twitter and banner communication, and all fixes (slower removal with a minimum-capacity safeguard, audit of other operational tools, reprioritized partitioning of the index subsystem into cells, and the SHD administration console changed to run across multiple AWS regions). https://aws.amazon.com/message/41926/
No secondary sources are used for facts on this page. Where press coverage cites dollar losses or lists of affected websites, those figures are not in Amazon's summary and are not used here.
- The trigger was not malice or overload: an authorized operator, using an established playbook, entered one input incorrectly — and the tool allowed too much capacity to be removed too quickly, with no floor to stop it.
- Restart time is part of your failure plan: neither key subsystem had been fully restarted in the larger regions for many years, so after years of growth the restart plus metadata safety checks took far longer than expected.
- Dependencies hide in the recovery tools too: for two hours Amazon could not update individual service status on its health dashboard because that console itself depended on the S3 service that was down.
- Amazon's fixes were mechanical, not motivational: remove capacity more slowly, never below a subsystem's minimum required capacity, audit other tools for the same gap, and split the index into smaller cells so recovery is faster and the blast radius is smaller.