Packet Path/

GitLab 2017: One Wrong Server, 300 Gigabytes Deleted

On January 31, 2017, a GitLab engineer trying to fix a lagging backup database wiped the data directory — on the wrong server. About 300 gigabytes of the live GitLab.com database vanished in a second or two. Then the team went looking for backups and found that every safety net had a hole in it: the regular backups had been failing silently, the disk snapshots did not cover the database at all, and the one snapshot that worked was six hours old and lived on slow disks. GitLab.com was down for about 18 hours. This is the documented story, from GitLab's own public postmortem.

Case Studies · Intermediate · 12 min · November 10, 2026

Editorial illustration on a dark teal background: two identical database server towers side by side, the left one glowing calm green and the right one glowing alarming red with its storage disks fading away, a faint warning line between them.

The short answer

On the night of January 31, 2017, GitLab.com — the hosted service where many software teams store their code — was already struggling under a load spike when its backup database fell behind and had to be rebuilt. Rebuilding it meant wiping its data directory (the folder where the database keeps its files) and copying a fresh copy from the main database. An engineer ran that wipe on the wrong machine: the primary database, db1.cluster.gitlab.com, instead of the secondary, db2. The engineer stopped the command a second or two after realising the mistake, but about 300 gigabytes of production data had already been removed. GitLab then discovered its backups were not there: the scheduled database backups had been failing silently, and a different snapshot system had never been switched on for the database servers. The service was restored from a snapshot taken at about 17:20 UTC, roughly six hours earlier, after a copy that took around 18 hours over slow disks. Database changes made between 17:20 and the deletion at about 23:30 UTC were lost — GitLab estimated roughly 5,000 projects, 5,000 comments, and 700 new user accounts. The code repositories themselves were not lost, because they were stored separately.

The company: what GitLab did

GitLab makes software that software teams use to store code, track work, and ship products. GitLab.com is its hosted service: instead of installing GitLab on their own servers, teams use the copy GitLab runs for everyone. If GitLab.com's database goes down, those teams cannot reliably open issues, comment, or manage their projects — even though the code files themselves may still exist somewhere else.

That split is the key to understanding both the damage and the limits of the damage here. A repository is the folder of code and its history. GitLab stored repositories and wikis (project documentation pages) separately, on NFS servers — NFS is a standard way of sharing files over a network. The database, by contrast, held the working records around the code: projects, issues, comments, user accounts, and snippets. GitLab's database was PostgreSQL, a widely used open-source database program.

The database setup, in GitLab's own postmortem, was simple and fragile: one primary server that handled the work, named db1.cluster.gitlab.com, and one secondary in hot standby, db2.cluster.gitlab.com, kept in step by replication — the primary continuously sends its changes to the secondary, so the secondary can take over if the primary fails. One database handling all the load, GitLab noted, was not ideal, and db1 had already been a single point of failure in earlier incidents. On January 31, that single point is exactly where the accident landed.

The problem: a tired night, two look-alike servers, and backups nobody had tested

No attacker caused this. The setup had four weaknesses, and they lined up on the same night.

First, the two database servers were easy to confuse. Their names differed by one character — db1 and db2 — and an engineer working late, in the middle of a live incident, had to be sure which terminal window was connected to which host before running a destructive command. GitLab's later fix list starts with making hosts look different, which tells you how similar they felt at the time.

Second, fixing replication required a manual, destructive step. When the secondary falls too far behind, one way to resync it is to empty its data directory and copy the database over again with a tool called pg_basebackup. That procedure was, in GitLab's words, neither automated nor properly documented. A manual wipe-and-copy at midnight is a risk even on the right server.

Third, the regular backups had been failing — silently. Every 24 hours, GitLab made a logical backup (a portable copy of the database contents) using a tool called pg_dump, and uploaded it to Amazon S3 storage. The backup job ran on a regular application server, not on the database server. That detail mattered: GitLab's packaging software chose which PostgreSQL version to use by looking for a database data directory on the machine. The application server had none, so it defaulted to PostgreSQL 9.2 — while the live database ran 9.6. A 9.2 backup tool cannot back up a 9.6 database, so every backup errored out. The job did send failure notifications by email — but those emails were rejected by the receiving mail server because they were not signed in the way GitLab's DMARC setup required (DMARC is an email-authentication system that helps receivers reject spoofed mail). No other alerting channel existed. The backups had been failing, the alarms had been bouncing, and nobody knew.

Fourth, the other snapshot systems did not cover the database. GitLab used Azure disk snapshots — point-in-time copies of a whole disk — for various servers, including the NFS servers holding the code. They were not enabled for any of the database servers, because the team assumed the other backup procedures were sufficient. And its LVM snapshots (LVM is a Linux disk-management layer; a snapshot is a frozen copy of a disk at a moment in time) were taken every 24 hours mainly to copy production data into a test environment — not designed as disaster recovery at all. As GitLab's root-cause analysis later put it, the backup procedure was not tested regularly because there was no ownership: nobody was responsible for testing it.

An analogy: imagine two identical storage units, numbered 1 and 2, and a night worker told to empty unit 2. The labels look the same in the dark. The spare key you counted on turns out never to have been cut — the key machine rejected every blank and emailed you about it, but the emails went to a spam bin nobody checks. And the security camera system covers every unit except this row. None of that requires malice. It just requires nobody testing the whole chain, end to end, before the night it is needed.

Four safety nets — three had holes, one was never a net PG_DUMP BACKUPS — FAILED SILENTLY Backup tool was version 9.2; the database ran 9.6, so it errored Failure emails were rejected under DMARC — nobody was told AZURE DISK SNAPSHOTS — NOT ENABLED Enabled for the NFS servers holding the code — but never switched on for the database servers at all REPLICATION — ALREADY BROKEN The secondary lagged, then its data was wiped to rebuild it — so it held no copy to fail over to when the primary was hit LVM SNAPSHOT — THE ONE THAT WORKED Taken at about 17:20 UTC for testing — about six hours old, never designed as disaster recovery, but it held the data A backup you have never restored is a hope, not a backup. Three of these four were hopes.

What this shows: the four recovery paths GitLab had on paper, and what each one actually held on the night. Only the six-hour-old test snapshot contained usable database data.

What happened on January 31, 2017

All times are UTC, as given in GitLab's postmortem. The postmortem marks several times with a plus-or-minus sign; this page keeps that honesty and says "about" where GitLab did.

January 31, 2017 — the night in UTC About 17:20 — an engineer snapshots the production database An LVM snapshot, taken to load fresh data into the test environment About 19:00 — database load spikes Suspected spam, plus a background job removing an employee account flagged for abuse About 23:00 — replication fails on the secondary The log segments it still needs were already deleted from the primary — manual resync required Late evening — the resync fights back pg_basebackup hangs silently; a settings change needs max_connections cut from 8000 to 2000 About 23:30 — the wipe runs on the primary Stopped in a second or two — but about 300 GB was already gone February 1, about 18:00 — restoration finished Database back at its 17:20 state, webhooks restored, service gradually re-enabled

What this shows: six hours separated the snapshot that saved GitLab from the mistake that needed it — and the incident that led to the mistake started hours before the deletion itself.

About 17:20 — the snapshot that would matter. An engineer working on a test setup took an LVM snapshot of the production database and loaded it into the staging (test) environment, so load testing would use fresh data. Snapshots like this normally happened automatically every 24 hours; this manual one was about six hours fresher than the schedule required. Nobody took it as a backup. It became one anyway.

About 19:00 — the load spike. GitLab.com's database load rose sharply. Users could not post comments on issues and merge requests, and getting the load under control took several hours. GitLab later traced the load to two things at once: what it suspected was spam, and a background job trying to remove a GitLab employee and their associated data — the employee's account had been flagged for abuse and accidentally scheduled for removal.

About 23:00 — replication breaks. Under that load, the secondary's replication lagged behind the primary. Replication in PostgreSQL works from WAL segments — write-ahead log files, the database's running journal of changes. The primary had already removed journal segments the secondary still needed, and GitLab was not archiving the journal elsewhere, so the secondary could not catch up on its own. It had to be resynchronised manually: wipe its data directory, then run pg_basebackup to copy the database across from the primary.

Late evening — the resync stalls, twice. An engineer wiped the secondary's directory and ran pg_basebackup — which hung, producing no meaningful output even in verbose mode. After several tries it reported it could not connect because the primary did not have enough replication connections available (a setting called max_wal_senders, then 3). The engineers raised that setting to 32 — and PostgreSQL then refused to restart, because another setting, max_connections, had been set to 8000 almost a year earlier, far too high; lowering it to 2000 let the database restart. Even then, pg_basebackup still appeared to hang. (Another engineer later explained this was normal: the tool waits silently until the primary starts sending data. That behaviour, GitLab noted, was documented neither in its runbooks nor clearly in the tool's own documentation.) Debugging with a tracing tool showed only that the tool was waiting.

About 23:30 — the wrong server. Thinking pg_basebackup's earlier attempts might have left files in the secondary's data directory, an engineer proceeded to wipe the PostgreSQL database directory — believing it was the secondary's. It was the primary's. The engineer terminated the process a second or two after noticing the mistake. By then, about 300 GB of data had already been removed. The team asked for help on Slack and went looking for the backups — and found, in GitLab's words, that the process of both finding and using backups failed completely: the pg_dump bucket in S3 was empty, Azure snapshots did not cover the database, and replication was broken with data already wiped from both hosts.

The damage

The outage: GitLab.com was down for about 18 hours in total, per the postmortem's root-cause section. For a hosted service that teams build their working day around, that is not a blip — it is a full working day, worldwide, with issues and comments unavailable.

The data loss: database data created or modified between 17:20 and 23:30 UTC was lost. GitLab's best estimate: roughly 5,000 projects, 5,000 comments, and 700 new user accounts, along with issues and snippets in that window. One nuance the postmortem is careful about: some projects created after 17:20 could be partially rebuilt from their repositories, because the repository still existed — but only in basic form, without the associated issues and merge requests held in the database.

What was NOT lost matters just as much. Code repositories and wikis were unavailable during the outage but were not affected by the data loss, because they lived on separate NFS storage. GitLab Enterprise customers, GitHost customers, and people running their own GitLab installations were not affected at all. The blast radius was the hosted service's database — large, but bounded by that one separation GitLab had got right.

The trust damage is harder to count, and GitLab treated it as real: the postmortem opens with a personal apology from its CEO, on behalf of the whole company, to the users whose data was lost. The company also published the incident details — including the unflattering backup findings — for anyone to read, and live-streamed the recovery. Transparency was part of the damage control, and this case study exists only because GitLab chose to document itself that openly.

How they recovered

Step one: pick the least-bad copy. The only usable database copy was the LVM snapshot from about 17:20 UTC — the one loaded into staging. The alternative, the previous day's snapshot, would have lost almost 24 hours of data instead of about six. GitLab chose the staging copy.

Step two: copy it back — slowly. The staging environment ran on cheaper Azure storage without Premium Storage, and its disks were the bottleneck: network disks throttled to around 60 Mbps. Copying the PostgreSQL data directory from staging back to the production host took around 18 hours. There was, GitLab noted, no network or processor bottleneck — the bottleneck was the drives, and there was no way to move from cheap storage to premium mid-incident. The recovery plan also had to handle webhooks carefully (webhooks are automatic notifications GitLab sends to other systems when things happen; the staging copy had had them removed so tests would not trigger real notifications). The plan: restore the main database from the staging copy, build a second database from the raw snapshot, extract the webhook table from it, import that into production, and increment all database sequences by 100,000 so new records could not reuse IDs that might have been used in the lost window.

The recovery: one old snapshot, one very slow copy 17:20 SNAPSHOT staging copy, about six hours old COPY BACK — 18 HOURS disks throttled to around 60 Mbps DATABASE BACK Feb 1, 17:00 UTC, without webhooks FULLY BACK about 18:00 UTC webhooks in THE DETAILS THAT KEPT THE RESTORE HONEST Sequences raised by 100,000 so restored records could not collide with IDs used in the lost window. GitLab kept a public incident document, streamed the recovery live (peak about 5,000 viewers), and posted updates on Twitter while the copy ran — users watched the restore happen in real time. The snapshot was six hours old because an engineer took it by hand. The schedule alone would have cost a day.

What this shows: recovery was mostly waiting — one long copy over throttled disks — followed by careful finishing work: webhooks restored from a second copy of the snapshot, ID sequences raised, and the service re-enabled gradually.

Step three: finish and re-enable, February 1. At 17:00 UTC on February 1, the database was restored without webhooks. The team then built a separate staging database from the snapshot without stripping webhooks, dumped that one table, and imported it into production. Around 18:00 UTC, the final restoration procedures were finished and everything was confirmed operating as expected, with GitLab.com gradually re-enabled.

Step four: tell everyone, while it was still happening. Throughout, GitLab kept a publicly visible incident document, streamed the recovery on YouTube (peak around 5,000 viewers), and posted updates on its status Twitter account. It is worth naming why that belongs in a recovery section and not just a publicity one: users whose work was frozen could see the state of the restore instead of guessing, and the public record GitLab created is the reason the failure modes in this case are documented well enough to learn from.

The fix: technology and controls that came out of it

GitLab published its improvement list as tracked issues in the postmortem itself. Grouped, they answer each weakness in order:

Make the host you are on unmistakable. Update the command prompt (PS1) across all hosts to more clearly differentiate between hosts and environments. GitLab was explicit that its main focus was better recovery and clearer hosts — not banning engineers from running dangerous commands, because a safer delete command would not protect against disk corruption or the many other ways data can be lost. The goal it stated: an environment where you can make mistakes but recover from them quickly, with minimal impact.

Monitor the backups, publicly. Prometheus monitoring for backups (Prometheus is a monitoring system), plus a public backup-monitoring dashboard so the age of the latest backup is visible. A backup job should page someone when it fails — through a channel that cannot itself silently bounce.

Test restores, automatically, and give them an owner. Automated testing of recovering PostgreSQL backups, and — the line that explains the whole incident — assign an owner for data durability. The root cause was not that testing was hard; it was that testing was nobody's job.

Take snapshots that actually cover the database, more often. Hourly LVM snapshots of the production databases instead of every 24 hours, Azure disk snapshots of the production databases, and investigations into point-in-time recovery with continuous archiving, pgbarman, and WAL-E — tools and methods that keep the database journal so a database can be replayed to a chosen moment, not just to the last daily copy.

Write the runbooks and fix the settings. Improved PostgreSQL replication documentation and runbooks (the silent pg_basebackup wait should never surprise anyone again), max_connections set to a sane value instead of 8000, more secondaries with load balancing work, and a fix to the abuse-reporting flow so an employee cannot be accidentally scheduled for hard deletion — the side door that helped start the load spike.

What this means for you

Enterprise network engineers

This is your case study, because its failures are the ordinary ones on any infrastructure team. Your equivalents of db1 and db2 are the core switch pair, the firewall cluster, the primary and standby anything — devices whose names differ by one character and whose terminal sessions sit in adjacent tabs at 2 a.m. Make production hosts look different (prompts, banners, colours), require the hostname to be spoken or pasted into the change ticket before a destructive step, and automate the wipe-and-rebuild procedures you currently do by hand. Then take the backup lesson whole: a backup is not the job that writes the file, it is the restore you have timed and tested. If your backup failure alert is an email, ask who has seen one this month — GitLab's were being rejected outright, and the silence looked exactly like success.

Hyperscaler engineers

Your version of the pg_dump failure is subtler and more common than it looks: a tool whose version, flags, or defaults depend on which machine it happens to run on. GitLab's backup broke because the packaging logic guessed the PostgreSQL version from a directory that did not exist on that host. Pin tool versions explicitly, run backup jobs from controlled images, and alert on the absence of a fresh backup artifact — not only on job errors, because the error channel itself failed here. And note the recovery bottleneck: the restore was limited by cheap staging disks at around 60 Mbps for 18 hours. Recovery time is set by the slowest storage you will actually restore from, so measure restores from the real tiers, not the fast ones.

HFT / low-latency engineers

Your databases are smaller and your windows are shorter, but the shape transfers directly: two near-identical hosts, a manual resync procedure, and a destructive command one character away from the wrong target. In a trading plant the equivalents are the primary and standby order gateways and the market-data handlers — and the control is the same one GitLab reached for: make the environment unmistakable before you restrict the command, and make recovery so practised that a mistake is an inconvenience instead of a day. Your journal is your WAL: if the change log is not archived off the box, a lagging standby has nothing to catch up from — exactly the gap that forced GitLab's manual rebuild in the first place.

Try it yourself

GitLab's silent backup failure was, at root, a configuration mismatch nobody compared: the backup host quietly ran different tooling from the database host. Packet Path's free Network Config Verifier builds exactly that habit — paste configurations from two devices side by side and it flags mismatches, inconsistencies, and risky leftovers. Run it on your primary and standby pair: if the two boxes that are supposed to match do not, you want to learn that from a comparison on a quiet afternoon, not from an empty backup bucket at midnight.

Sources

Primary source — GitLab, Postmortem of database outage of January 31 (published February 10, 2017). The database setup, the full timeline (17:20 snapshot, 19:00 load spike, 23:00 replication failure, the pg_basebackup and settings problems, the deletion at about 23:30 and the roughly 300 GB removed), the broken recovery procedures (pg_dump 9.2 versus PostgreSQL 9.6, the DMARC-rejected failure emails, Azure snapshots not enabled for database servers, the LVM snapshot position), the recovery (18-hour copy at around 60 Mbps, February 1 restoration times, webhook and sequence steps), the data-loss estimate (about 5,000 projects, 5,000 comments, 700 new user accounts), the 5 Whys root-cause analysis, and the improvement list are all taken from this postmortem: https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/

No other sources are used for facts on this page. Where GitLab's postmortem gives two framings of the loss window (17:20–00:00 in its introduction, 17:20–23:30 in its data-loss section), this page uses the 17:20–23:30 framing tied to the deletion time and the loss estimate, and says so here.

Key takeaways

Keep reading