Incident Postmortem: [Checkout failures after connection pool change]
Incident ID | [INC-2026-0412] |
Severity | [SEV1] |
Date | [Tuesday 17 November 2026] |
Duration | [47 minutes of customer impact (14:08–14:55 UTC)] |
Incident commander | [On-call engineering manager] |
Author | [Name, Senior Site Reliability Engineer] |
Status | [Final, reviewed 20 November 2026] |
This is a blameless review. It describes what the system and processes allowed, not who is at fault.
1. Summary
[A configuration change intended for the staging environment reduced the database connection pool for the checkout service in production from 100 to 10 connections per instance. Under normal afternoon traffic the pool was exhausted, and 38% of checkout attempts failed with an error page for 47 minutes. The change was rolled back once the cause was identified. No data was lost; affected customers could retry successfully afterwards.]
2. Impact
Measure | Value |
|---|---|
[Customer-facing impact] | [38% of checkout attempts failed (HTTP 502) from 14:08 to 14:55 UTC] |
[Orders affected] | [About 2,140 failed checkout attempts by about 1,600 customers] |
[Estimated lost revenue] | [About $41,000 (assuming 55% of failed customers did not return the same day)] |
[Customer contacts] | [312 chats and calls; average wait rose to 18 minutes] |
[SLA impact] | [Monthly checkout availability 99.89% against a 99.95% objective] |
[Data impact] | [None: no partial orders or duplicate payments] |
3. Timeline (UTC)
Time | Event |
|---|---|
[13:52] | [Configuration change merged to the shared config repository; targeted at staging but the file path applied to all environments] |
[14:05] | [Config sync rolls the change to production checkout instances] |
[14:08] | [Checkout error rate begins to rise; first customer errors] |
[14:12] | [Alert: checkout 5xx rate above 5% for 3 minutes; on-call SRE paged] |
[14:15] | [On-call SRE acknowledges; sees database latency normal and suspects payment provider] |
[14:21] | [Payment provider status page shows no issues; incident declared SEV1; incident commander assigned] |
[14:26] | [Customer service reports a spike in contacts; status page updated: "Some customers cannot complete checkout"] |
[14:34] | [Engineer notices "connection pool timeout" errors in checkout logs] |
[14:41] | [Config history shows the pool size change at 13:52; decision to roll back] |
[14:47] | [Rollback merged; config sync forced to production] |
[14:55] | [Error rate back to baseline; customer impact ends] |
[15:20] | [Monitoring confirms stability; incident resolved; status page updated] |
4. Detection
[Detected by automated alert 4 minutes after impact began, which met our detection target of 5 minutes. However, the alert pointed to symptoms (5xx rate) without context, and there was no alert on connection pool saturation.]
5. Response
[Time to identify the cause was 26 minutes. Investigation first focused on the payment provider because a provider incident had caused similar symptoms in October. The config change was not visible in the deploy dashboard because config sync is not recorded as a deployment.]
6. Root cause
[The shared configuration repository applies a file to every environment unless an environment-specific override exists. The change set the pool size in the base file instead of the staging override. Configuration changes are synced automatically to production without the staged rollout, canary checks or approval that code deployments receive.]
7. Contributing factors
[Config changes do not appear in the deployment timeline, so responders could not see a recent change.]
[No alert or dashboard panel for connection pool utilisation on the checkout service.]
[The config repository’s review rule requires one approval, but reviewers cannot easily see which environments a change affects.]
[The checkout service fails requests immediately when the pool is exhausted rather than queueing briefly.]
8. What went well
[Alerting detected the problem within 4 minutes.]
[Customer service and the status page were updated within 18 minutes of impact.]
[Rollback was quick once the cause was found, and checkout recovered fully with no data clean-up.]
9. What went poorly
[Investigation followed the most recent similar incident rather than checking recent changes first.]
[It took 20 minutes to establish an incident commander and a single communication channel.]
10. Where we got lucky
[The change happened at 14:05 on a Tuesday. At evening peak or during the Black Friday sale, failures would likely have exceeded 80% of checkouts.]
[A responder happened to know the config repository well and checked its history manually.]
11. Action items
# | Action | Type | Priority | Owner | Ticket | Due |
|---|---|---|---|---|---|---|
1 | [Route config changes through the same staged rollout and canary analysis as code deploys] | [Prevent] | [P1] | [Platform team lead] | [PLAT-1182] | [15 Dec 2026] |
2 | [Show config changes in the deployment timeline and incident dashboard] | [Detect] | [P1] | [Developer tools lead] | [DEVX-431] | [4 Dec 2026] |
3 | [Add connection pool saturation alerts for all database clients] | [Detect] | [P1] | [SRE lead] | [SRE-902] | [27 Nov 2026] |
4 | [Config review tool shows a diff per environment; production changes need approval from a service owner] | [Prevent] | [P2] | [Platform team lead] | [PLAT-1183] | [15 Jan 2027] |
5 | [Add "check recent changes" as step one in the incident runbook] | [Mitigate] | [P2] | [Incident management lead] | [OPS-77] | [27 Nov 2026] |
6 | [Checkout service: short bounded queue for pool waits and clear error metrics] | [Mitigate] | [P3] | [Checkout team lead] | [CHK-2210] | [31 Jan 2027] |
12. Lessons learned
[Configuration is code: it needs the same safety controls as code.]
[Responders need one place that shows every change to production.]
[Our incident commander handover needs practice; schedule a game day.]
13. Supporting information
[Incident channel export: [link]]
[Dashboards and graphs: [link]]
[Config change pull request: [link]]
14. Review
Role | Name | Date |
|---|---|---|
Author | [Name] | [Date] |
Incident commander | [Name] | [Date] |
Engineering director | [Name] | [Date] |