Skip to content
GraviPointThe Centre of Productivity
  • AI AgentNew
  • Pricing
  • Community
Log inGet started free
TemplatesProject & Task ManagementRisks, Issues & StatusIncident Postmortem

DocumentRisks, Issues & StatusFree

Incident Postmortem

A blameless post-incident review: impact, timeline, root cause, contributing factors and tracked action items.

Shown with sample data

Colour

Incident Postmortem: [Checkout failures after connection pool change]

Incident ID

[INC-2026-0412]

Severity

[SEV1]

Date

[Tuesday 17 November 2026]

Duration

[47 minutes of customer impact (14:08–14:55 UTC)]

Incident commander

[On-call engineering manager]

Author

[Name, Senior Site Reliability Engineer]

Status

[Final, reviewed 20 November 2026]

This is a blameless review. It describes what the system and processes allowed, not who is at fault.

1. Summary

[A configuration change intended for the staging environment reduced the database connection pool for the checkout service in production from 100 to 10 connections per instance. Under normal afternoon traffic the pool was exhausted, and 38% of checkout attempts failed with an error page for 47 minutes. The change was rolled back once the cause was identified. No data was lost; affected customers could retry successfully afterwards.]

2. Impact

Measure

Value

[Customer-facing impact]

[38% of checkout attempts failed (HTTP 502) from 14:08 to 14:55 UTC]

[Orders affected]

[About 2,140 failed checkout attempts by about 1,600 customers]

[Estimated lost revenue]

[About $41,000 (assuming 55% of failed customers did not return the same day)]

[Customer contacts]

[312 chats and calls; average wait rose to 18 minutes]

[SLA impact]

[Monthly checkout availability 99.89% against a 99.95% objective]

[Data impact]

[None: no partial orders or duplicate payments]

3. Timeline (UTC)

Time

Event

[13:52]

[Configuration change merged to the shared config repository; targeted at staging but the file path applied to all environments]

[14:05]

[Config sync rolls the change to production checkout instances]

[14:08]

[Checkout error rate begins to rise; first customer errors]

[14:12]

[Alert: checkout 5xx rate above 5% for 3 minutes; on-call SRE paged]

[14:15]

[On-call SRE acknowledges; sees database latency normal and suspects payment provider]

[14:21]

[Payment provider status page shows no issues; incident declared SEV1; incident commander assigned]

[14:26]

[Customer service reports a spike in contacts; status page updated: "Some customers cannot complete checkout"]

[14:34]

[Engineer notices "connection pool timeout" errors in checkout logs]

[14:41]

[Config history shows the pool size change at 13:52; decision to roll back]

[14:47]

[Rollback merged; config sync forced to production]

[14:55]

[Error rate back to baseline; customer impact ends]

[15:20]

[Monitoring confirms stability; incident resolved; status page updated]

4. Detection

[Detected by automated alert 4 minutes after impact began, which met our detection target of 5 minutes. However, the alert pointed to symptoms (5xx rate) without context, and there was no alert on connection pool saturation.]

5. Response

[Time to identify the cause was 26 minutes. Investigation first focused on the payment provider because a provider incident had caused similar symptoms in October. The config change was not visible in the deploy dashboard because config sync is not recorded as a deployment.]

6. Root cause

[The shared configuration repository applies a file to every environment unless an environment-specific override exists. The change set the pool size in the base file instead of the staging override. Configuration changes are synced automatically to production without the staged rollout, canary checks or approval that code deployments receive.]

7. Contributing factors

  • [Config changes do not appear in the deployment timeline, so responders could not see a recent change.]

  • [No alert or dashboard panel for connection pool utilisation on the checkout service.]

  • [The config repository’s review rule requires one approval, but reviewers cannot easily see which environments a change affects.]

  • [The checkout service fails requests immediately when the pool is exhausted rather than queueing briefly.]

8. What went well

  • [Alerting detected the problem within 4 minutes.]

  • [Customer service and the status page were updated within 18 minutes of impact.]

  • [Rollback was quick once the cause was found, and checkout recovered fully with no data clean-up.]

9. What went poorly

  • [Investigation followed the most recent similar incident rather than checking recent changes first.]

  • [It took 20 minutes to establish an incident commander and a single communication channel.]

10. Where we got lucky

  • [The change happened at 14:05 on a Tuesday. At evening peak or during the Black Friday sale, failures would likely have exceeded 80% of checkouts.]

  • [A responder happened to know the config repository well and checked its history manually.]

11. Action items

#

Action

Type

Priority

Owner

Ticket

Due

1

[Route config changes through the same staged rollout and canary analysis as code deploys]

[Prevent]

[P1]

[Platform team lead]

[PLAT-1182]

[15 Dec 2026]

2

[Show config changes in the deployment timeline and incident dashboard]

[Detect]

[P1]

[Developer tools lead]

[DEVX-431]

[4 Dec 2026]

3

[Add connection pool saturation alerts for all database clients]

[Detect]

[P1]

[SRE lead]

[SRE-902]

[27 Nov 2026]

4

[Config review tool shows a diff per environment; production changes need approval from a service owner]

[Prevent]

[P2]

[Platform team lead]

[PLAT-1183]

[15 Jan 2027]

5

[Add "check recent changes" as step one in the incident runbook]

[Mitigate]

[P2]

[Incident management lead]

[OPS-77]

[27 Nov 2026]

6

[Checkout service: short bounded queue for pool waits and clear error metrics]

[Mitigate]

[P3]

[Checkout team lead]

[CHK-2210]

[31 Jan 2027]

12. Lessons learned

  • [Configuration is code: it needs the same safety controls as code.]

  • [Responders need one place that shows every change to production.]

  • [Our incident commander handover needs practice; schedule a game day.]

13. Supporting information

  • [Incident channel export: [link]]

  • [Dashboards and graphs: [link]]

  • [Config change pull request: [link]]

14. Review

Role

Name

Date

Author

[Name]

[Date]

Incident commander

[Name]

[Date]

Engineering director

[Name]

[Date]

Text in [brackets] is yours to replace.

  • About
  • What’s included
  • How to use
  • Questions
  • Related resources

About this template

A blameless postmortem turns an outage into lasting improvement by asking how the system allowed the failure, not who caused it. This template records the summary and customer impact, a minute-by-minute timeline, detection and response, the root cause and contributing factors, what went well, what went badly and where you got lucky, and prioritised action items with owners and due dates, worked through for a fictional checkout outage caused by a configuration change.

Who it’s for

  • Production outages and major incidents (SEV1 and SEV2)
  • Security incidents after containment
  • Near misses worth learning from
  • Sharing learning across engineering teams

What’s included

  • 1. Summary
  • 2. Impact
  • 3. Timeline (UTC)
  • 4. Detection
  • 5. Response
  • 6. Root cause
  • 7. Contributing factors
  • 8. What went well
  • 9. What went poorly
  • 10. Where we got lucky

How to use it

  1. 1Start the document within 24 hours while memories and logs are fresh; assign one author.
  2. 2Build the timeline from chat logs, alerts and deploy records, in UTC.
  3. 3Describe impact in customer terms: who was affected, how, for how long, and the cost.
  4. 4Ask "why" repeatedly about the system and processes to find the root cause and contributing factors.
  5. 5Agree a small number of concrete, owned action items, review the document in a meeting and track the actions to completion.

Tips

  • Blameless does not mean no accountability. People are accountable for learning and fixing, not punished for mistakes the system made easy.
  • Write "the deploy pipeline allowed an unreviewed config change" rather than "Alex pushed a bad config". Name roles, not people, when describing actions.
  • Prefer fewer, high-impact actions that prevent a class of failure over a long list of small fixes that never get done.

Questions

What is a blameless postmortem?

A review that assumes people acted reasonably with the information and tools they had, and focuses on improving systems and processes. It encourages honest reporting, which is the only way to learn what really happened.

When should a postmortem be written?

For every incident above an agreed severity (typically customer-facing outages or data issues), and for near misses with high potential impact. Aim to publish within five working days.

What is the difference between a root cause and a contributing factor?

The root cause is the underlying condition without which the incident would not have happened. Contributing factors made it more likely, worse or longer, such as missing alerts or an unclear runbook.

Related searchesincident postmortem templatepost incident reviewblameless postmortemincident report templateoutage postmortemroot cause analysis it incident

Details

Format
Document
Category
Risks, Issues & Status
Setup
60 min
Level
Intermediate
Price
Free

Download a copy

Downloads need a free account.

DocumentRisks, Issues & StatusFree

Incident Postmortem

A blameless post-incident review: impact, timeline, root cause, contributing factors and tracked action items.

A free account keeps your copy safe so you can come back to it. Have one? Log in

Download a copy

Downloads need a free account.

Details

Format
Document
Category
Risks, Issues & Status
Setup
60 min
Level
Intermediate
Price
Free

Your copy is private to you.

More risks, issues & status templates

View all
Risk Register2 sheets

Open risks

7

High or critical

5

Risk reduced by plans

49%

ABCD
1RiskCategoryOwnerLikelihood
2Key supplier delivers hardware late, delaying installationSupplierTom4
3Lead engineer leaves mid-project, losing system knowledgePeopleAna2
4Scope creep from stakeholders delays launchScheduleAna4
5Data migration errors corrupt customer recordsTechnicalRaj3
Risks
SpreadsheetRisks, Issues & Status

Risk Register

Risks scored by likelihood and impact, before and after mitigation, with owners, review dates and a 5×5 heat map.

10 min setupPreview
RAID Log2 sheets

Open items

8

Open high priority

3

Overdue

4

ABCD
1TypeDescriptionPriorityOwner
2AssumptionClient provides anonymised test data by 15 NovHighRaj
3IssueStaging server out of disk space, blocking QAHighTom
4DependencyBrand guidelines from marketing team needed for designMediumLea
5RiskHoliday period reduces client availability for sign-offMediumAna
RAID
SpreadsheetRisks, Issues & Status

RAID Log

Risks, assumptions, issues and dependencies in one log, with owners, due dates, ageing and a summary by type.

10 min setupPreview

Project Status Report: Project Name

Reporting period: Date – Date · Project manager: Name · Sponsor: Name

OverallScheduleBudget
AmberAmberGreen

Summary

Three sentences: what happened this period, where we stand and what matters next.

Progress against milestones

MilestonePlannedForecast
Design sign-offNov 6Nov 13
Build completeDec 11Dec 11

Completed this period

DocumentRisks, Issues & Status

Project Status Report

A one-page weekly status update with RAG status, progress, risks and decisions needed.

20 min setupPreview

Practical guides and new resources, monthly.

One short email with new templates, calculators and how-tos. Unsubscribe any time.

GraviPointThe Centre of Productivity

Find the right resource. Make it work for you. Get things done with spreadsheets, documents, forms and calculators you can use right in your browser.

support@gravipoint.com
  • @gravipoint
  • @GraviPoint
  • GraviPoint
  • 𝕏@gravipoint

Templates

  • All templates
  • Budgets & expenses
  • Invoices & billing
  • Project management
  • HR & people
  • Inventory & stock
  • Sales & CRM
  • Financial calculators

Popular

  • Monthly budget
  • Invoice
  • Employee shift schedule
  • Weekly timesheet
  • Project plan & Gantt chart
  • Inventory tracker
  • Expense report
  • Mortgage calculator

Solutions

  • Individuals & households
  • Freelancers & consultants
  • Founders & startups
  • Small business owners
  • Teams & operations
  • Students & educators

Company

  • About
  • Features
  • Pricing
  • Community
  • Journal
  • Contact
© 2026 GraviPoint. All rights reserved.
  • Privacy
  • Terms
  • Help centre
  • FAQ