Hands-on tutorial

Tutorials · Monitoring

How to Handle a Major Network Outage: A NOC Runbook

When a site or service goes down, the NOC must restore it fast and keep everyone informed. This runbook covers detection, the first five minutes, isolating the fault layer by layer, communication and the post-incident review.

Intermediate15 min7 steps

Before you start

  • Monitoring of gateways, switches and access points (for example NavCloud with alerts enabled)
  • Out-of-band or cellular access to critical sites
  • A contact list for internet providers and site staff

Step by step

  1. Confirm the outage

    Check from two sources: the monitoring alert and a real test (ping, user report, failed transaction). Rule out a monitoring problem first.

  2. Declare and assign roles

    Name an incident lead, a technical lead and a communicator. One person talks to the business; the others fix.

  3. Find the scope

    One device, one site, one region or everything? NavCloud's site map shows which gateways, switches and APs are offline at the same time.

  4. Work up the layers

    Power and physical links, then the uplink and provider, then routing and VPN tunnels, then DNS and DHCP, then the application. Stop at the first broken layer.

  5. Restore service before root cause

    Fail over to the backup link, roll back the last change or reboot the faulty device. Investigation can continue after users are working again.

  6. Communicate on a fixed rhythm

    Send short updates every 30 minutes: what is affected, what is being done, next update time. Say 'unknown' instead of guessing.

  7. Close with a review

    Within five working days write a blameless review: timeline, root cause, what helped, what to change. Track the actions to completion.

Pro tips

  • Most outages follow a change: check the change calendar first.
  • Keep a printed copy of the runbook and contact list for when the network itself is down.