Hands-on tutorial
Tutorials · Monitoring
How to Handle a Major Network Outage: A NOC Runbook
When a site or service goes down, the NOC must restore it fast and keep everyone informed. This runbook covers detection, the first five minutes, isolating the fault layer by layer, communication and the post-incident review.
Before you start
- Monitoring of gateways, switches and access points (for example NavCloud with alerts enabled)
- Out-of-band or cellular access to critical sites
- A contact list for internet providers and site staff
Step by step
Confirm the outage
Check from two sources: the monitoring alert and a real test (ping, user report, failed transaction). Rule out a monitoring problem first.
Declare and assign roles
Name an incident lead, a technical lead and a communicator. One person talks to the business; the others fix.
Find the scope
One device, one site, one region or everything? NavCloud's site map shows which gateways, switches and APs are offline at the same time.
Work up the layers
Power and physical links, then the uplink and provider, then routing and VPN tunnels, then DNS and DHCP, then the application. Stop at the first broken layer.
Restore service before root cause
Fail over to the backup link, roll back the last change or reboot the faulty device. Investigation can continue after users are working again.
Communicate on a fixed rhythm
Send short updates every 30 minutes: what is affected, what is being done, next update time. Say 'unknown' instead of guessing.
Close with a review
Within five working days write a blameless review: timeline, root cause, what helped, what to change. Track the actions to completion.
Pro tips
- Most outages follow a change: check the change calendar first.
- Keep a printed copy of the runbook and contact list for when the network itself is down.
FAQ
Frequently asked questions
What is MTTR?
Mean time to restore: the average time from the start of an outage to service being back. Good runbooks reduce it.
Who decides to fail over?
The incident lead, based on the runbook's criteria, so the decision is not delayed.
Keep learning
Related guides
Monitor WAN latency and packet loss
Read more →Network alerts by email and webhook
Read more →Uptime monitoring and status page
Read more →How to Monitor Your Network With SNMP and Syslog
Know about failures before users call: collect interface statistics with SNMPv3 and central logs with syslog.
Read more →How to Synchronise Time on Network Devices With NTP
Accurate clocks are essential for logs, certificates, schedules and CCTV evidence. Configure NTP everywhere in minutes.
Read more →How to Back Up and Restore Network Device Configurations
A configuration backup turns a failed device into a quick swap. Set up regular backups and test a restore.
Read more →Stuck on a step?
Ask Navijin technical support or an authorised partner for help with your configuration.