Monitoring Operations

Monitoring Operations is a community skill for managing day-to-day operational monitoring workflows, covering alert triage, incident response coordination, runbook automation, on-call management, and post-incident review for production system reliability.

What Is This?

Overview

Monitoring Operations provides tools for running effective operational monitoring programs. It covers alert triage that classifies incoming alerts by severity and routes them to appropriate responders based on service ownership and escalation rules, incident response coordination that manages communication and task assignment during active production incidents across teams, runbook automation that documents and automates standard remediation procedures for known operational issues, on-call management that schedules rotation coverage and defines escalation paths for different service tiers, and post-incident review that conducts structured analysis of resolved incidents to identify systemic improvements. The skill enables operations teams to respond efficiently to production events.

Who Should Use This

This skill serves operations engineers handling production incidents, SRE teams building on-call processes and runbooks, and engineering managers establishing incident response procedures for their organizations.

Why Use It?

Problems It Solves

Unstructured incident response leads to confusion about who is handling what during active outages. Alert storms from cascading failures overwhelm on-call engineers with redundant notifications. Remediation knowledge stays locked in individual engineers heads instead of documented runbooks. Inconsistent post-incident reviews miss systemic patterns that cause repeated similar failures across the organization.

Core Highlights

Alert router classifies and directs notifications based on severity and service ownership mappings. Incident coordinator manages roles, communication channels, and task tracking during active incidents. Runbook engine stores and executes documented remediation procedures for common failure scenarios. Review facilitator structures post-incident analysis with timeline reconstruction and action items.

How to Use It?

Basic Usage

incident_management:
  severity_levels:
    - name: SEV1
      description: >
        Customer-facing outage
      response_time: 5m
      escalation:
        - on-call-primary
        - on-call-secondary
        - engineering-manager
      channels:
        - '#incident-war-room'
        - pagerduty

    - name: SEV2
      description: >
        Degraded performance
      response_time: 15m
      escalation:
        - on-call-primary
      channels:
        - '#ops-alerts'

    - name: SEV3
      description: >
        Non-urgent issue
      response_time: 4h
      escalation:
        - team-queue
      channels:
        - '#ops-tickets'

  on_call:
    rotation: weekly
    handoff_day: monday
    handoff_time: '09:00'
    timezone: UTC
    override_allowed: true

Real-World Examples

import subprocess
import logging

logger = logging.getLogger(
  'runbook')

class RunbookExecutor:
  def __init__(self):
    self.runbooks = {}

  def register(
    self,
    alert_name: str,
    steps: list[dict]
  ):
    self.runbooks[
      alert_name] = steps

  def execute(
    self,
    alert_name: str,
    context: dict
  ) -> dict:
    steps = self.runbooks\
      .get(alert_name, [])
    results = []
    for step in steps:
      logger.info(
        f'Running: '
        f'{step["name"]}')
      cmd = step['command']\
        .format(**context)
      result = subprocess\
        .run(
          cmd, shell=True,
          capture_output=True,
          text=True,
          timeout=step.get(
            'timeout', 60))
      results.append({
        'step': step['name'],
        'exit_code':
          result.returncode,
        'output':
          result.stdout})
      if (result.returncode
          != 0
          and step.get(
            'required',
            True)):
        logger.error(
          f'Failed: '
          f'{step["name"]}')
        break
    return {
      'alert': alert_name,
      'steps_run':
        len(results),
      'results': results}

executor = RunbookExecutor()
executor.register(
  'HighMemoryUsage',
  [{'name': 'check_procs',
    'command':
      'ps aux --sort=-rss'
      ' | head -10',
    'timeout': 30},
   {'name': 'clear_cache',
    'command':
      'sync && echo 3 >'
      ' /proc/sys/vm/'
      'drop_caches',
    'required': False}])

Advanced Tips

Implement alert correlation rules that group related notifications from cascading failures into a single incident to reduce noise during outages. Define clear escalation timeouts that automatically promote unacknowledged alerts to prevent incidents from being missed. Maintain runbooks alongside the services they support using version control to keep remediation procedures current with system changes.

When to Use It?

Use Cases

Build an incident response workflow with severity classification and automatic escalation for a production platform. Create automated runbooks for common operational issues like disk pressure and memory exhaustion. Establish on-call rotation schedules with handoff procedures and coverage documentation.

Related Topics

Incident management, on-call operations, runbook automation, alert triage, SRE practices, post-incident review, and operational reliability.

Important Notes

Requirements

Incident management platform such as PagerDuty or Opsgenie for alert routing and on-call scheduling. Communication tools like Slack or Teams for incident coordination channels. Documentation system for storing runbooks and post-incident reports.

Usage Recommendations

Do: test runbooks regularly in non-production environments to verify remediation steps remain valid. Rotate on-call responsibilities across team members to distribute operational knowledge and prevent burnout. Write blameless post-incident reviews that focus on systemic improvements rather than individual actions.

Don't: automate remediation steps that modify production state without human approval gates for critical operations. Skip post-incident reviews for lower-severity incidents since they often reveal patterns that prevent larger failures. Let runbooks become stale by only updating them during incident response rather than scheduling regular reviews.

Limitations

Automated runbook execution carries risk when remediation commands interact with production data in unexpected states. Alert correlation accuracy depends on consistent labeling across all services that generate monitoring notifications. On-call scheduling tools require integration with multiple communication platforms to reach responders reliably across different channels and time zones.