Nelson

Nelson is an AI skill that provides automated system monitoring, alerting, and incident response coordination for software infrastructure. It covers metric collection configuration, alert rule design, escalation policy management, incident timeline tracking, and post-incident analysis that maintain system reliability and reduce mean time to resolution.

What Is This?

Overview

Nelson offers structured workflows for monitoring and responding to infrastructure issues across software systems. It handles configuring metric collection from application servers and infrastructure components, designing alert rules that catch real problems while minimizing false positives, building escalation policies that route alerts to the right responders, tracking incident timelines with automated status updates, coordinating response actions across multiple team members, and generating post-incident reports with root cause analysis and action items.

Who Should Use This

This skill serves site reliability engineers building monitoring infrastructure, on-call developers who need efficient incident response workflows, DevOps teams designing alerting strategies for production systems, and engineering managers establishing incident management processes.

Why Use It?

Problems It Solves

Alert fatigue from noisy monitoring systems causes responders to ignore warnings, including genuine incidents. Without structured escalation, critical alerts sit unacknowledged while the responsible engineer is unavailable. Incident response without coordination tools leads to duplicated effort and missed communication. Post-incident reviews that lack structured data produce vague action items that never get implemented.

Core Highlights

Intelligent alert rules reduce noise by correlating signals and suppressing transient spikes. Escalation chains ensure every alert reaches a responder within defined time windows. Timeline tracking creates an automatic record of incident progression for post-mortem analysis. Structured post-incident templates produce actionable improvement recommendations.

How to Use It?

Basic Usage

from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class AlertRule:
    name: str
    metric: str
    threshold: float
    duration_seconds: int
    severity: str
    team: str

class MonitoringConfig:
    def __init__(self):
        self.rules = []

    def add_rule(self, name, metric, threshold,
                 duration=300, severity="warning",
                 team="platform"):
        rule = AlertRule(name, metric, threshold,
                         duration, severity, team)
        self.rules.append(rule)
        return rule

    def standard_web_service(self, service_name):
        self.add_rule(
            f"{service_name}_error_rate",
            f"{service_name}.http.5xx_rate",
            threshold=0.05, severity="critical"
        )
        self.add_rule(
            f"{service_name}_latency_p99",
            f"{service_name}.http.latency_p99",
            threshold=2000, severity="warning"
        )
        self.add_rule(
            f"{service_name}_cpu",
            f"{service_name}.system.cpu_percent",
            threshold=85, severity="warning"
        )
        return self.rules

Real-World Examples

class IncidentTracker {
  constructor(incidentId, title, severity) {
    this.id = incidentId;
    this.title = title;
    this.severity = severity;
    this.timeline = [];
    this.responders = [];
    this.status = "active";
  }

  addEvent(description, author) {
    this.timeline.push({
      time: new Date().toISOString(),
      description,
      author,
    });
  }

  assignResponder(name, role) {
    this.responders.push({ name, role, assignedAt: new Date() });
    this.addEvent(`${name} assigned as ${role}`, "system");
  }

  resolve(resolution) {
    this.status = "resolved";
    this.addEvent(`Resolved: ${resolution}`, "system");
    this.resolvedAt = new Date();
  }

  generateReport() {
    const start = new Date(this.timeline[0].time);
    const end = this.resolvedAt || new Date();
    const duration = (end - start) / 60000;
    return {
      id: this.id,
      title: this.title,
      severity: this.severity,
      duration_minutes: Math.round(duration),
      responders: this.responders.map((r) => r.name),
      timeline_entries: this.timeline.length,
      status: this.status,
    };
  }
}

Advanced Tips

Set alert thresholds based on statistical analysis of normal behavior rather than arbitrary values. Group related alerts into a single incident to prevent notification storms during cascading failures. Automate the creation of incident channels and status pages when critical alerts fire to reduce response coordination overhead.

When to Use It?

Use Cases

Use Nelson when building monitoring and alerting for a new production service, when reducing alert fatigue from an existing noisy monitoring setup, when establishing on-call incident response processes for a growing team, or when improving post-incident analysis with structured review templates.

Related Topics

Prometheus and Grafana monitoring stacks, PagerDuty escalation configuration, SRE practices from Google, runbook automation, and blameless post-mortem culture complement incident monitoring.

Important Notes

Requirements

Metric collection infrastructure sending data from application and infrastructure components. An alerting platform that supports threshold rules and notification routing. On-call rotation schedules with defined escalation policies.

Usage Recommendations

Do: review and tune alert thresholds quarterly based on actual incident data and false positive rates. Include runbook links in alert notifications so responders have immediate access to resolution steps. Track mean time to acknowledge and mean time to resolve as key performance indicators.

Don't: alert on metrics without understanding their normal variation, as this produces false positives. Create alerts for every possible failure mode at once, since the resulting noise reduces response quality. Skip post-incident reviews for resolved incidents, because recurring patterns remain unaddressed.

Limitations

Threshold-based alerting cannot detect anomalies in metrics with highly variable baselines. Automated incident response covers known failure modes but cannot handle novel situations. Alert correlation across distributed systems requires significant configuration effort.