Observability Designer

Observability Designer is a community skill for architecting comprehensive observability systems, covering telemetry pipeline design, distributed tracing, metric correlation, log structuring, and service level objective definition for production system visibility.

What Is This?

Overview

Observability Designer provides tools for planning and implementing observability architectures across distributed systems. It covers telemetry pipeline design that routes metrics, traces, and logs through collection and storage backends, distributed tracing that tracks request flow across service boundaries with span correlation, metric correlation that connects related measurements across services to identify failure patterns, log structuring that defines consistent logging formats and contextual fields for searchable log data, and service level objective definition that establishes measurable reliability targets with error budget tracking. The skill enables teams to design effective observability for complex systems.

Who Should Use This

This skill serves platform engineers designing observability infrastructure, SRE teams establishing monitoring standards for distributed services, and architects planning telemetry strategies for microservice deployments.

Why Use It?

Problems It Solves

Distributed systems generate telemetry across many services without a unified strategy for collection and correlation. Unstructured logs from different services cannot be searched or correlated effectively during incident investigation. Missing distributed traces make it impossible to follow request paths across service boundaries. Reliability goals without measurable SLOs lack the precision needed for data-driven decisions.

Core Highlights

Telemetry architect designs collection pipelines for metrics, traces, and logs. Trace designer plans distributed tracing instrumentation with context propagation. Correlation engine links related signals across telemetry types. SLO builder defines service objectives with error budget calculations.

How to Use It?

Basic Usage

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 1024
    timeout: 5s
  attributes:
    actions:
      - key: environment
        value: production
        action: upsert

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  jaeger:
    endpoint:
      jaeger:14250
  loki:
    endpoint:
      http://loki:3100
      /loki/api/v1/push

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters:
        [prometheus]
    traces:
      receivers: [otlp]
      processors:
        [batch,
         attributes]
      exporters:
        [jaeger]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

Real-World Examples

from dataclasses import (
  dataclass)

@dataclass
class SLODefinition:
  name: str
  target: float
  window_days: int = 30

class SLOTracker:
  def __init__(
    self,
    slo: SLODefinition
  ):
    self.slo = slo
    self.total = 0
    self.good = 0

  def record(
    self,
    total: int,
    good: int
  ):
    self.total += total
    self.good += good

  @property
  def current(self)\
      -> float:
    if self.total == 0:
      return 1.0
    return (
      self.good
      / self.total)

  @property
  def error_budget(
    self
  ) -> float:
    allowed = (
      1 - self.slo.target)
    consumed = (
      1 - self.current)
    if allowed == 0:
      return 0.0
    return max(0, 1 - (
      consumed / allowed))

  def report(self) -> dict:
    return {
      'slo': self.slo.name,
      'target':
        self.slo.target,
      'current':
        round(
          self.current, 5),
      'budget_remaining':
        round(self
          .error_budget
          * 100, 2)}

Advanced Tips

Use OpenTelemetry as the instrumentation standard to avoid vendor lock-in and enable flexible backend switching. Design correlation identifiers that propagate across metrics, traces, and logs to enable unified investigation during incidents. Implement tiered storage retention policies that keep recent telemetry at full resolution while downsampling older data.

When to Use It?

Use Cases

Design an OpenTelemetry collection pipeline that routes metrics to Prometheus, traces to Jaeger, and logs to Loki. Define SLOs for critical services with error budget tracking and burn rate alerting. Plan structured logging standards that enable correlation with distributed traces across service boundaries.

Related Topics

Observability, OpenTelemetry, distributed tracing, SLOs, telemetry pipelines, structured logging, and site reliability engineering.

Important Notes

Requirements

OpenTelemetry collector or compatible telemetry routing infrastructure. Storage backends for metrics, traces, and logs such as Prometheus, Jaeger, and Loki. Instrumentation libraries for application services in the target languages.

Usage Recommendations

Do: instrument all service boundaries including external API calls, database queries, and inter-service communication. Include trace context in structured log entries to enable correlation between signals. Define SLOs collaboratively between engineering and product teams to align reliability targets with business impact.

Don't: collect every possible metric since high-cardinality telemetry creates storage and query performance problems. Implement distributed tracing without context propagation standards since disconnected spans provide limited value. Set SLO targets at one hundred percent since this leaves zero error budget and blocks all changes.

Limitations

Full observability instrumentation adds overhead to service latency and resource consumption from telemetry generation. Trace sampling reduces cost but also reduces visibility into infrequent error patterns that sampling may miss. Correlating signals across different storage backends requires consistent identifier propagation that all services must implement correctly.