Observability & Monitoring Skill

Comprehensive frameworks for implementing observability including structured logging, metrics, distributed tracing, and alerting.

When to Use

• Setting up application monitoring
• Implementing structured logging
• Adding metrics and dashboards
• Configuring distributed tracing
• Creating alerting rules
• Debugging production issues

Three Pillars of Observability

┌─────────────────┬─────────────────┬─────────────────┐
│     LOGS        │     METRICS     │     TRACES      │
├─────────────────┼─────────────────┼─────────────────┤
│ What happened   │ How is system   │ How do requests │
│ at specific     │ performing      │ flow through    │
│ point in time   │ over time       │ services        │
└─────────────────┴─────────────────┴─────────────────┘

Structured Logging

Log Levels

Best Practice

// Good: Structured with context
logger.info('User action completed', {
  action: 'purchase',
  userId: user.id,
  orderId: order.id,
  duration_ms: 150
});

// Bad: String interpolation
logger.info(`User ${user.id} completed purchase`);

See templates/structured-logging.ts for Winston setup and request middleware

Metrics Collection

RED Method (Rate, Errors, Duration)

Essential metrics for any service:

• Rate - Requests per second
• Errors - Failed requests per second
• Duration - Request latency distribution

Prometheus Buckets

// HTTP request latency
buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5]

// Database query latency
buckets: [0.001, 0.01, 0.05, 0.1, 0.5, 1]

See templates/prometheus-metrics.ts for full metrics configuration

Distributed Tracing

OpenTelemetry Setup

Auto-instrument common libraries:

• Express/HTTP
• PostgreSQL
• Redis

Manual Spans

tracer.startActiveSpan('processOrder', async (span) => {
  span.setAttribute('order.id', orderId);
  // ... work
  span.end();
});

See templates/opentelemetry-tracing.ts for full setup

Alerting Strategy

Severity Levels

Key Alerts

See templates/alerting-rules.yml for Prometheus alerting rules

Health Checks

Kubernetes Probes

Readiness Response

{
  "status": "healthy|degraded|unhealthy",
  "checks": {
    "database": { "status": "pass", "latency_ms": 5 },
    "redis": { "status": "pass", "latency_ms": 2 }
  },
  "version": "1.0.0",
  "uptime": 3600
}

See templates/health-checks.ts for implementation

Observability Checklist

Implementation

• JSON structured logging
• Request correlation IDs
• RED metrics (Rate, Errors, Duration)
• Business metrics
• Distributed tracing
• Health check endpoints

Alerting

• Service outage alerts
• Error rate thresholds
• Latency thresholds
• Resource utilization alerts

Dashboards

• Service overview
• Error analysis
• Performance metrics

Extended Thinking Triggers

Use Opus 4.5 extended thinking for:

• Incident investigation - Correlating logs, metrics, traces
• Alert tuning - Reducing noise, catching real issues
• Architecture decisions - Choosing monitoring solutions
• Performance debugging - Cross-service latency analysis

Templates Reference