librefang-registry/hands/devops/SKILL.md

name, version, description, runtime

name	version	description	runtime
devops-hand-skill	1.0.0	Expert knowledge for AI DevOps automation -- CI/CD patterns, infrastructure monitoring, deployment strategies, and incident response playbooks	prompt_only

DevOps Expert Knowledge

CI/CD Pipeline Patterns

GitHub Actions Reference

Basic workflow structure:

name: CI/CD Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: make build
      - name: Test
        run: make test
      - name: Lint
        run: make lint

  deploy:
    needs: build
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Deploy
        run: make deploy

Useful API endpoints:

# List workflow runs
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \
  "https://api.github.com/repos/OWNER/REPO/actions/runs?per_page=10"

# Get workflow run details
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" \
  "https://api.github.com/repos/OWNER/REPO/actions/runs/RUN_ID"

# Re-run failed jobs
curl -s -X POST -H "Authorization: Bearer $GITHUB_TOKEN" \
  "https://api.github.com/repos/OWNER/REPO/actions/runs/RUN_ID/rerun-failed-jobs"

Pipeline Optimization Checklist

• Cache dependencies (node_modules, .cargo, pip cache)
• Parallelize independent jobs
• Use matrix builds for multi-version testing
• Skip unnecessary steps on non-code changes
• Use shallow clones for faster checkout
• Optimize Docker layer caching
• Run expensive tests only on main branch

---

Infrastructure Monitoring

Health Check Patterns

HTTP endpoint check:

curl -s -o /dev/null -w "%{http_code} %{time_total}s" --max-time 10 "$URL"

TCP port check:

nc -z -w5 hostname port && echo "UP" || echo "DOWN"

SSL certificate expiry:

echo | openssl s_client -servername HOST -connect HOST:443 2>/dev/null | \
  openssl x509 -noout -dates

DNS resolution:

dig +short hostname

Disk usage:

df -h | grep -v tmpfs

Memory usage:

free -h

The Four Golden Signals

Signal	What to Measure	Alert Threshold
Latency	Request duration	P95 > 500ms
Traffic	Requests per second	Deviation > 50% from baseline
Errors	Error rate percentage	> 1% of requests
Saturation	Resource utilization	CPU/Memory > 80%

Docker Monitoring Commands

# Container status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Resource usage
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"

# Container logs (last 100 lines)
docker logs --tail 100 CONTAINER_NAME

# Inspect container health
docker inspect --format='{{.State.Health.Status}}' CONTAINER_NAME

Kubernetes Monitoring Commands

# Pod status across all namespaces
kubectl get pods --all-namespaces -o wide

# Resource usage
kubectl top pods --all-namespaces
kubectl top nodes

# Recent events (errors and warnings)
kubectl get events --sort-by=.lastTimestamp --field-selector type!=Normal

# Pod logs
kubectl logs POD_NAME -n NAMESPACE --tail=100

# Describe failing pod
kubectl describe pod POD_NAME -n NAMESPACE

---

Deployment Strategies

Blue-Green Deployment

1. Run current version on "Blue" environment
2. Deploy new version to "Green" environment
3. Run health checks on Green
4. Switch traffic from Blue to Green
5. Keep Blue as rollback target
6. After validation period, decommission Blue

Canary Deployment

1. Deploy new version to small subset (5-10% of traffic)
2. Monitor error rates and latency
3. If healthy, gradually increase traffic (25% -> 50% -> 100%)
4. If problems detected, route all traffic back to old version

Rolling Update

1. Update instances one at a time
2. Wait for health check to pass before updating next
3. If any instance fails health check, pause and alert
4. Continue until all instances updated

Deployment Checklist

• All tests passing in CI
• Database migrations compatible (backward and forward)
• Feature flags configured for new features
• Monitoring and alerting in place
• Rollback procedure documented and tested
• On-call engineer notified
• Change request approved (if required)

---

Incident Response

Incident Lifecycle

Detection -> Triage -> Mitigation -> Investigation -> Resolution -> Post-mortem

Severity Levels

Level	Impact	Response Time	Example
SEV1	Full outage	Immediate	Production down
SEV2	Major impact	15 min	Core feature broken
SEV3	Minor impact	1 hour	Non-critical feature degraded
SEV4	Low impact	Next business day	Cosmetic issue

Incident Response Template

# Incident Report: [Title]
**Severity**: SEV[1-4]
**Status**: [Investigating | Mitigated | Resolved]
**Duration**: [Start time] - [End time]

## Timeline
- HH:MM - [Event or action taken]
- HH:MM - [Event or action taken]

## Root Cause
[What caused the incident]

## Impact
[Who was affected and how]

## Mitigation
[What was done to restore service]

## Resolution
[What was done to fix the root cause]

## Action Items
- [ ] [Preventive measure 1]
- [ ] [Preventive measure 2]

## Lessons Learned
[What we can improve]

---

Infrastructure as Code

Terraform Quick Reference

# Initialize
terraform init

# Plan changes
terraform plan -out=tfplan

# Apply changes
terraform apply tfplan

# Show current state
terraform show

# Destroy resources (DANGEROUS)
terraform destroy

Docker Compose Quick Reference

# Start services
docker compose up -d

# Stop services
docker compose down

# View logs
docker compose logs -f SERVICE_NAME

# Rebuild and restart
docker compose up -d --build SERVICE_NAME

# Scale a service
docker compose up -d --scale SERVICE_NAME=3

---

Common Failure Diagnosis Playbooks

Memory Leak Detection

# Track memory growth over time
while true; do
  ps aux --sort=-%mem | head -5 | awk '{print strftime("%H:%M:%S"), $2, $4"%", $11}'
  sleep 60
done

# Check for OOM kills
dmesg | grep -i "oom\|killed" | tail -20

# Kubernetes memory pressure
kubectl top pods --sort-by=memory | head -10

DNS Failure Cascade

# Test DNS resolution
dig hostname +short
dig @8.8.8.8 hostname +short  # Bypass local DNS

# Check /etc/resolv.conf
cat /etc/resolv.conf

# Test from inside a container
kubectl exec -it POD -- nslookup hostname

Database Connection Pool Exhaustion

# Check active connections (PostgreSQL)
psql -c "SELECT count(*) FROM pg_stat_activity WHERE state = 'active';"
psql -c "SELECT max_conn FROM pg_settings WHERE name = 'max_connections';"

# Check for long-running queries
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
         FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 10;"

Certificate Expiry Monitoring

# Check cert expiry for a list of domains
for domain in api.example.com app.example.com; do
  expiry=$(echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | \
    openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
  echo "$domain: $expiry"
done

---

Worked Examples

Example 1: Zero-Downtime Deployment Pipeline

Full lifecycle from code merge to production traffic switch with rollback safety.

Phases: CI build and push image -> deploy to Green -> health-gate -> traffic switch -> monitor -> done (or rollback).

Traffic switch script (the critical step):

#!/bin/bash
set -euo pipefail
# Record current slot for rollback, then switch
CURRENT=$(kubectl get svc myapp-active -n production -o jsonpath='{.spec.selector.slot}')
echo "$CURRENT" > /tmp/rollback-slot
kubectl patch svc myapp-active -n production -p '{"spec":{"selector":{"slot":"green"}}}'
# Verify
sleep 5 && curl -sf https://api.example.com/api/health | jq '.version'

Rollback: read /tmp/rollback-slot, patch the service selector back, verify health.

Decision flowchart:

Code merged to main
  |
  v
CI build + test ----[FAIL]----> Block merge, notify author
  |
  [PASS]
  v
Deploy to Green
  |
  v
Health checks ------[FAIL]----> Alert on-call, keep Blue active
  |
  [PASS]
  v
Switch traffic to Green
  |
  v
Monitor 15 min -----[ERROR SPIKE]----> Rollback to Blue, open incident
  |
  [STABLE]
  v
Mark Green as new Blue, done

---

Example 2: Production Incident Response

Walkthrough of a real-world SEV1 incident: API latency spike caused by a database connection pool exhaustion.

Timeline

Time (UTC)	Event	Actor
14:02	PagerDuty alert: P95 latency > 2s on /api/orders	Monitoring
14:04	On-call acknowledges, opens incident channel #inc-2025-0312	On-call engineer
14:06	Check dashboard: request queue depth spiking, error rate at 12%	On-call engineer
14:10	Identify DB connection pool at 100% utilization	On-call engineer
14:12	Find long-running query from analytics job (started 13:55)	On-call engineer
14:14	Kill the runaway query, pool starts draining	On-call engineer
14:18	Latency returns to normal, error rate drops to 0.2%	Monitoring
14:20	Incident mitigated, continue monitoring	On-call engineer
14:45	Root cause confirmed: analytics cron job without query timeout	Investigation
15:00	Incident resolved, post-mortem scheduled	Incident commander

Detection -- Alerting rules that fired

# Prometheus alerting rule
groups:
  - name: api-latency
    rules:
      - alert: HighAPILatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="api"}[5m])) > 2
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "P95 latency above 2s for 2+ minutes"
          runbook: "https://wiki.internal/runbooks/high-latency"

Triage -- Quick diagnosis commands

# 1. Check if it's a specific endpoint or global
curl -s "http://prometheus:9090/api/v1/query?query=topk(5,rate(http_request_duration_seconds_sum[5m])/rate(http_request_duration_seconds_count[5m]))" | jq '.data.result[] | {endpoint: .metric.handler, avg_latency: .value[1]}'

# 2. Check database connection pool
psql -c "SELECT count(*) as total, state FROM pg_stat_activity GROUP BY state;"

# 3. Find the blocking query
psql -c "SELECT pid, now() - query_start AS duration, query
         FROM pg_stat_activity
         WHERE state = 'active' AND now() - query_start > interval '1 minute'
         ORDER BY duration DESC LIMIT 5;"

Mitigation -- Kill the offending query

# Kill the long-running query by PID
psql -c "SELECT pg_terminate_backend(12345);"

# Verify pool is recovering
watch -n 2 'psql -t -c "SELECT count(*) FROM pg_stat_activity WHERE state = '\''active'\'';"'

Prevention -- Fix applied after incident

-- Set statement timeout for analytics role
ALTER ROLE analytics_readonly SET statement_timeout = '300s';

# Add connection pool monitoring alert
- alert: DBConnectionPoolNearCapacity
  expr: pg_stat_activity_count / pg_settings_max_connections > 0.8
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "DB connection pool above 80% capacity"

Post-mortem action items:

• Add statement_timeout to all non-interactive database roles
• Add connection pool utilization alerts (threshold: 80%)
• Move analytics queries to read replica
• Add circuit breaker to API when pool utilization exceeds 90%

---

Example 3: Infrastructure Scaling Event

Scaling a Kubernetes deployment in response to sustained load increase.

Phase 1 -- Alert triggers

Alert: HighCPUUtilization
  Condition: avg(cpu_usage) > 80% for 10 minutes
  Current:   87% across 3 pods
  Namespace: production
  Deployment: order-service

Phase 2 -- Capacity analysis

kubectl top pods -l app=order-service -n production   # Per-pod CPU/memory
kubectl describe nodes | grep -A 5 "Allocated resources"  # Node headroom
kubectl get hpa order-service -n production            # Current HPA state
# Result: 87% CPU across 3 pods, 842 req/s (2x baseline)

Phase 3 -- Scaling decision matrix

Metric	Current	Target	Action
CPU usage	87%	< 70%	Scale out
Request rate	842/s	-	2x normal, sustained
Memory	258Mi avg	512Mi limit	Headroom OK
Pod count	3	6 (estimated)	Double replicas
Node capacity	72%	< 85%	Sufficient for 6 pods

Phase 4 -- Implement scaling

# Option A: Manual scale (immediate)
kubectl scale deployment order-service -n production --replicas=6

# Option B: Adjust HPA for sustained load (preferred)
kubectl patch hpa order-service -n production \
  -p '{"spec":{"minReplicas":5,"maxReplicas":15}}'

# Monitor rollout
kubectl rollout status deployment/order-service -n production

# Watch pods come up
kubectl get pods -l app=order-service -n production -w

Phase 5 -- Verify scaling

Confirm via kubectl top pods (CPU should drop to ~45% per pod), check P95 latency is back below SLO, and verify error rate < 1%.

Post-scaling actions:

• Investigate root cause of traffic increase (marketing event? bot traffic? organic growth?)
• Update capacity planning spreadsheet
• If sustained, adjust resource requests/limits and HPA baselines
• Set calendar reminder to review and potentially scale down in 48h

---

Observability Deep Dive

Structured Logging

Use consistent JSON log format across all services for machine-parseable aggregation.

Log format standard: JSON with required fields: timestamp, level, service, trace_id, span_id, request_id, message. Add error and context (structured key-value) as needed.

Log levels -- when to use each:

Level	Purpose	Example	Persisted
error	Requires human attention	Payment processing failed	90 days
warn	Degraded but recoverable	Retry succeeded on 2nd attempt	30 days
info	Business-significant events	Order placed, user logged in	14 days
debug	Developer troubleshooting	Cache hit/miss, query timing	3 days
trace	Fine-grained flow tracking	Function entry/exit, variable state	1 day (sampled)

Correlation IDs: generate X-Request-ID at API gateway, propagate through all downstream calls. Query across services by filtering on request_id.keyword in Elasticsearch/OpenSearch.

Distributed Tracing

Core concepts: A Trace is the end-to-end request path. Each service call is a Span with timing. Spans nest to show the call tree (e.g., API Gateway -> Order Service -> DB Query + Payment Service -> Stripe API).

OpenTelemetry propagation: traceparent: 00-<trace-id>-<span-id>-<flags>, tracestate: vendor=value.

Useful trace queries (Jaeger/Tempo):

curl -s "http://jaeger:16686/api/traces?service=order-service&minDuration=1s&limit=20"  # Slow traces
curl -s "http://jaeger:16686/api/traces?service=order-service&tags=error%3Dtrue&limit=20"  # Error traces

Alerting Best Practices

Avoid alert fatigue -- rules of thumb:

• Every alert must have a runbook link
• Every alert must be actionable (if no one needs to act, it is a log, not an alert)
• Group related alerts to avoid notification storms
• Use inhibition rules: if the cluster is down, suppress per-pod alerts

SLO-based alerting (burn rate):

# SLO: 99.9% availability = 43.2 min/month error budget
# Fast burn (exhausts budget in 2h): error_ratio > 14.4 * 0.001 for 2m -> critical
# Slow burn (exhausts budget in 3d): error_ratio > 3 * 0.001 for 15m -> warning
groups:
  - name: slo-burn-rate
    rules:
      - alert: SLOBurnRateCritical
        expr: sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > (14.4 * 0.001)
        for: 2m
        labels: { severity: critical }
      - alert: SLOBurnRateWarning
        expr: sum(rate(http_requests_total{code=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > (3 * 0.001)
        for: 15m
        labels: { severity: warning }

Runbook template: Each alert runbook should cover: what the alert means (one sentence), impact scope, diagnosis steps (dashboard + commands), mitigation (quick fix vs proper fix), and escalation path (who to contact after 15 min).

Metrics Collection Patterns

RED Method (request-scoped services):

Metric	What	PromQL Example
Rate	Requests per second	sum(rate(http_requests_total[5m]))
Errors	Failed requests per second	sum(rate(http_requests_total{code=~"5.."}[5m]))
Duration	Latency distribution	histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

USE Method (infrastructure resources):

Metric	What	Example Check
Utilization	% time resource is busy	avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]))
Saturation	Queue depth / backlog	node_load1 / count(node_cpu_seconds_total{mode="idle"})
Errors	Error event count	rate(node_disk_io_time_weighted_seconds_total[5m])

When to use which:

• RED for services that handle requests (APIs, web servers, message consumers)
• USE for infrastructure (CPU, memory, disk, network interfaces, queues)
• Combine both for a complete picture

---

Security Operations

Secret Management

Principles:

• Never store secrets in source code, environment variables (in Dockerfiles), or container images
• Use a secrets manager (Vault, AWS Secrets Manager, K8s Secrets with encryption at rest)
• Rotate secrets on a schedule and immediately after any suspected compromise
• Audit all secret access

Vault pattern -- inject secrets at runtime:

# Store a secret
vault kv put secret/myapp/db \
  username="app_user" \
  password="$(openssl rand -base64 32)"

# Read a secret (application startup)
vault kv get -format=json secret/myapp/db | jq -r '.data.data.password'

# Enable audit logging
vault audit enable file file_path=/var/log/vault-audit.log

Kubernetes secrets -- from Vault using sidecar injector:

Annotate the pod template with vault.hashicorp.com/agent-inject: "true", specify the role and secret path. The Vault agent sidecar renders secrets to /vault/secrets/ and the app sources them at startup. Key annotations: agent-inject-secret-<name> for the path, agent-inject-template-<name> for the rendering template.

Secret rotation checklist:

• Generate new secret value
• Update secret in secrets manager
• Restart/reload affected services (rolling, not all-at-once)
• Verify services authenticate with new secret
• Revoke the old secret value
• Confirm no services are still using the old secret

Container Security Scanning

# Scan image for vulnerabilities (Trivy) -- fail CI on critical
trivy image --exit-code 1 --severity CRITICAL registry.example.com/myapp:$CI_COMMIT_SHA

# Scan K8s cluster for misconfigurations
trivy k8s --report summary cluster

Dockerfile security essentials: use pinned base image tags (not :latest), run as non-root (USER app), copy only needed files, never bake secrets into image layers.

Network Security Policies

Kubernetes NetworkPolicy -- default deny with explicit allow:

# Default deny all ingress, then allow specific paths
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-to-orders
  namespace: production
spec:
  podSelector:
    matchLabels: { app: order-service }
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: api-gateway }
      ports:
        - { protocol: TCP, port: 8080 }

Apply a default-deny-ingress policy (empty podSelector, policyTypes: [Ingress]) per namespace first, then layer allow rules on top.

Compliance as Code

Policy enforcement with OPA/Gatekeeper: use K8sRequiredResources constraints to enforce limits.cpu, limits.memory, requests.cpu, requests.memory on all pods in production namespaces.

Quick compliance audit commands:

# Find pods without resource limits
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].resources.limits == null) | .metadata.namespace + "/" + .metadata.name'
# Find containers running as root
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].securityContext.runAsNonRoot != true) | .metadata.namespace + "/" + .metadata.name'
# Find ingress without TLS
kubectl get ingress -A -o json | jq -r '.items[] | select(.spec.tls == null) | .metadata.namespace + "/" + .metadata.name'

---

Automation Patterns

Auto-Remediation

Restart on OOM (Kubernetes):

# Built-in: set resource limits and let K8s handle OOM restarts
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: myapp
          resources:
            limits:
              memory: "512Mi"
            requests:
              memory: "256Mi"
          # Liveness probe: restart if unhealthy
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 3

Scale on load (HPA with custom metrics):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: order-service }
  minReplicas: 3
  maxReplicas: 20
  behavior:
    scaleUp:   { stabilizationWindowSeconds: 60,  policies: [{ type: Percent, value: 50, periodSeconds: 60 }] }
    scaleDown: { stabilizationWindowSeconds: 300, policies: [{ type: Percent, value: 25, periodSeconds: 120 }] }
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }
    - type: Pods
      pods: { metric: { name: http_requests_per_second }, target: { type: AverageValue, averageValue: "1000" } }

Rotate secrets on expiry (CronJob):

Use a K8s CronJob (e.g., monthly "0 2 1 * *") with a secret-rotator service account that: generates new password -> updates Vault -> alters DB role password -> triggers rolling restart via kubectl rollout restart.

GitOps Workflow

Repository as source of truth:

infrastructure-repo/
  |-- apps/
  |   |-- order-service/
  |   |   |-- deployment.yaml
  |   |   |-- service.yaml
  |   |   |-- hpa.yaml
  |   |   `-- kustomization.yaml
  |   `-- payment-service/
  |       |-- deployment.yaml
  |       `-- kustomization.yaml
  |-- base/
  |   |-- namespace.yaml
  |   |-- network-policies.yaml
  |   `-- resource-quotas.yaml
  `-- overlays/
      |-- staging/
      |   `-- kustomization.yaml
      `-- production/
          `-- kustomization.yaml

Reconciliation loop (ArgoCD application):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: order-service
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/infrastructure-repo.git
    targetRevision: main
    path: apps/order-service
  destination: { server: "https://kubernetes.default.svc", namespace: production }
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]
    retry: { limit: 3, backoff: { duration: 5s, factor: 2, maxDuration: 3m } }

GitOps deployment flow:

Developer pushes image tag update to infrastructure-repo
  |
  v
ArgoCD detects drift between git state and cluster state
  |
  v
ArgoCD syncs: applies manifests from git to cluster
  |
  v
Kubernetes rolls out new pods
  |
  v
ArgoCD verifies health (readiness probes pass)
  |
  [HEALTHY] --> Done
  [DEGRADED] --> ArgoCD marks sync as failed, alerts on-call

Database Backup and Restore

Automated backup (PostgreSQL) -- run via cron 0 */6 * * *:

#!/bin/bash
set -euo pipefail
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
DB_NAME="production"
BACKUP_DIR="/backups/postgres"

pg_dump -Fc -Z 9 "$DB_NAME" > "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.dump"
aws s3 cp "${BACKUP_DIR}/${DB_NAME}_${TIMESTAMP}.dump" \
  "s3://backups-bucket/postgres/" --storage-class STANDARD_IA
find "$BACKUP_DIR" -name "*.dump" -mtime +30 -delete

Restore procedure:

#!/bin/bash
set -euo pipefail
BACKUP_FILE=$1  # e.g., "production_20250315_060000.dump"
RESTORE_DB="production_restore"

aws s3 cp "s3://backups-bucket/postgres/$BACKUP_FILE" /tmp/restore.dump
psql -c "DROP DATABASE IF EXISTS $RESTORE_DB;" && psql -c "CREATE DATABASE $RESTORE_DB;"
pg_restore -d "$RESTORE_DB" -j 4 --no-owner /tmp/restore.dump
# Verify: check row counts on key tables, then clean up
rm /tmp/restore.dump

Disaster Recovery Runbook Template

Recovery Objectives: Define RTO (e.g., 1 hour) and RPO (e.g., 6 hours) per service.

Prerequisites: backup storage access, Terraform state access, DNS management access, stakeholder comms channel.

Scenario	Key Steps
Single service failure	Check pod status -> restart deployment -> if fails, kubectl rollout undo -> verify health
Database failure	pg_isready -> promote replica (or restore from backup) -> update connection strings -> verify data integrity
Full region outage	Confirm via provider status page -> notify stakeholders -> switch DNS to DR region -> verify traffic -> failback when primary recovers

Communication template: Subject [INCIDENT] Service -- Status. Body: what happened, impact, current status, ETA, next update time.

Post-recovery checklist: health checks passing, data integrity verified, monitoring restored, backups resumed, incident report filed, post-mortem scheduled within 48h.