Skip to content

Playbooks

Playbooks provide structured, step-by-step response procedures that agents and operators can follow during incidents. They can be automatically matched to investigations based on alert labels.

Overview

Playbooks define repeatable response procedures for common operational scenarios:

  • Procedure playbooks document standard operating procedures (SOPs)
  • Mitigation playbooks guide incident response and remediation
  • Label selectors automatically match playbooks to alert contexts
  • Ordered steps with titles, descriptions, durations, and optional commands
  • Automatic enrichment into investigation prompts when labels match

Playbook Fields

FieldTypeDescription
titlestringPlaybook name
kindstringprocedure or mitigation
summarystringBrief description of the playbook's purpose
service_idUUIDOptional linked service for scoping
label_selectorsarrayFlat key→value maps for matching alert labels (exact equality; prefix value with ~ for regex)
tagsarrayCategorization tags for filtering
stepsarrayOrdered list of execution steps

Steps

Each step within a playbook contains:

FieldTypeDescription
step_numberintOrdering position of the step
titlestringStep name
descriptionstringDetailed instructions
expected_durationstringEstimated time to complete (e.g., "5m")
commandstringOptional shell command to execute

Creating Playbooks

Create a playbook with label selectors and steps:

sh
curl -X POST http://localhost:8080/api/v1/playbooks \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Database Connection Pool Exhaustion",
    "kind": "mitigation",
    "summary": "Respond to database connection pool exhaustion alerts",
    "label_selectors": [
      {"alertname": "DBConnectionPoolExhausted"}
    ],
    "tags": ["database", "connectivity"],
    "steps": [
      {
        "title": "Check active connections",
        "description": "Query pg_stat_activity for active connection count and wait events",
        "expected_duration": "5m",
        "command": "psql -c \"SELECT count(*) FROM pg_stat_activity\""
      },
      {
        "title": "Identify long-running queries",
        "description": "Find queries running longer than 60 seconds",
        "expected_duration": "5m"
      },
      {
        "title": "Terminate blocked queries",
        "description": "Terminate queries that are blocking connection pool slots",
        "expected_duration": "3m"
      }
    ]
  }'

API Endpoints

Playbook Management

MethodPathPermissionDescription
GET/api/v1/playbooksplaybooks:readList playbooks (query: kind, service_id, tag, search, limit, skip)
POST/api/v1/playbooksplaybooks:writeCreate playbook
GET/api/v1/playbooks/{id}playbooks:readGet playbook with steps
PATCH/api/v1/playbooks/{id}playbooks:writeUpdate playbook
DELETE/api/v1/playbooks/{id}playbooks:deleteDelete playbook

Step Management

MethodPathPermissionDescription
POST/api/v1/playbooks/{id}/stepsplaybooks:writeAdd step
PATCH/api/v1/playbooks/{id}/steps/{stepId}playbooks:writeUpdate step
DELETE/api/v1/playbooks/{id}/steps/{stepId}playbooks:deleteDelete step
PUT/api/v1/playbooks/{id}/steps/reorderplaybooks:writeReorder steps

Automatic Matching

The investigation scheduler automatically matches playbooks to investigations:

  1. When an investigation is created, the scheduler evaluates label selectors against the alert's labels
  2. Matched playbooks are included in the investigation prompt sent to the agent
  3. Steps are displayed in the incident detail sidebar under Mitigation Playbooks
  4. Agents can reference playbook steps during investigation and mark them as completed

Label selectors are a flat map of label key to value, matched with exact equality. There is no operator field. To match with a regular expression instead, prefix the value with ~.

json
// Exact match: labels.alertname == "DBConnectionPoolExhausted"
{"alertname": "DBConnectionPoolExhausted"}

// Regex match: labels.namespace matches the pattern
{"namespace": "~prod-.*"}

A selector matches when all of its key/value pairs match the alert's labels. Multiple selectors in the label_selectors array are OR'd — a playbook matches if any selector matches.

See Also

Released under the MIT License.