Triage

Alga's triage system evaluates correlated alerts and decides whether they should become incidents. It acts as the gate between alert correlation and incident creation — only alerts classified as incident-worthy proceed to the incident management pipeline.

How Triage Works

Triage is a two-stage pipeline: rules first, then LLM.

Correlated Alerts → Triage Rules (ordered, deterministic) → Match? → Decision
                                                              │
                                                         No match
                                                              ↓
                                                    LLM Evaluation (smart, costs tokens)
                                                              ↓
                                                          Decision

Correlation groups related alerts within the CORRELATION_WINDOW (see AI Investigation)
Stage 1 — Rules: The triage engine evaluates the correlated alert group against ordered triage rules. Rules are deterministic, fast, and free (no token cost). The first matching rule determines the decision.
Stage 2 — LLM: If no rule matches, the group is sent to the configured LLM for intelligent classification. The LLM returns a decision with confidence, reasoning, and suggested actions. This stage costs tokens and is gated by TRIAGE_ENABLED.
Result is stored with confidence score, reasoning, and suggested actions
If the decision is investigate or escalate, an investigation is dispatched automatically (not an incident). Incidents are created separately by the incident worker for correlated critical-severity alerts.
Operators can override any triage decision after review

Triage Decisions

Decision	Description
`investigate`	Dispatch an agent investigation (no incident created by triage itself)
`escalate`	Dispatch an investigation immediately (escalation-policy paging is not wired at this layer)
`suppress`	Dismiss the alert group — no incident created
`auto_resolve`	Automatically resolve the alerts without incident
`enrich_only`	Enrich alert metadata without creating an incident

Note (gated decisions): The auto_resolve and suppress decisions are gated by the config flags TRIAGE_AUTO_RESOLVE_ENABLED and TRIAGE_SUPPRESS_ENABLED respectively. When a flag is disabled, that decision downgrades to enrich_only. Additionally, any decision whose confidence falls below TRIAGE_CONFIDENCE_THRESHOLD (default 0.7) also downgrades to enrich_only.

Triage Outcomes

Each triage result has an outcome that tracks operator review:

Outcome	Description
`pending`	Awaiting operator review
`confirmed`	Operator agreed with the automated decision
`overridden`	Operator changed the decision

Triage Rules

Triage rules are evaluated in priority order (lower priority number = evaluated first). The first matching rule determines the outcome.

Rule Fields

Field	Type	Description
`name`	string	Rule name (required)
`description`	string	Rule description
`conditions`	array	Condition objects matching against alert labels/annotations
`match_mode`	string	How conditions are combined: `"all"` or `"any"`
`decision`	string	Triage decision (`investigate`, `escalate`, `suppress`, `auto_resolve`, `enrich_only`)
`severity`	string	Severity to assign if an incident is created
`category`	string	Category label for grouping
`enrichment`	object	Additional data to attach to the result
`priority`	integer	Rule priority — lower values are evaluated first
`enabled`	boolean	Whether the rule is active

Creating a Rule

bash

curl -X POST http://localhost:8080/api/v1/triage/rules \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SESSION" \
  -d '{
    "name": "Critical production alerts",
    "description": "Escalate all critical alerts from production",
    "conditions": [
      {"field": "labels.severity", "operator": "exact", "value": "critical"},
      {"field": "labels.namespace", "operator": "exact", "value": "production"}
    ],
    "match_mode": "all",
    "decision": "escalate",
    "severity": "critical",
    "category": "infrastructure",
    "priority": 10,
    "enabled": true
  }'

Ordering Rules

Rules are evaluated in priority order. Reorder them with a single call:

bash

curl -X PUT http://localhost:8080/api/v1/triage/rules/reorder \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SESSION" \
  -d '{"ids": ["rule-id-1", "rule-id-2", "rule-id-3"]}'

Rules are evaluated in the order specified. Use this to ensure high-priority rules are checked first.

Listing Rules

bash

# List all rules
curl http://localhost:8080/api/v1/triage/rules \
  -H "Authorization: Bearer $SESSION"

# Filter to enabled rules only
curl "http://localhost:8080/api/v1/triage/rules?enabled=true" \
  -H "Authorization: Bearer $SESSION"

# Search rules by name
curl "http://localhost:8080/api/v1/triage/rules?search=critical" \
  -H "Authorization: Bearer $SESSION"

Triage Results

Every triage evaluation produces a result record that captures the decision, reasoning, and confidence score.

Result Fields

Field	Description
`triage_number`	Unique sequential identifier
`correlation_key`	Correlation key of the alert group
`alert_count`	Number of alerts in the group
`alert_fingerprints`	Fingerprints of the grouped alerts
`alert_labels`	Merged labels from the alert group
`decision`	Automated triage decision
`confidence`	Confidence score (0.0 – 1.0)
`severity_classified`	Severity determined by triage
`category`	Category label
`reasoning`	Explanation of the decision
`suggested_actions`	Recommended next steps
`outcome`	Review status (`pending`, `confirmed`, `overridden`)
`overridden_to`	New decision if overridden
`model_used`	AI model used for classification (if applicable)
`triage_duration_ms`	Time taken to evaluate
`severity_input`	Original severity from the alert group before classification
`enrichment`	Additional data attached to the result
`context_used`	Context sources referenced during evaluation
`overridden_by`	User ID who overrode the decision
`overridden_at`	Timestamp when the override occurred
`override_reason`	Free-text reason supplied with the override
`trace_id`	Correlation trace ID for debugging

Viewing Results

bash

# List all results
curl http://localhost:8080/api/v1/triage/results \
  -H "Authorization: Bearer $SESSION"

# Filter by decision
curl "http://localhost:8080/api/v1/triage/results?decision=suppress" \
  -H "Authorization: Bearer $SESSION"

# Filter by outcome
curl "http://localhost:8080/api/v1/triage/results?outcome=pending" \
  -H "Authorization: Bearer $SESSION"

# Filter by date range
curl "http://localhost:8080/api/v1/triage/results?start_date=2026-05-01&end_date=2026-05-10" \
  -H "Authorization: Bearer $SESSION"

Overriding a Decision

Operators with the triage:override permission can change a triage decision:

bash

curl -X POST http://localhost:8080/api/v1/triage/results/{id} \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SESSION" \
  -d '{
    "decision": "investigate",
    "reason": "This alert group indicates a real outage"
  }'

The decision must be one of the triage decisions above (otherwise 400). The reason is persisted as override_reason together with overridden_by/overridden_at, and the override is recorded in the audit log. The stored reason is returned by GET /api/v1/triage/results/{id}.

Triage Stats

The stats endpoint tracks accuracy over time:

bash

curl http://localhost:8080/api/v1/triage/stats \
  -H "Authorization: Bearer $SESSION"

Response includes:

Field	Description
`total`	Total triage evaluations
`accuracy`	Percentage of confirmed vs overridden results
`by_decision`	Count breakdown by decision type
`by_category`	Count breakdown by category
`avg_confidence`	Average confidence score across all results
`avg_duration_ms`	Average evaluation time in milliseconds
`volume_trend_30d`	Daily volume counts for the last 30 days

CLI Commands

bash

# Show triage accuracy and volume stats
./alga triage stats

Output:

Triage Stats:
  Total: 142 (confirmed: 128, overridden: 6, pending: 8)
  Accuracy: 95.5%
  By decision: map[investigate:89 suppress:38 escalate:15]

Configuration

Variable	Default	Description
`TRIAGE_ENABLED`	`true`	Enable/disable the triage pipeline
`TRIAGE_LLM_URL`	—	LLM endpoint URL for stage-2 evaluation
`TRIAGE_LLM_API_KEY`	—	API key for the LLM provider
`TRIAGE_LLM_MODEL`	—	Model name to use for LLM triage
`TRIAGE_MAX_CONCURRENT`	`3`	Maximum concurrent LLM triage evaluations
`TRIAGE_CONFIDENCE_THRESHOLD`	`0.7`	Minimum confidence for non-investigate decisions; below this, `auto_resolve`, `suppress`, and other non-`investigate` decisions downgrade to `enrich_only`. `investigate` decisions are unaffected
`TRIAGE_AUTO_RESOLVE_ENABLED`	`true`	Gate for the `auto_resolve` decision
`TRIAGE_SUPPRESS_ENABLED`	`true`	Gate for the `suppress` decision
`TRIAGE_CONTEXT_EPISODIC_LIMIT`	`3`	Max episodic memories injected into LLM context
`TRIAGE_CONTEXT_NOTES_LIMIT`	`3`	Max knowledge notes injected into LLM context
`TRIAGE_CONTEXT_MEMORIES_LIMIT`	`5`	Max agent memories injected into LLM context
`TRIAGE_AUTO_PROMOTE_CONFIRMED_COUNT`	`3`	Confirmed results of the same correlation key + decision before new triages for that key skip the LLM (0 disables)
`MaxConcurrentTriage`	`5`	Config-level concurrency cap for triage worker (config file default)

Context Injection

When the LLM stage runs, the triage engine injects contextual data to improve classification accuracy:

Episodic memory — up to TRIAGE_CONTEXT_EPISODIC_LIMIT past investigations with the same correlation key
Knowledge notes — up to TRIAGE_CONTEXT_NOTES_LIMIT matching operator-authored notes
Agent memories — up to TRIAGE_CONTEXT_MEMORIES_LIMIT semantically-relevant memories from past investigations

Auto-Promote

When TRIAGE_AUTO_PROMOTE_CONFIRMED_COUNT (default 3) triage results with outcome confirmed share the same correlation key and decision, the engine auto-promotes the pattern — the next triage for that key applies the confirmed decision directly (confidence 1.0) without any LLM call, and records the promotion in the result's reasoning. The lookup resolves the latest confirmed decision for the key; lookup failures fall through to the LLM.

API Endpoints

Rules

Method	Path	Permission	Description
`GET`	`/api/v1/triage/rules`	`triage:read`	List triage rules (supports `?search=`, `?enabled=true`, `?limit=`, `?skip=`)
`POST`	`/api/v1/triage/rules`	`triage:write`	Create triage rule
`GET`	`/api/v1/triage/rules/{id}`	`triage:read`	Get triage rule
`PUT`	`/api/v1/triage/rules/{id}`	`triage:write`	Update triage rule
`DELETE`	`/api/v1/triage/rules/{id}`	`triage:write`	Delete triage rule
`PUT`	`/api/v1/triage/rules/reorder`	`triage:write`	Reorder triage rules

Results

Method	Path	Permission	Description
`GET`	`/api/v1/triage/results`	`triage:read`	List triage results (supports `?decision=`, `?outcome=`, `?category=`, `?severity=`, `?search=`, `?start_date=`, `?end_date=`, `?limit=`, `?skip=`)
`GET`	`/api/v1/triage/results/{id}`	`triage:read`	Get triage result
`POST`	`/api/v1/triage/results/{id}`	`triage:override`	Override triage decision

Stats

Method	Path	Permission	Description
`GET`	`/api/v1/triage/stats`	`triage:read`	Get triage accuracy stats

Best Practices

Start with broad rules, then refine. Begin with a few high-priority rules that catch obvious cases (e.g., suppress known noise), then add more specific rules as you observe triage results.
Order rules by specificity. Place more specific rules (many conditions, match_mode: "all") before catch-all rules (match_mode: "any").
Review pending results regularly. Use the ?outcome=pending filter to find results that need operator confirmation. Confirming or overriding results improves accuracy tracking.
Use suppress for known noise. Alerts from test environments, synthetic monitors, or known maintenance should be suppressed to avoid incident fatigue.
Assign categories. Categories help you track triage patterns over time and identify areas where rules need tuning.
Monitor accuracy trends. Watch the accuracy stat and volume_trend_30d to spot degradation. A dropping accuracy rate means your rules need updating.
Enrich rather than discard. Prefer enrich_only over suppress when alerts might be useful later — enriched metadata is available for future triage evaluations.
Test rule changes in small batches. Disable rules before editing them, then re-enable to avoid mid-evaluation inconsistencies.

Triage ​

How Triage Works ​

Triage Decisions ​

Triage Outcomes ​

Triage Rules ​

Rule Fields ​

Creating a Rule ​

Ordering Rules ​

Listing Rules ​

Triage Results ​

Result Fields ​

Viewing Results ​

Overriding a Decision ​

Triage Stats ​

CLI Commands ​

Configuration ​

Context Injection ​

Auto-Promote ​

API Endpoints ​

Rules ​

Results ​

Stats ​

Best Practices ​

Triage

How Triage Works

Triage Decisions

Triage Outcomes

Triage Rules

Rule Fields

Creating a Rule

Ordering Rules

Listing Rules

Triage Results

Result Fields

Viewing Results

Overriding a Decision

Triage Stats

CLI Commands

Configuration

Context Injection

Auto-Promote

API Endpoints

Rules

Results

Stats

Best Practices