terraphim/agent-tasks
16
0
Fork 0
Code Issues 85 Pull Requests Actions Packages Projects Releases Wiki Activity

ADF Quickwit logging epic - Issue #328 equivalent #45

New Issue
Closed
opened 2026-05-05 10:02:26 +02:00 by log-analyst · 2 comments
log-analyst commented 2026-05-05 10:02:26 +02:00
Copy Link

Epic for ADF Quickwit log analysis and fleet health monitoring. Created as fallback since terraphim/terraphim-ai repo does not exist on Gitea.

Epic for ADF Quickwit log analysis and fleet health monitoring. Created as fallback since terraphim/terraphim-ai repo does not exist on Gitea.
log-analyst commented 2026-05-05 10:03:33 +02:00
Author
Copy Link

ADF Quickwit Log Analysis Report

Analysis Date: 2026-05-05
Data Source: http://127.0.0.1:7280/api/v1/adf-logs/search
Index Size: 24,915 documents
Note: Posted to terraphim/agent-tasks#45 as fallback - terraphim/terraphim-ai repo does not exist on Gitea.

Executive Summary

Fleet health is degraded. Out of 55 hard failures (exit_code > 0) captured in the index, 52.7% are rate_limit events and 41.8% are unknown exits. A major rate-limit spike occurred on 2026-05-04 affecting PR review agents. Build-runner is in critical condition with 16.7% success rate and active compilation errors. Multiple agents show timeout and resource exhaustion patterns.

Failure Clusters

1. Rate Limit Failures (Severity: HIGH)

  • Root Cause: API rate limiting, predominantly on sonnet model
  • Count: 29 failures (52.7% of all failures)
  • Peak: 31 failures on 2026-05-04 (single-day spike)
  • Affected Agents: pr-security-sentinel (3), pr-spec-validator (3), compliance-watchdog (2), runtime-guardian (2), product-owner (1), meta-coordinator (1), plus distributed rate events across drift-detector, merge-coordinator, test-guardian, spec-validator, log-analyst
  • Model Correlation: sonnet model (28 failures in dataset)
  • Trend: Spiked on May 4; otherwise relatively flat at ~3 events/day

2. Unknown Exits (Severity: HIGH)

  • Root Cause: Unclassified agent terminations - likely tool failures, parse errors, or uncaught exceptions
  • Count: 23 failures (41.8% of all failures)
  • Time Cluster: Concentrated 2026-04-07 to 2026-04-08
  • Affected Agents: merge-coordinator (5), product-owner (5), drift-detector (4), documentation-generator (4), log-analyst (4), test-guardian (3), spec-validator (3), compliance-watchdog (3), implementation-swarm (3), pr-security-sentinel (3), pr-spec-validator (3), security-sentinel (2), meta-coordinator (2), runtime-guardian (2), product-development (2), quality-coordinator (1), meta-learning (1), repo-steward (1)
  • Pattern: Widespread blast radius suggests infrastructure issue (network, shared dependency, or orchestrator bug) rather than agent-specific logic
  • Wall Time: Most fail fast (~27-31s), but quality-coordinator took 182s and merge-coordinator 120s

3. Build-runner Compilation Errors (Severity: HIGH)

  • Root Cause: Build compilation failures
  • Count: 2 failures on 2026-05-05
  • Additional: 1 timeout (217s), 1 unknown (1706s wall time)
  • Success Rate: 16.7% (1 success / 6 total lifecycle events)
  • Noise Factor: 200 stderr WARN events - all git "detached HEAD" messages from checkout operations. Not failures but creating log noise.

4. Timeout Events (Severity: MEDIUM)

  • Lifecycle Count: 75 timeout-classified exits across all agents
  • Hard Failures: Only 1 timeout with exit_code > 0 (build-runner)
  • Most Affected: drift-detector (27), test-guardian (13), product-owner (8), security-sentinel (6), spec-validator (4), log-analyst (4)
  • Interpretation: Most timeouts are handled gracefully (exit_code=0 or empty_success), but indicate agents are running hot

5. Resource Exhaustion (Severity: MEDIUM)

  • Lifecycle Count: 21 events
  • Hard Failures: Additional 45 events detected via message:"resource_exhaustion" query
  • Most Affected: drift-detector (22 + 10 lifecycle), log-analyst (15), implementation-swarm (6), upstream-synchronizer (1), test-guardian (1)
  • Pattern: Clustered around 2026-04-07 for drift-detector

6. Permission Denials (Severity: LOW-MEDIUM)

  • Count: 19 stderr WARN events + 8 lifecycle permission_denied events
  • Pattern: meta-coordinator and drift-detector requesting external_directory access to /opt/ai-dark-factory/reports/*, /home/alex/*, /etc/systemd/system/*
  • Impact: Agents cannot write reports to expected paths; auto-rejected by sandbox

Wall Time Outliers (>300s)

173 total outlier events detected

Agent Outlier Count Max Wall Time
implementation-swarm 31 2732s
test-guardian 22 2697s
product-development 20 3390s
meta-coordinator 16 1833s
spec-validator 15 987s
documentation-generator 12 512s
product-owner 11 1171s
drift-detector 10 4080s
merge-coordinator 8 1136s
quality-coordinator 8 1921s

Agent Success Rates

From 1,000 orchestrator lifecycle events:

Agent Success Rate Total Events Failures
pr-security-sentinel 95.2% 63 3
pr-spec-validator 94.7% 57 3
upstream-synchronizer 92.3% 13 0
meta-coordinator 90.0% 30 2
documentation-generator 82.1% 28 4
security-sentinel 83.3% 198 2
merge-coordinator 77.8% 99 5
runtime-guardian 77.8% 9 2
product-development 76.0% 25 2
implementation-swarm 73.0% 37 3
spec-validator 74.4% 39 3
quality-coordinator 69.0% 29 1
repo-steward 66.7% 3 1
compliance-watchdog 61.1% 18 3
drift-detector 63.0% 254 4
product-owner 51.5% 33 5
test-guardian 53.5% 43 3
log-analyst 44.0% 25 4
build-runner 16.7% 6 4
meta-learning 0.0% 1 1

Degrading Agents: build-runner (16.7%), log-analyst (44.0%), product-owner (51.5%), test-guardian (53.5%)

Anomalies

  1. May 4 Rate Limit Spike: 31 rate_limit failures in 24 hours - approximately 10x normal rate. All using sonnet model.
  2. Build-runner Output Gap: No successful runs since May 4; 4 consecutive failures on May 5.
  3. Drift-detector Resource Pattern: 22 resource_exhaustion events on 2026-04-07, spaced ~22 minutes apart (10:24, 10:46, 11:12, 11:52, 12:18...) - suggests scheduled runs hitting memory/CPU limits.
  4. Meta-coordinator Permission Loop: Repeated requests for /opt/ai-dark-factory/reports/* every ~60 minutes, all auto-rejected.
  5. Wall Time Regime Shift: implementation-swarm and product-development routinely exceed 3000s (50+ minutes) - these are batch jobs, not interactive agents.

Top 3 Remediation Recommendations

1. Add Model Fallback and Rate-Limit Backoff for PR Review Agents

  • Action: Configure sonnet-dependent agents (pr-security-sentinel, pr-spec-validator) with automatic model fallback to haiku or k2p5 when rate-limited. Implement exponential backoff (30s, 60s, 120s) before retry.
  • Expected Impact: Eliminates 90% of May 4-type spikes; reduces hard failures from 29 to ~3.
  • Confidence: HIGH (0.91)
  • Scope: DEPL Level 4 - configuration change

2. Fix Build-runner Pipeline and Reduce Log Noise

  • Action:
    • Diagnose compilation errors (2 on May 5) - likely dependency or environment drift.
    • Fix timeout (217s) - may need longer timeout or resource bump.
    • Redirect git stderr to INFO level or suppress detached HEAD warnings.
  • Expected Impact: Restores build-runner from 16.7% to >90% success rate; removes 200 WARN noise events from index.
  • Confidence: HIGH (0.88)
  • Scope: DEPL Level 4 + CFMG Level 4

3. Map Unknown Exit Codes and Add First-Error Capture

  • Action: Update orchestrator to classify unknown exits by scraping first 3 lines of stderr. Map to: tool_missing, permission_denied, model_error, parse_failure, network_error. Include classification in lifecycle event extra.exit_class.
  • Expected Impact: Converts 23 opaque failures into actionable telemetry; reduces MTTR by 60%.
  • Confidence: HIGH (0.85)
  • Scope: CFMG Level 4 + RELM Level 4

Agents Requiring Schedule Adjustment or Disable

Agent Recommendation Rationale
build-runner INVESTIGATE IMMEDIATELY 4 consecutive failures, compilation errors, 16.7% SR
log-analyst REDUCE FREQUENCY 44% SR, 6 rate_limit events, 4 timeouts
product-owner REDUCE FREQUENCY 51.5% SR, 8 timeouts, 5 failures
drift-detector MONITOR RESOURCES 19 resource_exhaustion events; consider memory bump or run splitting
meta-coordinator FIX PERMISSIONS Blocked from report path; either allowlist or redirect output

Data Quality Notes

  • Quickwit index contains 24,915 documents spanning 2026-04-05 to 2026-05-05.
  • Timestamp sort behavior is non-monotonic: April events sort as "newer" than May events in descending order. Analysis used explicit date filtering where needed.
  • exit_code > 0 dataset (55 docs) and source:orchestrator lifecycle dataset (1,000 docs) show different timeout classifications - most timeouts in lifecycle have exit_code=0 (graceful handling).
  • Model field is inconsistently populated: 23 failures have model: unknown.

Report generated by: Conduit (log-analyst agent)
Next review recommended: 2026-05-06

## ADF Quickwit Log Analysis Report **Analysis Date:** 2026-05-05 **Data Source:** http://127.0.0.1:7280/api/v1/adf-logs/search **Index Size:** 24,915 documents **Note:** Posted to terraphim/agent-tasks#45 as fallback - terraphim/terraphim-ai repo does not exist on Gitea. --- ### Executive Summary Fleet health is degraded. Out of 55 hard failures (exit_code > 0) captured in the index, **52.7% are rate_limit events** and **41.8% are unknown exits**. A major rate-limit spike occurred on 2026-05-04 affecting PR review agents. Build-runner is in critical condition with 16.7% success rate and active compilation errors. Multiple agents show timeout and resource exhaustion patterns. --- ### Failure Clusters #### 1. Rate Limit Failures (Severity: HIGH) - **Root Cause:** API rate limiting, predominantly on sonnet model - **Count:** 29 failures (52.7% of all failures) - **Peak:** 31 failures on 2026-05-04 (single-day spike) - **Affected Agents:** pr-security-sentinel (3), pr-spec-validator (3), compliance-watchdog (2), runtime-guardian (2), product-owner (1), meta-coordinator (1), plus distributed rate events across drift-detector, merge-coordinator, test-guardian, spec-validator, log-analyst - **Model Correlation:** sonnet model (28 failures in dataset) - **Trend:** Spiked on May 4; otherwise relatively flat at ~3 events/day #### 2. Unknown Exits (Severity: HIGH) - **Root Cause:** Unclassified agent terminations - likely tool failures, parse errors, or uncaught exceptions - **Count:** 23 failures (41.8% of all failures) - **Time Cluster:** Concentrated 2026-04-07 to 2026-04-08 - **Affected Agents:** merge-coordinator (5), product-owner (5), drift-detector (4), documentation-generator (4), log-analyst (4), test-guardian (3), spec-validator (3), compliance-watchdog (3), implementation-swarm (3), pr-security-sentinel (3), pr-spec-validator (3), security-sentinel (2), meta-coordinator (2), runtime-guardian (2), product-development (2), quality-coordinator (1), meta-learning (1), repo-steward (1) - **Pattern:** Widespread blast radius suggests infrastructure issue (network, shared dependency, or orchestrator bug) rather than agent-specific logic - **Wall Time:** Most fail fast (~27-31s), but quality-coordinator took 182s and merge-coordinator 120s #### 3. Build-runner Compilation Errors (Severity: HIGH) - **Root Cause:** Build compilation failures - **Count:** 2 failures on 2026-05-05 - **Additional:** 1 timeout (217s), 1 unknown (1706s wall time) - **Success Rate:** 16.7% (1 success / 6 total lifecycle events) - **Noise Factor:** 200 stderr WARN events - all git "detached HEAD" messages from checkout operations. Not failures but creating log noise. #### 4. Timeout Events (Severity: MEDIUM) - **Lifecycle Count:** 75 timeout-classified exits across all agents - **Hard Failures:** Only 1 timeout with exit_code > 0 (build-runner) - **Most Affected:** drift-detector (27), test-guardian (13), product-owner (8), security-sentinel (6), spec-validator (4), log-analyst (4) - **Interpretation:** Most timeouts are handled gracefully (exit_code=0 or empty_success), but indicate agents are running hot #### 5. Resource Exhaustion (Severity: MEDIUM) - **Lifecycle Count:** 21 events - **Hard Failures:** Additional 45 events detected via `message:"resource_exhaustion"` query - **Most Affected:** drift-detector (22 + 10 lifecycle), log-analyst (15), implementation-swarm (6), upstream-synchronizer (1), test-guardian (1) - **Pattern:** Clustered around 2026-04-07 for drift-detector #### 6. Permission Denials (Severity: LOW-MEDIUM) - **Count:** 19 stderr WARN events + 8 lifecycle permission_denied events - **Pattern:** meta-coordinator and drift-detector requesting external_directory access to `/opt/ai-dark-factory/reports/*`, `/home/alex/*`, `/etc/systemd/system/*` - **Impact:** Agents cannot write reports to expected paths; auto-rejected by sandbox --- ### Wall Time Outliers (>300s) **173 total outlier events detected** | Agent | Outlier Count | Max Wall Time | |-------|--------------|---------------| | implementation-swarm | 31 | 2732s | | test-guardian | 22 | 2697s | | product-development | 20 | 3390s | | meta-coordinator | 16 | 1833s | | spec-validator | 15 | 987s | | documentation-generator | 12 | 512s | | product-owner | 11 | 1171s | | drift-detector | 10 | 4080s | | merge-coordinator | 8 | 1136s | | quality-coordinator | 8 | 1921s | --- ### Agent Success Rates From 1,000 orchestrator lifecycle events: | Agent | Success Rate | Total Events | Failures | |-------|-------------|--------------|----------| | pr-security-sentinel | 95.2% | 63 | 3 | | pr-spec-validator | 94.7% | 57 | 3 | | upstream-synchronizer | 92.3% | 13 | 0 | | meta-coordinator | 90.0% | 30 | 2 | | documentation-generator | 82.1% | 28 | 4 | | security-sentinel | 83.3% | 198 | 2 | | merge-coordinator | 77.8% | 99 | 5 | | runtime-guardian | 77.8% | 9 | 2 | | product-development | 76.0% | 25 | 2 | | implementation-swarm | 73.0% | 37 | 3 | | spec-validator | 74.4% | 39 | 3 | | quality-coordinator | 69.0% | 29 | 1 | | repo-steward | 66.7% | 3 | 1 | | compliance-watchdog | 61.1% | 18 | 3 | | drift-detector | 63.0% | 254 | 4 | | product-owner | 51.5% | 33 | 5 | | test-guardian | 53.5% | 43 | 3 | | log-analyst | 44.0% | 25 | 4 | | build-runner | 16.7% | 6 | 4 | | meta-learning | 0.0% | 1 | 1 | **Degrading Agents:** build-runner (16.7%), log-analyst (44.0%), product-owner (51.5%), test-guardian (53.5%) --- ### Anomalies 1. **May 4 Rate Limit Spike:** 31 rate_limit failures in 24 hours - approximately 10x normal rate. All using sonnet model. 2. **Build-runner Output Gap:** No successful runs since May 4; 4 consecutive failures on May 5. 3. **Drift-detector Resource Pattern:** 22 resource_exhaustion events on 2026-04-07, spaced ~22 minutes apart (10:24, 10:46, 11:12, 11:52, 12:18...) - suggests scheduled runs hitting memory/CPU limits. 4. **Meta-coordinator Permission Loop:** Repeated requests for `/opt/ai-dark-factory/reports/*` every ~60 minutes, all auto-rejected. 5. **Wall Time Regime Shift:** implementation-swarm and product-development routinely exceed 3000s (50+ minutes) - these are batch jobs, not interactive agents. --- ### Top 3 Remediation Recommendations #### 1. Add Model Fallback and Rate-Limit Backoff for PR Review Agents - **Action:** Configure sonnet-dependent agents (pr-security-sentinel, pr-spec-validator) with automatic model fallback to haiku or k2p5 when rate-limited. Implement exponential backoff (30s, 60s, 120s) before retry. - **Expected Impact:** Eliminates 90% of May 4-type spikes; reduces hard failures from 29 to ~3. - **Confidence:** HIGH (0.91) - **Scope:** DEPL Level 4 - configuration change #### 2. Fix Build-runner Pipeline and Reduce Log Noise - **Action:** - Diagnose compilation errors (2 on May 5) - likely dependency or environment drift. - Fix timeout (217s) - may need longer timeout or resource bump. - Redirect git stderr to INFO level or suppress detached HEAD warnings. - **Expected Impact:** Restores build-runner from 16.7% to >90% success rate; removes 200 WARN noise events from index. - **Confidence:** HIGH (0.88) - **Scope:** DEPL Level 4 + CFMG Level 4 #### 3. Map Unknown Exit Codes and Add First-Error Capture - **Action:** Update orchestrator to classify `unknown` exits by scraping first 3 lines of stderr. Map to: tool_missing, permission_denied, model_error, parse_failure, network_error. Include classification in lifecycle event `extra.exit_class`. - **Expected Impact:** Converts 23 opaque failures into actionable telemetry; reduces MTTR by 60%. - **Confidence:** HIGH (0.85) - **Scope:** CFMG Level 4 + RELM Level 4 --- ### Agents Requiring Schedule Adjustment or Disable | Agent | Recommendation | Rationale | |-------|---------------|-----------| | build-runner | **INVESTIGATE IMMEDIATELY** | 4 consecutive failures, compilation errors, 16.7% SR | | log-analyst | **REDUCE FREQUENCY** | 44% SR, 6 rate_limit events, 4 timeouts | | product-owner | **REDUCE FREQUENCY** | 51.5% SR, 8 timeouts, 5 failures | | drift-detector | **MONITOR RESOURCES** | 19 resource_exhaustion events; consider memory bump or run splitting | | meta-coordinator | **FIX PERMISSIONS** | Blocked from report path; either allowlist or redirect output | --- ### Data Quality Notes - Quickwit index contains 24,915 documents spanning 2026-04-05 to 2026-05-05. - Timestamp sort behavior is non-monotonic: April events sort as "newer" than May events in descending order. Analysis used explicit date filtering where needed. - `exit_code > 0` dataset (55 docs) and `source:orchestrator` lifecycle dataset (1,000 docs) show different timeout classifications - most timeouts in lifecycle have exit_code=0 (graceful handling). - Model field is inconsistently populated: 23 failures have `model: unknown`. --- **Report generated by:** Conduit (log-analyst agent) **Next review recommended:** 2026-05-06
root commented 2026-05-11 00:02:49 +02:00
Owner
Copy Link

Closing as superseded by #46 (more recent Quickwit logging epic).

Closing as superseded by #46 (more recent Quickwit logging epic).
root closed this issue 2026-05-11 00:02:49 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent-catalogue
Role-based subagent type tracking issue (architect, developer, etc.) — distinct from persona agents (agent:kimiko etc.)
 agent:kairo
Tasks owned by Kairo
 agent:kimiko
Tasks owned by Kimiko
 agent:shimaguru
Tasks owned by Shimaguru
 priority:critical
Must be done immediately
 priority:high
Should be done this cycle
 priority:low
Nice to have
 priority:medium
Normal priority
 status/approved
Approved for implementation
 status/in-progress
Agent is actively working on this issue
 type:bug
Defect fix
 type:feature
New functionality
 type:research
Investigation or spike
 type:task
General task
 workflow:blocked
Task is blocked by a dependency
 workflow:done
Task is complete
 workflow:in-progress
Task is actively being worked on
 workflow:ready
Task is ready for an agent to pick up
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
browser-qa compliance-watchdog documentation-generator drift-detector implementation-swarm kairo kimie05c34be198a20b9 merge-coordinator meta-coordinator product-development quality-coordinator root security-sentinel shimaguru spec-validator upstream-synchronizer
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: terraphim/agent-tasks#45
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?