Triage failed CI builds and draft fixes for engineer review

When a build fails in CI, an engineer has to open the run, scroll through long logs, work out whether it is a real bug, a flaky test, a dependency change or a broken runner, and then decide who should fix it. Failures on the main branch block everyone, and flaky tests get re-run until nobody trusts the pipeline.

When a CI run fails, a webhook or scheduled check hands the agent the failed job, its logs and the commits since the last green run. The agent pulls out the failing step and error, compares it with recent failures to spot known flaky tests, and checks whether the failing area matches files changed in the new commits. It posts a short triage note on the pull request or in the team channel: the likely category (code change, flaky test, dependency, infrastructure), the evidence, and the suspected commit. For simple, well-scoped cases such as a failing snapshot, a lint error or an obvious type error, it can open a draft pull request with a proposed fix. It never merges, never disables tests, and never re-runs a job more than the limit you set.

1. Start with one repository and its main branch pipeline
2. Trigger the agent on failed workflow runs through your CI provider's webhook or a scheduled check
3. Give it read access to run logs and the repository, and write access only to comments and draft pull requests
4. Keep a simple record of past failures and known flaky tests so it can compare new failures against them
5. Define the categories and what the note must include: failing step, error, evidence and suspected commit
6. Allow at most one automatic re-run for a suspected flaky test, and log every re-run
7. Allow draft fix pull requests only for small, named classes of failure, with CI running on the fix
8. Review its triage notes weekly with the team and correct its categories

• GitHub Actions
• GitLab CI
• Claude Code
• Cursor
• Temporal
• n8n + an LLM

Not ranked. Tools marked Featured pay for a highlighted listing; that never decides which tools we include. Check each vendor's current features and pricing.

Make one of these tools? Feature your tool on every playbook where it is already listed.

Engineers review and merge every fix; the agent only opens drafts. Disabling, skipping or deleting a test is always a human decision. Infrastructure and secrets problems go to the person who owns the pipeline. The team decides when a test is labeled flaky.

An agent that 'fixes' a failing test by weakening it makes the pipeline green and the product worse, so block test deletions and assertion changes in its fixes unless a person asks. Logs can contain secrets; make sure masked values stay masked and the agent's output is not posted anywhere public. Limit re-runs so it cannot burn CI minutes in a loop. Make its long-running work retry safely if a worker crashes, so a half-finished triage does not post twice.

Track your own numbers before and after: time from a red main branch to a diagnosis, how often the triage category was right, re-runs per week, and how many flaky tests get fixed rather than ignored. The gain is faster diagnosis and less log reading, with engineers still owning every change.