Home › Software Engineering

Investigate production errors from logs

DifficultyHard
SetupSome code
Best forDevOps
ToolsDatadog Bits AI, PagerDuty AI, Claude Code with MCP servers, Custom agent
The problem

On-call engineers spend the first part of an incident just gathering logs and recent deploys.

How the agent works

When an alert fires, the agent pulls related logs, traces, and recent deploys, correlates them, proposes a likely cause, and posts findings in the incident channel.

Setup steps
  1. Give read-only access to logs and deploy history
  2. Trigger on alerts
  3. Ask for evidence-backed hypotheses
  4. Post to the incident channel
  5. Humans decide remediation
Tools that can do this
  • Datadog Bits AI
  • PagerDuty AI
  • Claude Code with MCP servers
  • Custom agent

Not ranked and not sponsored. Check each vendor's current features and pricing.

Human in the loop

No write actions to production without approval.

Risks and guardrails

Read-only scopes only; agents can be confidently wrong about causes.

ROI notes

Shortens time-to-diagnosis during incidents.

Get a first-pass review on every pull request

Reviewers are a bottleneck and miss routine issues.

Software EngineeringEasyNo-code

Hand well-scoped issues to a coding agent

Small bugs and chores sit in the backlog because nobody has time.

Software EngineeringMediumCode

Write missing unit tests for legacy code

Untested legacy code makes every change risky.

Software EngineeringMediumCode