I built an open-source agent that investigates failed Databricks jobs. It runs inside the workspace, as two Databricks Jobs and a Databricks App.
What it does:
- Collects read-only evidence for a failed task: error and stack trace, run history, table history and schema changes, lineage, permission changes, upstream job status, and the code around the failing line.
- Classifies the failure: code or configuration error, schema change, data issue, upstream failure, infrastructure, permissions, library installation, or resource limit.
- Creates an incident record and a Markdown report. Every claim in the report links to its supporting evidence.
How it works:
- The confidence level (LOW, MEDIUM, HIGH) is calculated in code from weighted evidence.
- The LLM suggests hypotheses, requests additional checks, and writes the report text. If the model fails, the result still comes from the rules, and the report is written from a template.
- No tool reads table rows. Error messages and code lines are masked for secrets and literals before they are sent to the model.
- Three ways to start an investigation: a handler task in the job (push), a scheduled poller, and a button in the app. A failure is never investigated twice.
Repository: github.com/mohsenim/databricks-agent-investigator
