At Fiatbitcoin, around two dozen small agents run in the background. They fetch prices, read the mempool, score news, compute tax deadlines. Where a lot happens, things break. Until now a sensor caught those errors and sent me a Telegram message. Handy, but that is where the actual work started: read the stacktrace, find the file, understand what went wrong.
The question was whether the system could go one step further. Not just report that something is broken, but analyze the error and hand me a finished diagnosis I can work from directly. One thing up front, so nobody gets the wrong idea: the system does not write code and does not merge anything on its own. It detects, analyzes, and opens a GitLab issue. The rest is on me.
The starting point: a sensor that sees errors
The foundation was already there. An ErrorTrackerSensor polls the table of error_tracker and emits a signal for every new occurrence onto a PubSub topic. The signal carries everything that matters:
data: %{
occurrence_id: occurrence.id,
error_id: occurrence.error_id,
error_kind: occurrence.error.kind,
error_reason: occurrence.error.reason,
error_fingerprint: occurrence.error.fingerprint,
context: occurrence.context,
captured_at: occurrence.inserted_at
}
A Telegram consumer already sits on this signal. The new mechanism simply parks next to it, without touching the existing one. A second listener on the same topic that kicks off the pipeline.
A pipeline with brakes
My first instinct was a single agent that does everything. I dropped that quickly. The steps differ too much in their risk to throw them into one pot.
Triage only reads. The analysis also only reads, but it lets a language model weigh in. And the step that is one day supposed to produce real code is the most dangerous of them all. These steps deserve different treatment.
So I chained them one after another, cleanly separated. Each stage may abort, and when it does, the next one never starts. In Elixir that is a with that almost reads itself:
def heal(data, context \\ %{}) do
occurrence = Map.get(context, :occurrence) || load_occurrence(data.occurrence_id)
params = %{data: data, occurrence: occurrence}
with {:ok, %{decision: :proceed}} <- Triage.run(params, context),
{:ok, analysis} <- Analyze.run(params, context),
{:ok, result} <- ReportIssue.run(%{data: data, analysis: analysis}, context) do
{:ok, result}
else
{:ok, %{decision: :skip, reason: reason}} -> {:ok, {:skipped, reason}}
{:error, reason} -> {:error, reason}
end
end
Triage, analysis, report. Whoever bails out in the middle returns cleanly, without anything expensive or risky happening further down. That is the most important property of the whole construction.
Triage: most errors should stay out
The first stage is deliberately dumb and free of any magic. Its job is to reject almost everything. An automatic diagnosis run for every single occurrence would quickly be more noise than value.
def run(%{data: data} = params, _context) do
occurrence = Map.get(params, :occurrence)
cond do
not supported_kind?(data.error_kind) -> skip(:unsupported_kind)
not actionable_source?(occurrence) -> skip(:no_actionable_source)
already_reported?(data.error_fingerprint) -> skip(:already_reported)
true -> {:ok, %{decision: :proceed}}
end
end
Three filters, in this order. First the error class against a small whitelist of analyzable Elixir errors. Then the question of whether the stacktrace points at our own code at all and not into a dependency. Finally a check against GitLab for whether an open issue already exists for the same fingerprint. Without that last filter, an error firing every second would trigger an avalanche of identical issues.
A skip here is not a failure, it is a normal outcome. The module returns {:ok, %{decision: :skip, reason: ...}} and the pipeline ends quietly.
Analysis: read, do not guess
Once an error makes it through triage, the second stage gathers context. It pulls the stacktrace, looks up which file and function blew up, and reads the affected code section if it can. On top of that it asks the system's memory whether this fingerprint has shown up before. Only then does the collected material go to a language model that is supposed to formulate a root-cause hypothesis.
It mattered to me that this stage also works without a language model. If no API key is set or the call fails, the pipeline should not just grind to a halt. A stacktrace plus the error location is valuable even without prose:
hypothesis =
case llm.generate_text(build_prompt(diagnostic), @client_name, max_tokens: 800) do
{:ok, text} ->
String.trim(text)
{:error, reason} ->
Logger.info("Self-healing analysis without LLM (#{inspect(reason)})")
nil
end
If the model drops out, the hypothesis is simply nil, and the issue carries an honest note about it. The diagnosis gets thinner, but it still arrives.
The exit: an issue, not a merge
Here I had to make a decision that ran against the original idea at first. I wanted a merge request at the end that I would just wave through. Except a merge request needs a branch with a change. As long as the system produces no code, there is nothing to merge.
So in the current state the last stage opens an issue, not a merge request. The issue contains the fingerprint, the stacktrace, the hypothesis, and a note that it needs manual review. The client behind it is a thin wrapper over the GitLab REST API built on Req. If the configuration is missing, it bows out cleanly instead of dragging the pipeline down with it:
defp config do
cfg = Application.get_env(:integration_gateway, :gitlab, [])
token = cfg[:token]
project_id = cfg[:project_id]
if is_binary(token) and token != "" and not is_nil(project_id) do
{:ok, %{token: token, project_id: project_id, base_url: cfg[:base_url] || "https://gitlab.com"}}
else
{:error, :not_configured}
end
end
As soon as the actual repair stage arrives, the issue becomes a merge request. The function for that is already sitting in the client. By the way, I opened the very first merge request for this feature with exactly that function, which was a nice little bit of dogfooding.
Why deliberately no auto-merge
It would be tempting to close the loop: detect, fix, merge, done. I think that is a mistake, at least here and at this level of maturity.
Code produced by a language model is a guess. Before something like that lands in a branch, there needs to be a hard check in front of it that decides deterministically and without a model: does this compile, do the tests pass green. Only what clears that check is even allowed to get a merge request. And the human at the end of the chain stays. Not as a stopgap, but as a feature. The planned next step runs the repair in an isolated Git worktree, checks it against the compiler and the test run, and opens a merge request only on a green result. Auto-merge is deliberately not on the list.
Lessons learned
Three things from building this, one of them something I would sharpen up.
1. Abort points beat cleverness.
The whole pipeline lives off aborting early and often. No clever do-it-all, but four stations, each of which may say no. That makes the behavior predictable and ensures the expensive steps only run when the cheap ones have waved them through.
2. A good diagnosis is already half the fix.
I was disappointed at first that the first iteration only produces an issue and no patch. In practice the diagnosis is exactly what eats the time. When the issue carries the fingerprint, the stacktrace, the affected function, and a plausible cause, the actual fix is often a matter of minutes. The step from issue to merge request is smaller than it looks.
3. What I would sharpen up.
Deduplication currently runs only over open issues. A dedicated time-based cooldown per fingerprint is still missing, as is the real repair stage with its check via compiler and test run. Both are described in the concept but not built yet. I would rather ship an honest small step that works cleanly than a big promise that falls apart in production.
If a particular spot interests you, the routing over the error signal, the triage filters, or the planned check in the worktree, drop it in the comments. I am happy to go deeper.
Kommentare