Who picks the tool

The previous post ended with an agent that writes a missing tool into a running node. Generate, sanitize, persist, compile, load, execute, about 25 milliseconds after the model answers. I was happy with that for roughly a week. Then I looked at the demo page again and counted the text fields. Tool name. What it should do. Parameters as JSON. I had filled in all three.

The gap-finding was real. The deciding was mine.

MetaPlannerAgent.run("Publish the article", [
  %{tool: "classic_plan", params: %{goal: "Write draft, review, publish"}},
  %{tool: "slugify_text", description: "Turns a title into a URL slug",
    params: %{text: "Hello World"}}
])

Everything interesting in that call sits in the second argument, and I typed it. The agent resolved slugify_text, noticed it wasn't in the catalog and had it written. It never asked whether the task needed a slug at all.

So this post is about the agent in front of that, the one that gets a sentence and works out the rest. Which tools the task needs, which of them already exist, which have to be written first. The machinery from the last post is still down there, and from up here it is one branch of a cond.

The agent, in one function

def run(task, opts \\ []) do
  context = %{tools: available_tools(), history: Keyword.get(opts, :history, [])}

  with {:ok, plan} <- ToolPlanner.plan(task, context, opts) do
    agent = new(state: %{task: task, reasoning: plan.reasoning})
    set(agent, %{steps: Enum.map(plan.steps, &run_step(&1, agent, opts))})
  end
end

Plan, then run every planned step. The interesting line is the one that decides where a step's tool comes from, and it has three answers.

defp origin(tool) do
  cond do
    Map.has_key?(MetaPlannerAgent.catalog(), tool) -> :catalog
    match?({:ok, _entry}, Storage.lookup(tool)) -> :reused
    true -> :generated
  end
end

Hand written, generated in an earlier turn, or not there at all. Only the third one wakes up the pipeline from the last post. After that the three are indistinguishable: each becomes a %Jido.Instruction{} and goes through the agent's cmd/2 like everything else, which is the same trick as before, one level up. The agent doesn't have a special case for code that was written thirty seconds ago.

What comes back is one entry per step.

%{tool: "celsius_to_fahrenheit", origin: :generated,
  params: %{"celsius" => 30}, result: %{fahrenheit: 86.0}, error: nil}

:origin is the only field that remembers the difference, and it exists for the user interface, not for the logic.

Two model calls, three ways a step can resolve, one instruction chain at the end.

Two models, two jobs

The first thing I got wrong was assuming this was one prompt. Deciding which tool a task needs and writing the Elixir for it are different problems. Different context, different failure modes, different things to check afterwards. They're now two calls behind two behaviours.

@callback plan(task :: String.t(), context :: context()) ::
            {:ok, plan()} | {:error, term()}

The planner returns data and nothing else. It never generates, never compiles, never runs anything. What it hands back is a list of steps with concrete parameters, and an empty list is a legitimate answer.

{:ok, %{
  reasoning: "...",
  steps: [
    %{tool: "celsius_to_fahrenheit",
      description: "Converts degrees Celsius to degrees Fahrenheit",
      params: %{"celsius" => 30}}
  ]
}}

Each backend has a mock next to it, same as the code generator, so the whole demo runs without an API key. The mock planner matches keywords and pulls parameters out of the text, which is deliberately dumb, and the contrast to the real one is the point of having both.

One detail took several attempts. The plan comes back as forced JSON, and params is a map with keys nobody knows in advance. A free-form object is awkward to pin down in a strict output schema, and the obvious escape, a list of key/value strings, turns 30 into "30". So the params travel as a JSON string that I decode on arrival. Ugly in the schema, correct in the types, and the types turn out to matter more than I expected.

Reuse is the whole point, and where it broke

A planner that can't see what already exists is a version generator. Ask for a temperature conversion twice, phrase it differently, and you get celsius_to_fahrenheit and then celsius_converter sitting next to each other, both generated, both persisted, neither reused. Reuse happens by name, so the planner has to be shown the names.

def available_tools do
  catalog = Enum.map(MetaPlannerAgent.catalog(), ...)   # hand written
  dynamic = Enum.map(Storage.list(), ...)               # generated earlier
  Enum.sort_by(catalog ++ dynamic, & &1.name)
end

The static catalog plus the registry from the last post, each with its description and its parameter names. That's the context every planning call gets, and it is the whole of what makes the second turn cheaper than the first.

Then the second message in my own chat failed.

Rechne 30 Grad Celsius in Fahrenheit um

celsius_to_fahrenheit   reused   %{"celsius" => 30}
Invalid parameters for Action (...CelsiusToFahrenheit.V1):
invalid value for :celsius option: expected float, got: 30

The first time I asked, it worked. The tool was generated in that moment, and Claude had written celsius: [type: :float, required: true]. The second time, the same integer met the same schema and bounced.

It took me longer than it should have to see why the first call can't fail this way. A freshly generated tool gets its schema from the very values you passed it. The example parameters go into the prompt, the model reads the types off them, and whatever it declares fits what is about to be sent. Generation and first call are shaped by the same input, so they agree by construction. Reuse is the first time a tool meets a value it didn't shape. Every type mismatch in the system was invisible until the agent started reusing things, and then it arrived on message two.

The first call of a generated tool cannot disagree with it. The second one can.

The fix is small and only goes one way. Integers widen to floats where the loaded schema asks for a float. Floats do not narrow to integers, because that discards data and I'd rather see the error. It reads like a papered-over bug and I went back and forth on it, but the alternative is teaching every planner, including ones I haven't written, to guess Elixir's numeric tower from a German sentence. The action's schema is right there at call time. Use it.

The diff that lied

Steps run one at a time so each keeps its own result. The chat needs that, because a task that resolves to three tools and shows one answer is useless.

My first version ran them cumulatively, the way the old agent did, and worked out what each step had contributed by diffing the agent state before and after. That's fine until two steps return the same thing.

Zähle die Wörter in „Hallo Welt aus Elixir" und mach einen Slug daraus

count_words     %{result: "text=Hallo Welt aus Elixir", tool: "count_words"}
slugify_text    %{tool: "slugify_text"}

The second step looks like it returned almost nothing. It didn't. It returned the same result value as the first one, the diff saw no change and dropped the key. A bug that only appears when two steps agree, which in a demo with a mock generator is most of the time, and in production would be the rare report that makes no sense.

Now every step runs against the same base state and its result is simply what that state gained. No diffing, less code, and a test that runs the identical step twice and asserts both results are complete. The steps share no state any more, which is a real limitation, and I'll come back to it.

The other thing the chat forced: a failing step must not take the working ones with it. Each step carries its own :result or its own :error, and only a planner that doesn't answer at all fails the whole message.

Making the invisible decision visible

Every step in the chat gets a badge saying where its tool came from. Built in, reused, generated just now. It's three lines of markup and it's the part I'd keep if I had to throw the rest away.

Without it, "the agent handled it" is all you see, and you can't tell the run that reused a compiled module in microseconds from the one that spent seconds in the model and wrote a file to disk. With it, you watch the second message in a session get faster than the first, and you notice when the planner invents a new name for something that already exists. Both are things I would otherwise have found in a log, three days later, by accident.

The agent carries the previous turns into the next planning call, which is what makes "and now in Kelvin" resolvable at all. Only a real model does anything with them; the keyword matcher ignores the history completely, and I left that visible rather than faking it.

The boundary the agent plans against

Lese die Seite mit der URL https://jido.run/ und gib den title oder den h1 aus

Fetching a web page requires network access, which a tool here is not allowed to perform. Tools must be pure, deterministic functions without external access. If you give me the HTML source, I can extract the title and h1 from it.

Nothing was generated. Nothing was compiled. The agent declined at planning time, and that's the property I actually care about: it knows what it is allowed to build, because the planner's prompt and the sanitizer's allow list say the same thing. When those two drift apart, the agent cheerfully plans tools that get rejected three stations later, and the person in the chat gets a compile error instead of an answer.

Which is why widening what a tool may do is two edits and not one. I widened it to reading a page: one small module of mine went on the allow list with a single get/1 on it, so a generated tool can do a GET and nothing else, and the prompt line that used to forbid network calls now names that function instead. Gate and prompt moved together. The next message produced a tool that fetches the page and pulls <title> and <h1> out with two regexes. Title Build AI Agents That Run in Production · Agent Jido. The <h1> came back as Search, because the first one on that page belongs to a search box, which is what two regexes get you and exactly the sort of thing you'd sharpen in the description on the next turn.

It also changed what acquiring a tool costs. Before handing a new action out, the pipeline calls it once with the example parameters, which for a fetch tool is a real request, so generating one hits the target site twice. And that trial call's timeout, five seconds back when the worst case was an infinite loop, now sat below my HTTP client's ten. The first fetch tool I generated was killed halfway through its trial and rejected as hanging, twice, before I noticed which of the two numbers was wrong.

I'd rather be precise about what this bought. A GET reaches everything the node reaches, including the internal service or metadata endpoint that happens to be one hop away, and a query string moves data outwards perfectly well. The URL comes from a model. Restricting where it may point belongs on the network, not in an AST checker.

What it still can't do

Steps don't chain. Step two cannot see what step one returned. That's why the fetch tool has to fetch and parse in one function, and why the planner's prompt now says so in as many words.

I left it that way on purpose, for now. Chaining means the planner has to refer to an earlier result, which means results need stable names, which means the plan stops being a flat list and starts being a small dataflow graph, and the failure modes multiply: a step referencing an output that never arrived, a type that changes shape between steps, a cycle. It's the obvious next post. It is not a small change dressed up as one.

Who decides what

	Static catalog	Dynamic action (last post)	Task agent (this post)
Which tool runs	I decide, at deploy time	I decide, per run, in a form	the model decides, per message
Where the tool comes from	written by hand	generated on demand	catalog, registry or generated
Reuse	trivially, it's a module	manual, if I type the same name	by name, from the registry
When types are checked	at deploy	at first call, which can't disagree	at reuse, which can
Visible to the user	nothing to see	which station failed	which tool, and where it came from
Cost per message	microseconds	seconds, once	one planning call, plus generation for anything new

If you're thinking about building this

Split the two model calls. Choosing a tool and writing one are separate jobs and they fail differently. Keeping them apart gave me two prompts I could tune independently, two mocks, and error messages that say which half went wrong.

Show the planner what already exists, keyed by name. Reuse doesn't happen because the model is clever, it happens because the name it picked matched something in the registry. Everything else is a new version on disk.

The first call of a generated tool proves less than it looks like. It was shaped by the values you passed, so it agrees with them by construction. The second call, with a value the tool didn't shape, is the first real test, and if your system never reuses anything you will never run it.

Keep the planner's prompt and the enforcing gate in sync, and treat a refusal at planning time as a success. An agent that declines before generating anything is cheaper, faster and far easier to explain than one that produces a rejected module three stations downstream.

Give every step its own result and its own error. It sounds like a display concern until the message that resolved into three tools fails on the last one and throws away the two that worked.

The whole thing is on GitLab at gitlab.com/public_elixir/agent_demo. If the chaining question is the one you'd want next, or you think the float widening is a mistake, say so in the comments. I'm not certain about that one either.