The tool that wasn't there, so the agent wrote it

Last time I had two agents doing the same job, one on fixed rules, one asking a language model. The AI agent was free to decide the order of its steps, but it could only pick from tools I had written beforehand. Everything it was able to do, I had guessed in advance. So what happens when it needs something that isn't in the catalog? Normally a polite refusal. I wanted to know what the other answer costs, the one where the agent writes the tool itself, in the running system, without a restart. I built it over a weekend. Getting code out of the model worked after two hours. The rest of the Sunday went into everything that comes after that.

A catalog is a list, and lists end

Tool calling works the same way everywhere. You give the model a list of functions with their parameters, it picks one, you run it, you hand the result back. That's fine, and it has a ceiling that's easy to overlook. The agent can only ever be as capable as my list. For a support bot with eleven known operations, perfect. For an agent that gets handed whatever goal a user happens to type, the list is the limit, and every new entry means a commit, a review, a deploy.

Letting the agent write the missing tool itself sounds careless when you say it out loud, and it is, if you do it the obvious way. Take the model's answer, Code.eval_string/1, done. That works, and it hands a stranger a shell on your server. Which is why hardly any of the work is in the generating. It sits in the gap between the model's answer and the first call.

Five stations

Seen from outside it's one function call.

{:ok, module} =
  AgentDemo.Jido.Dynamic.ensure_action(%{
    name: "slugify_text",
    description: "Turns a title into a URL slug",
    params: %{text: "Hello World"}
  })

Jido.Exec.run(module, %{text: "Hello World"})

Inside there are five stations. Generate, sanitize, persist, compile and load, execute. Each of them returns {:ok, _} or {:error, reason}, and the reasons are specific enough to do something with, {:generation_failed, _}, {:unsafe_ast, _}, {:compile_error, _} and so on. I need that granularity because the UI shows which station failed, and because a model that didn't answer should produce a log line, not an exception.

Five stations and one gate. If the check says no, nothing gets written and nothing gets compiled.

What the model gets told

The prompt is nothing special. It names the exact module the code has to define, asks for use Jido.Action with a parameter schema and a run/2 that returns {:ok, map}, and then lists the rules.

- You may only call these modules: #{module_list()}.
- Forbidden: shell access, ports, processes, message passing, network calls,
  runtime code evaluation, dynamic dispatch (`apply/3`, `mod.fun()`),
  atom creation from strings, and any file system access.
- Code that breaks a rule is rejected by a static checker before it runs.

module_list() reads Sanitizer.allowed_modules/0, the same list the checker enforces, so prompt and enforcement can't drift apart. Saying the rules out loud is worth it, by the way. With them in the prompt most answers pass on the first attempt. Without them most answers reach for File or Task somewhere.

None of that is protection. A prompt is a wish. Everything after it is written as if the model had never read a word of it.

The gate

The sanitizer parses the code with Code.string_to_quoted/1, walks the tree with Macro.prewalk/2 and stops at the first violation. It has two layers.

The obvious one first, a map from {module, function} to a reason.

@denied_calls %{
  {System, :cmd} => "shell execution via System.cmd/2,3",
  {:os, :cmd} => "shell execution via :os.cmd/1",
  {Port, :open} => "external programs via Port.open/2",
  {Code, :eval_string} => "runtime code evaluation via Code.eval_string/1",
  {Kernel, :apply} => "dynamic dispatch via apply/2,3",
  {String, :to_atom} => "atom creation via String.to_atom/1",
  # ...
}

On its own that list isn't worth much. I named a dozen doors. The BEAM has hundreds, :rpc, Node, Task, Process, :persistent_term, :ets, Module, plus whatever the next release adds. A deny list only blocks what its author happened to think of.

So the second layer turns the question around. About two dozen modules may be called at all, Enum, String, Map, Regex, Jason, the date modules, that kind of thing, and everything not on the list is out by default. Task.async/1 doesn't fail because I forbade it. It fails because I never allowed it. I kept the deny list anyway, because it produces a reason worth reading in an audit log. "Task is not on the allow list" says less than "process creation via spawn/1,3 is not allowed".

Then come the structural rules, which took me longer than both lists together. Exactly one defmodule, and with the name I expect. Inside it only use Jido.Action, module attributes and def/defp. No import, no alias, no nested modules, no receive, no quote.

And here's where I sat in iex for a while. A generated action has to be able to write params.text. It must not be able to write mod.run(x) where mod came out of its parameters. In the AST both are the same kind of node, a dot call on something that isn't a literal module.

iex> Code.string_to_quoted("params.text")
{:ok,
 {{:., [line: 1], [{:params, [line: 1], nil}, :text]},
  [no_parens: true, line: 1], []}}

That no_parens: true looked like the whole answer. Field access has it, a real call doesn't. Then I tried a capture.

iex> Code.string_to_quoted("&System.cmd/2")
{:ok,
 {:&, [line: 1],
  [
    {:/, [line: 1],
     [
       {{:., [line: 1],
         [{:__aliases__, [line: 1], [:System]}, :cmd]},
        [no_parens: true, line: 1], []},
       2
     ]}
  ]}}

Same no_parens: true, no arguments either, and System.cmd sitting right in the middle of it. Had I shipped the first version, Enum.map(list, &System.cmd/2) would have walked through untouched. The distinction I actually needed isn't the parentheses, it's what stands to the left of the dot. A literal alias or atom goes through the normal module check. A variable is rejected. Only a variable with no parentheses and no arguments counts as field access.

The whole gate runs in well under a millisecond, so there's never a reason to skip it. A rejection comes back like this, no matter whether the code came from the model or from a file somebody edited on disk.

{:error,
 {:unsafe_ast,
  %{kind: :denied_call, line: 5,
    message: "shell execution via System.cmd/2,3 is not allowed"}}}

None of this makes it a sandbox, and I want to be clear about that. A static check sees what's in the AST and nothing else. It makes an attack considerably more expensive, it doesn't prevent one, and once the module is loaded it can do everything the node can do. So it belongs behind a model you trust, next to somebody who actually reads the persisted files, and next to an operating system that isn't running this as root.

Loading it into a live system

The loader is short enough to quote nearly in full.

def load(module, ast, file) do
  if :erlang.module_loaded(module), do: :code.soft_purge(module)

  with {:ok, compiled} <- compile(ast, file),
       :ok <- ensure_defined(module, compiled),
       :ok <- ensure_action_contract(module) do
    {:ok, module}
  end
end

Code.compile_quoted/2 turns the checked AST into a BEAM binary in memory and hands it to the code server. No build directory, no mix compile, no release, no restart. About 25 milliseconds later the module answers calls, in the same process that asked for it.

The contract check in the third line is where I got something wrong. To be sure a generated module was actually usable I asked function_exported?(module, :run, 2). Green, all good. Then I wrote a test with a module that only defines run/1, expected a contract violation, and it loaded without complaint. use Jido.Action injects its own run/2, one that returns {:error, config_error} if you never override it. My check was asking whether the macro had done its job, and it always has. The guarantee that a run/2 written by the author exists comes from the sanitizer one station earlier, which reads it out of the AST. What's left in the loader catches modules that never used the behaviour at all. Worth having, just not the thing I thought I was testing.

Every version gets its own module name, Elixir.AgentDemo.Actions.Dynamic.SlugifyText.V1, then .V2. The numbers come from an atomic ETS counter, and the approved source ends up in priv/generated_actions/slugify_text_v1.ex, so it can be read, diffed and blamed later. On the next boot those files go through the sanitizer again before anything is loaded. A file that was edited on disk doesn't come back.

Replacing a tool interrupts nobody. V1 stays alive for whoever is inside it and new calls go to V2.

That boot path cost me an evening, and it annoyed me because the code looked right. Restoring ran in the handle_continue of the storage process. Idiomatic, keeps init/1 fast, everybody does it that way. I generated an action, stopped the node, started it again, and the first call to the restored module came back with cannot compile module ... :nofile, which is what you get when a module simply isn't there. The supervisor had returned long before handle_continue was finished, so from the outside the boot was over while the actions were still compiling. Restoring inside init/1 fixes it and blocks the boot for as long as compiling takes, a few hundred milliseconds for a handful of files. I'll take that. The alternative is a window in which an action exists on disk, is registered nowhere and fails on first use, and that's the kind of bug you find in production and not in a test.

Why this is so much easier on the BEAM

This is the actual reason the whole thing is in Elixir and not somewhere else. Not one of the properties below was invented for language models. That's the part I find interesting about them.

Replacing code in a running system is a documented guarantee here, not a workaround. Erlang was built for telephone switches that weren't allowed to stop, so two versions of a module can be loaded at the same time. A process that entered the old one finishes there, new calls land in the new one, and :code.soft_purge/1 frees the old code once nobody's inside it any more. That's from the eighties, and it's why retiring an old version is two lines in my loader. On the JVM you build something similar out of classloaders. In Go you don't. In Python you reload the module and hope nothing kept a reference.

The compiler is part of the runtime, and the AST is ordinary data. Code.string_to_quoted/1 gives you tuples, atoms and lists. Inspecting them is Macro.prewalk/2 and pattern matching on {{:., _, [module, fun]}, meta, args}. The entire gate is 350 lines of unremarkable Elixir, half of it lists and documentation. I didn't write a parser, I didn't go near bytecode, and I could try the tricky cases in iex while writing it, which is how I found the capture problem above. Python gets fairly close with ast and compile. Almost nothing else in production does.

Processes are a boundary I didn't have to build. Jido runs every action in a supervised task with a timeout, thirty seconds by default. Code that crashes takes its own process down and nothing else. There's no shared heap it could leave corrupted and no exception that unwinds through my call stack. "Let it crash" sounds odd as an operating principle until you apply it to a function a model wrote thirty milliseconds ago.

An infinite loop can't take the node with it. The BEAM preempts on a reduction count, not when the running code feels like yielding, so a generated Stream.cycle that never terminates burns one scheduler slot until the timeout kills it. In a runtime with one thread per request the same code is an outage. max_heap_size does the same job for runaway memory.

Two smaller things that add up. A generated action gets a map and returns a map, and there is no reference through which it could quietly change something of mine, so reviewing what a stranger's function can reach is a short job. And the registry of active versions is just a named public ETS table, with concurrent reads that need no lock, atomic counters for the version numbers, no extra dependency, gone when the node is gone. For a registry of code that was generated at runtime, that last part is exactly the lifetime I want.

What the BEAM doesn't give you belongs in here too. There are no capabilities. Once a module is loaded it may call anything the node may call, and there's no switch that says "this module may not use :os". That's the entire reason the gate sits before Code.compile_quoted/2 and not after. The atom table is the other one. It isn't garbage collected, so nothing that came out of a model ever goes into String.to_atom/1. Only String.to_existing_atom/1, after the module is loaded and its schema keys exist as atoms.

Back in the agent loop

All of this is worthless if the new tool arrives as a special case. The agent resolves every planned step against its catalog, and the branch for a missing one is three lines.

defp resolve(step, opts) do
  case Map.fetch(@catalog, step.tool) do
    {:ok, module} -> {:ok, module}
    :error -> Dynamic.ensure_action(spec(step), opts)
  end
end

Either way the result is a module, the module becomes a %Jido.Instruction{}, and that goes into the same chain as everything else. The agent's cmd/2 can't tell a generated action from one I wrote in January, and that was the whole point of doing it this way.

The demo has a page for it where you can watch the five stations light up per run, switch the generator between a deterministic mock and Claude, force a new version to see the swap happen, and paste your own code into a test bench to see what the gate makes of it. Watching System.cmd come back rejected with a line number did more for my confidence than any table.

What it costs

Property	Static tool catalog	Dynamic action
Capability boundary	what I anticipated	what the model can write within the allow list
First call	microseconds	seconds for the model, then ~25 ms to compile
Every call after	microseconds	microseconds, it's a normal module now
Determinism	fixed at deploy time	fixed after generation, but generating isn't
Review	before the deploy	afterwards, on the persisted file
Failure modes	bugs	bugs, plus unsafe code, code that won't compile, wrong schema
Security surface	my code	model code behind a static gate
Right for	everything you can enumerate	the cases you couldn't foresee

Dynamic actions don't replace the catalog, they sit behind it. Anything I'd write twice belongs in the catalog, by hand, reviewed, tested. What this buys me is that the agent isn't capped by what I happened to think of on the day I wrote it.

If you're thinking about building this

Generating the code is the small part. A decent model writes a correct small Elixir module more often than not, and the work is in the gate, the versioning, the loading, the retiring, the boot path and the error taxonomy. Budget accordingly, because I didn't.

The deny list feels productive and buys little. Twenty lines that invert the question, everything forbidden unless explicitly allowed, did more for safety than every hour I spent naming dangerous functions. The deny list is still in there, for the error messages.

And the runtime decides whether this is a reasonable idea at all. The same feature without hot code loading means a build server, an artifact store and a rolling restart, days of work and an operational story nobody wants to own. Here it's roughly 1200 lines and one child in the supervision tree. Not because Elixir is fashionable, but because an inspectable AST, code replacement in a live node, process isolation and preemptive scheduling all exist for reasons that predate language models by decades, and happen to be exactly the set of properties that code written by one requires.

The whole thing is on GitLab at gitlab.com/public_elixir/agent_demo. If one part interests you more than the rest, how the gate tells field access from dynamic dispatch, why versions get their own module names instead of reusing one, or what the boot path does with a file that was tampered with, ask in the comments. I'll go into it.