Splitting an ML model and a web app across two BEAM nodes — the technical blueprint

The web app (Phoenix + Postgres) runs in production on a small Linux box with no GPU. The embedding model (intfloat/multilingual-e5-base, ~278 MB, XLM-RoBERTa backbone, EXLA-compiled) wants compute that isn't there. That compute lives in a MacBook Pro. Goal: offload only the heavy inference to the Mac, keep everything else on the server — and the calling code shouldn't know where the math happens.

The usual answer would be a second service with a gRPC/REST API, serialization, service discovery, timeouts, retries, and a second deployment artifact. On the BEAM, "another machine" is a platform primitive instead. What it takes isn't new infrastructure, but a clean split of one codebase into two releases and three node roles.

The mental model: one codebase, two releases, three roles

There is exactly one codebase (an umbrella project) and no second language. The only thing that varies is what a started node brings up. A single environment variable, FINANCE_ROLE, controls that:

Role	Node	Starts	FINANCE_ROLE
:web	Linux	Phoenix, Postgres, Jido, classification logic	web
:ml	MacBook	only the Nx.Serving (model + EXLA)	ml
:all	Dev/Test	everything in one node	(default)

Important: the web node brings up the classification logic (label vectors from the DB, cosine similarity); it offloads only the one heavy step — embedding the text into a vector — to the ML node. The ML node has no database, no Phoenix, no DATABASE_URL, and no SECRET_KEY_BASE.

Three building blocks make this work, in this order:

1. Dependency partitioning — which libraries end up in which release at all.
2. Role-based supervision tree — what each node starts at runtime.
3. Distributed Nx.Serving — how the inference call is routed across nodes without the caller noticing.

Building block 1: partitioning dependencies with runtime: false

This is the part that's least obvious and most important. The ML stack (bumblebee, exla, tokenizers) ships architecture-specific NIFs: libexla, libex_tokenizers. These are compiled for the machine the release is built on. A libexla built on macOS (arm64) simply won't load on a Linux x86_64 host. So it must not end up in the web release in the first place.

The lever for that is runtime: false in the mix.exs of the finance app:

# apps/finance/mix.exs
defp deps do
  [
    {:libcluster, "~> 3.5"},
    {:ecto_sql, "~> 3.13"},
    {:postgrex, ">= 0.0.0"},
    # ...
    # nx runs on BOTH nodes: the web node needs it for the cosine
    # similarity and the distributed Nx.Serving.batched_run. Declared
    # directly so it stays in the web release even though its usual
    # sources (bumblebee/exla) are runtime: false.
    {:nx, "~> 0.12"},
    # The whole ML stack only runs on the finance_ml node. runtime: false
    # keeps these — and their arch-specific NIFs (libexla, libex_tokenizers) —
    # out of the OTP application closure, so they don't ship in the web release.
    {:bumblebee, "~> 0.6", runtime: false},
    {:exla, "~> 0.9", runtime: false},
    {:tokenizers, "~> 0.5", runtime: false, override: true},
    {:jido, "~> 2.2"}
  ]
end

runtime: false means: the dependency is used at compile time, but not included in the OTP application closure that mix release computes for the boot order. So it isn't auto-started and — crucially — isn't packaged into the release by default.

nx is deliberately a normal dependency, because it runs on both nodes: the web node needs Nx for the cosine similarity and for the batched_run call itself. Only the heavy compilation (EXLA) and model loading (Bumblebee) are exclusive to the ML node.

Two release definitions

The release definitions in the umbrella mix.exs draw the line:

# mix.exs (umbrella root)
defp releases do
  [
    # Web node (Linux): Phoenix + DB + classifier orchestration.
    finance_web: [
      applications: [
        finance: :permanent,
        finance_web: :permanent
      ],
      include_erts: false,
      steps: [:assemble, :tar]
    ],
    # ML node (MacBook): Nx.Serving only, no web/DB. Excludes finance_web,
    # so no endpoint ever starts here. Pulls the ML stack back in explicitly
    # (bumblebee/exla/tokenizers are runtime: false in finance); bumblebee
    # transitively brings axon/safetensors/etc. along.
    finance_ml: [
      applications: [
        finance: :permanent,
        bumblebee: :permanent,
        exla: :permanent,
        tokenizers: :permanent
      ],
      include_erts: true,
      steps: [:assemble, :tar]
    ]
  ]
end

Two things are subtle here:

• finance_web contains finance_web (Phoenix), but NOT the ML stack. Because bumblebee/exla/tokenizers are runtime: false, they're not in finance's closure and therefore don't land in the web release. That's exactly what we want: no macOS NIFs on the Linux host.
• finance_ml contains finance (for the serving definition and the role logic), the ML stack — but NOT finance_web. No Phoenix endpoint ever starts on the ML node. By listing bumblebee, exla, and tokenizers explicitly in applications:, we pull them back into exactly this release despite runtime: false.

include_erts: why the two releases differ

• finance_web: include_erts: false. The release ships no Erlang runtime; the Linux host must provide a compatible Erlang/Elixir version.
• finance_ml: include_erts: true. The ML node runs on the same machine it's built on (the Mac), so bundling the ERTS is safe — and it avoids depending on whichever Erlang the run directory resolves via asdf/PATH. A mismatch there leads to a load_failed of kernel/stdlib at boot.

The build consequence: build once per architecture

Because NIFs and (for finance_ml) the ERTS aren't portable: each release is built on its target architecture. finance_web on a Linux x86_64 host, finance_ml on macOS arm64. Cross-compiling is not a path this setup takes.

One gotcha while building: mix release --overwrite leaves stale app directories behind (e.g. exla, which is excluded from finance_web). So the release directory is wiped before every build:

release_web:
	rm -rf _build/prod/rel/finance_web
	MIX_ENV=prod mix release finance_web

release_ml:
	rm -rf _build/prod/rel/finance_ml
	MIX_ENV=prod mix release finance_ml

Building block 2: the role-based supervision tree

At runtime, Application.start/2 reads the role and assembles the supervision tree from three parts. It's the same binary logic in both releases — only the role decides which children start:

# apps/finance/lib/finance/application.ex
def start(_type, _args) do
  role = role()

  children =
    cluster_children() ++ web_children(role) ++ serving_children(role)

  case Supervisor.start_link(children, strategy: :one_for_one, name: Finance.Supervisor) do
    {:ok, _} = ok ->
      if role in [:web, :all] do
        if Application.get_env(:finance, :seed_default_labels, true) do
          Finance.Transactions.ensure_default_labels()
        end

        ensure_classification_agent()
      end

      ok

    other ->
      other
  end
end

defp role, do: Application.get_env(:finance, :role, :all)

The three helpers make the split explicit:

# Everything except the Nx.Serving. The classification logic (label embeddings,
# cosine similarity) lives here — it offloads only the embedding call to the
# ml node via distributed Nx.Serving.batched_run.
defp web_children(:ml), do: []

defp web_children(_role) do
  [
    Finance.Repo,
    {Phoenix.PubSub, name: Finance.PubSub},
    Finance.Jido
  ]
end

# The Nx.Serving (model load + EXLA). Started on the ml node, and on :all when
# a real classifier is configured (Bumblebee in dev, Stub → nothing).
defp serving_children(:web), do: []
defp serving_children(:ml), do: [Finance.Classifier.Bumblebee]

defp serving_children(:all) do
  case Application.get_env(:finance, :classifier) do
    nil -> []
    Finance.Classifier.Stub -> []
    module -> [module]
  end
end

Read it as a matrix:

Child	:web	:ml	:all
Finance.Repo (Postgres)	✅	❌	✅
Phoenix.PubSub, Jido	✅	❌	✅
Nx.Serving (Bumblebee)	❌	✅	(config-dep.)
Seed default labels	✅	❌	✅
Start classification agent	✅	❌	✅

So the web node has the full app, but no Nx.Serving. The ML node has only the Nx.Serving. In dev/test (:all) everything runs in one process — with the Stub classifier in tests, so no model has to be loaded.

Building block 3: distributed Nx.Serving — location transparency

Now the question that ties it all together: if the web node starts no serving, how does its classification code call inference on the ML node?

The answer: no differently than locally. Nx.Serving is location-transparent by design. A started serving registers under its name in a process group that's visible cluster-wide. batched_run/2 resolves the name to a serving member — whether local or on another node — and routes the input there. The calling code is identical in both cases:

# apps/finance/lib/finance/classifier/bumblebee.ex
#
# Single entry point to the Nx.Serving — prepends the E5 prefix.
# The serving may run on a remote node. When that node is unreachable,
# batched_run *exits* with :noproc rather than raising. We translate that
# into a tagged throw so classify/1 can report {:error, :classifier_unavailable}
# — distinct from a genuine failure.
defp embed(text, prefix) do
  Nx.Serving.batched_run(@serving_name, prefix <> text)
catch
  :exit, reason -> throw({:classifier_unavailable, reason})
end

@serving_name is the same atom on both nodes (...Bumblebee.Serving). Only the string travels over the wire (the normalized transaction with a "query: " prefix); the embedding tensor comes back. All the matrix multiplication happens on the Mac.

On the ML node, the serving is started in the usual way:

def start_link do
  # exla/bumblebee/tokenizers are runtime: false, so they aren't
  # auto-started. Start them explicitly here — the only code path that
  # needs the model + EXLA compiler.
  {:ok, _} = Application.ensure_all_started(:exla)
  {:ok, _} = Application.ensure_all_started(:bumblebee)

  :persistent_term.erase(@label_cache_key)

  repo = hf_repo(@model_repo)
  {:ok, model_info} = Bumblebee.load_model(repo)
  {:ok, tokenizer} = Bumblebee.load_tokenizer(repo)

  serving =
    Bumblebee.Text.text_embedding(model_info, tokenizer,
      output_attribute: :hidden_state,
      output_pool: :mean_pooling,
      embedding_processor: :l2_norm,
      compile: [batch_size: 1, sequence_length: 160],
      defn_options: [compiler: EXLA]
    )

  Nx.Serving.start_link(
    serving: serving,
    name: @serving_name,
    batch_size: 1,
    batch_timeout: 100
  )
end

Note the Application.ensure_all_started(:exla) / (:bumblebee): because these deps are runtime: false, the VM doesn't start them automatically. This code path — the only one that needs the model and the EXLA compiler at all — starts them explicitly. On the web node, start_link/0 is never called, so EXLA/Bumblebee are never touched there (and, as noted, aren't even in the release).

The web node still needs Nx — but no EXLA

The web node computes the cosine similarities locally (Nx.dot/2 on two 768-dim vectors is trivial). The pure Elixir backend is enough for that; EXLA would be pointless and isn't installed. runtime.exs sets that:

# config/runtime.exs
if role == :web do
  config :nx, :default_backend, Nx.BinaryBackend
end

Connecting nodes: libcluster, cookie, EPMD

For batched_run to resolve the name across nodes, the nodes must be a connected Erlang cluster. libcluster handles that with the static Epmd strategy. The peer names come as a comma-separated list from the environment:

# config/runtime.exs
cluster_hosts =
  System.get_env("FINANCE_CLUSTER_HOSTS", "")
  |> String.split(",", trim: true)
  |> Enum.map(&String.trim/1)
  |> Enum.reject(&(&1 == ""))
  |> Enum.map(&String.to_atom/1)

if cluster_hosts != [] do
  config :libcluster, :topologies,
    finance: [
      strategy: Cluster.Strategy.Epmd,
      config: [hosts: cluster_hosts]
    ]
end

The Cluster.Supervisor is only started when a topology is configured — which keeps dev/test single-node:

# application.ex
defp cluster_children do
  case Application.get_env(:libcluster, :topologies) do
    nil -> []
    topologies -> [{Cluster.Supervisor, [topologies, [name: Finance.ClusterSupervisor]]}]
  end
end

The rest of the distribution plumbing is standard release variables. From the local two-node smoke test (both on the same Mac) in the Makefile:

COOKIE = secret
WEB_NODE = finance_web@127.0.0.1
ML_NODE  = finance_ml@127.0.0.1
CLUSTER_HOSTS = $(WEB_NODE),$(ML_NODE)

# Start the ml node FIRST, so the serving is up before the web node
# classifies. The ml node downloads the model (~278 MB) and pays the
# EXLA compile on first boot.
run_ml:
	FINANCE_ROLE=ml \
	RELEASE_DISTRIBUTION=name \
	RELEASE_NODE=$(ML_NODE) \
	RELEASE_COOKIE=$(COOKIE) \
	FINANCE_CLUSTER_HOSTS="$(CLUSTER_HOSTS)" \
	_build/prod/rel/finance_ml/bin/finance_ml start_iex

run_web:
	FINANCE_ROLE=web \
	RELEASE_DISTRIBUTION=name \
	RELEASE_NODE=$(WEB_NODE) \
	RELEASE_COOKIE=$(COOKIE) \
	FINANCE_CLUSTER_HOSTS="$(CLUSTER_HOSTS)" \
	PHX_HOST=localhost PORT=4000 \
	SECRET_KEY_BASE=... \
	DATABASE_URL=ecto://postgres:postgres@localhost/finance_dev \
	_build/prod/rel/finance_web/bin/finance_web start_iex

The essentials:

• A shared RELEASE_COOKIE on both nodes — the Erlang distribution protocol's authentication secret. Without an identical cookie, the nodes won't connect.
• RELEASE_DISTRIBUTION=name + long names (finance_web@127.0.0.1).
• FINANCE_CLUSTER_HOSTS lists both nodes so libcluster finds them.
• Boot order: ml node first, so the serving is registered before the web node fires the first batched_run. (If it isn't, the fallback below kicks in — no crash.)

If a firewall sits between real machines, EPMD (port 4369) and a fixed distribution port must be open (via -kernel inet_dist_listen_min/max in a rel/vm.args.eex). In the local setup that isn't needed.

Failure handling: "honestly unsure" at the infrastructure level

The critical question for any distributed inference: what if the other node is gone? A naive distributed app would crash or hang. Nx.Serving.batched_run doesn't raise here, it exits with :noproc (no registered serving process found). We catch exactly that and translate it into a distinct state rather than an error:

def classify(text) when is_binary(text) do
  case label_embeddings() do
    [] -> {:ok, @fallback_label}
    embeddings ->
      %{embedding: text_embedding} = embed(text, @query_prefix)
      # ... cosine similarity against all label vectors, best score ...
  end
rescue
  e -> {:error, e}
catch
  # Thrown by embed/2 when the (possibly remote) Nx.Serving is unreachable.
  :throw, {:classifier_unavailable, _reason} -> {:error, :classifier_unavailable}
end

The behaviour makes that state a contract:

# apps/finance/lib/finance/classifier.ex
# {:error, :classifier_unavailable} is reserved for "the serving could not be
# reached" (e.g. the finance_ml node is down). Callers treat it as "try again
# later," not as a classification result.
@callback classify(text :: String.t()) ::
            {:ok, label :: String.t()}
            | {:error, :classifier_unavailable}
            | {:error, term()}

The result: if the MacBook is off, the transaction is left unclassified instead of being filed incorrectly. It's the same principle as the similarity threshold at the model level ("rather Sonstiges than confidently wrong"), one layer deeper: rather pending than wrong.

The fact that classification runs asynchronously through a Jido agent anyway (one transaction per pass, re-triggering itself) makes the case cheap: an unreachable ML node never blocks a web request, and once the node is back, the next pass simply goes through.

Migrations in the release

Since mix is absent in a release, Ecto migrations run via eval on the web node (the ML node has no DB):

bin/finance_web eval "Finance.Release.migrate()"

eval starts the runtime config (so DATABASE_URL must be set) but doesn't boot the supervision tree — the migration runs against a one-off connection that's torn down when the eval finishes.

The bottom line: what this costs and what it doesn't

What this setup costs in terms of "distributed system" is modest and almost entirely contained in this article:

• a few lines of role logic in the supervision tree,
• runtime: false on three deps plus two release definitions,
• one catch for the unreachable node,
• a handful of env variables (cookie, distribution, cluster hosts).

What it doesn't cost: no API definition, no serialization layer, no service mesh, no second language, no second repo. The calling code (Nx.Serving.batched_run/2) changes by exactly zero lines between "all local" and "model on the Mac."

The honest limits:

• Cross-arch builds are manual. Each release must be built on its target architecture. There's no CI pipeline abstracting that away — deliberately, for a project this size.
• Static topology. FINANCE_CLUSTER_HOSTS lists fixed node names. For a dynamically scaling cluster (Kubernetes etc.), a different libcluster strategy (DNS, Gossip) would be the right call — here it would be ballast.
• A single ML node. There's no load balancing across multiple ML nodes. Nx.Serving would support that out of the box via the process group, but the setup doesn't need it (yet).

The real point: distributed ML inference is treated in many stacks as a standalone infrastructure project. On the BEAM, "compute that elsewhere" is a platform building block that's already there before you start — fault tolerance, distribution, and concurrency included. You don't pay for the split with architecture, but with configuration.