At Fiatbitcoin the portfolio needs to know what is happening on the
users’ Bitcoin addresses. Which payment came in, what is still
unconfirmed, what the history looks like. The convenient route there is
a block explorer: mempool.space, blockchain.info, one REST call, done.
Except that explorer learns, in the process, which addresses belong to
which user. For an app whose whole pitch is sovereignty, that is the
wrong trade-off.
So we talk directly to our own Electrum server, which sits on our own
Bitcoin node. No third party in the loop. The catch: Electrum does not
speak HTTP. It speaks JSON-RPC over a persistently open TCP/SSL socket,
with notifications the server sends on its own. That is exactly where
Req, our usual HTTP client, stops being useful. This
article is about the core of the small client I built for it: how to
keep many open requests apart over a single socket, tell replies from
server pushes, and translate those pushes into Phoenix PubSub so a
LiveView wakes up the moment a payment arrives.
Why talk directly, and
why not over HTTP
Two reasons, one political, one technical.
The political one: a block explorer is an observer. Every query for
an address tells it that this address is of interest to someone, and
over time that builds an address-to-user graph. Our own Electrum server,
querying only our own node, makes that observer unnecessary.
The technical one: the Electrum protocol is not request/response over
HTTP. It is a newline-delimited JSON-RPC 2.0 stream over a persistent
connection. You send a request, a reply comes back eventually, but in
between and afterwards the server can push messages unprompted at any
time. An HTTP client does not know this model. What you need here is a
socket, a process that holds it, and some bookkeeping.
Three modules, three
responsibilities
I split the thing into three parts, cleanly separated by state:
Protocol is stateless and side-effect free. It encodes
and decodes JSON-RPC and computes the Electrum script hash. Trivial to
test, callable from any process.Connection is the GenServer. It holds the socket,
multiplexes requests, handles reconnect, and translates server pushes
into PubSub. All the unpleasant state lives here.Client is a thin facade on top:
get_history, list_unspent,
subscribe_scripthash, and so on. Readable functions that
internally just call Connection.rpc/2.
The core
problem: one socket, many open requests
A TCP socket does not deliver a clean stream of messages, it delivers
a stream of bytes. In active mode the bytes arrive as chunks via
handle_info, in arbitrary pieces. A JSON line can be split
across two chunks, or two lines can arrive in one. So you buffer and cut
on \n:
defp process_buffer(buffer, state) do
parts = String.split(buffer, "\n")
{complete, [remainder]} = Enum.split(parts, -1)
state =
Enum.reduce(complete, state, fn
"", acc -> acc
line, acc -> handle_json_line(line, acc)
end)
{remainder, state}
end
Everything before the last newline is complete, the rest goes back
into the buffer and waits for the next chunk.
The genuinely interesting part is the bookkeeping. Several callers
can have a request open at the same time, all over the same socket. Each
request gets a running id, and a pending map remembers
which id belongs to which waiting caller:
def handle_call({:rpc, method, params}, from, state) do
{id, state} = next_id(state)
case send_encoded(state, Protocol.encode_request(id, method, params)) do
{:ok, state} ->
{:noreply, put_pending(state, id, {:call, from, method, t0})}
{:error, reason} ->
{:reply, {:error, reason}, disconnect(state)}
end
end
Instead of replying immediately, the GenServer parks the
from under the id and returns {:noreply, ...}.
When the reply comes back over the socket, it looks the id up and
answers the right caller with GenServer.reply/2. That way
Connection.rpc/2 feels synchronous to the caller, even
though a dozen requests may be in flight at once under the hood.
The whole routing logic hangs off a single distinction in the
protocol. A reply carries an id. A notification pushed by
the server carries a method name and no id.
That is all of it:
def parse_message(json_line) do
case Jason.decode(json_line) do
{:ok, %{"id" => id, "result" => result}} -> {:reply, id, result}
{:ok, %{"id" => id, "error" => error}} -> {:error_reply, id, error}
{:ok, %{"method" => method, "params" => params}} -> {:notification, method, params}
_ -> {:error, :parse_error}
end
end
Subscriptions,
or: how the wallet wakes up on its own
Polling addresses every second would be wasteful and slow. Electrum
offers something better: you subscribe to a script hash, and the moment
anything about it changes, an incoming transaction, a confirmation, the
server sends a notification on its own.
This is where two worlds collide. A GenServer cannot hand an async
push back to someone who asked half a minute ago. Nobody is waiting for
a reply anymore. So Connection translates every push into a
Phoenix.PubSub broadcast on a per-script-hash topic.
Whoever subscribed to that topic, a LiveView, a worker, wakes up via
handle_info:
defp broadcast_notification("blockchain.scripthash.subscribe", [hash, status]) do
Phoenix.PubSub.broadcast(
@pubsub,
"electrum:scripthash:#{hash}",
{:scripthash_status, hash, status}
)
end
This cleanly decouples the socket from its consumers.
Connection knows nothing about LiveViews, and the LiveView
knows nothing about sockets. All that sits between them is a topic
name.
This model has one quirk, and it is a genuine footgun: you must
subscribe to the PubSub topic before you send the Electrum
subscription. Otherwise you miss the initial broadcast that
Connection fires as soon as the server confirms the
subscription. In real use it looks like this, here in the
XpubScanner that derives the addresses from an xpub:
# Subscribe to the topic first, then send the Electrum subscription,
# so the initial broadcast is not lost.
Phoenix.PubSub.subscribe(@pubsub, "electrum:scripthash:#{script_hash}")
Client.subscribe_scripthash(address)
The flow in practice: the scanner derives the first twenty addresses
per chain from the xpub (the gap limit), subscribes each script hash,
and then goes to sleep. When a payment lands on address number seven,
Electrum pushes, Connection broadcasts, the scanner reacts,
and the portfolio updates. No polling.
The Bitcoin quirk:
script hash, not address
Electrum does not index by address, it indexes by the hash of the
output script. The algorithm is a small, easy-to-miss convention: SHA256
over the scriptPubKey, then reverse the bytes, then
hex-encode.
def script_to_hash(script) do
:crypto.hash(:sha256, script)
|> :binary.bin_to_list()
|> Enum.reverse()
|> :binary.list_to_bin()
|> Base.encode16(case: :lower)
end
Reversing the bytes is the part you are guaranteed to forget the
first time, and then nothing lines up. Before that comes converting the
address into the scriptPubKey, and that depends on the
address type. A Taproot address (bc1p…) yields a different
script than a SegWit v0 output (bc1q…) or an old
1… address. For the Bech32 cases I build the script
straight from the witness program:
{:ok, {_network, 0, program}} when length(program) == 20 ->
# P2WPKH: OP_0 <20-byte key hash>
script = <<0x00, 0x14>> <> :binary.list_to_bin(program)
{:ok, script_to_hash(script)}
{:ok, {_network, 1, program}} when length(program) == 32 ->
# P2TR (Taproot): OP_1 <32-byte x-only pubkey>
script = <<0x51, 0x20>> <> :binary.list_to_bin(program)
{:ok, script_to_hash(script)}
Reconnect, without going deaf
A long-lived socket drops eventually. The server restarts, the
network hiccups, a firewall reaps an idle connection. When that happens,
two things must not occur: the in-flight callers must not hang until
timeout, and the subscriptions must not be lost, or the wallet goes
deaf.
So on disconnect, Connection fails all open callers
immediately instead of letting them wait:
Enum.each(state.pending, fn
{_id, {:call, from, _method, _t0}} -> GenServer.reply(from, {:error, :disconnected})
{_id, {:sub, from, _method, _params, _t0}} -> GenServer.reply(from, {:error, :disconnected})
_ -> :ok
end)
And all subscriptions live in a MapSet of
{method, params} that is replayed after every reconnect.
The initial status gets broadcast again in the process, so subscribers
re-synchronize. On top of that, a ping every sixty seconds so the
connection does not quietly die. None of this is an extra. It is the
difference between a demo and something that survives a night.
Lessons learned
Three things that paid off, and one I would sharpen up.
1. Pull out the stateless part.
Protocol has no state and no side effects. Encode,
decode, compute the script hash, nothing more. That makes exactly the
fiddly part, the byte reversal and the address types, testable without a
running server. All the unpleasant state sits concentrated in one
GenServer, which has stayed manageable as a result.
2. PubSub as a translator between push and
request.
The mismatch between “the server pushes whenever it wants” and “a
caller wants a reply” cannot be argued away. Resolving it through a
broadcast, instead of trying to attach the push to a caller who is long
gone, decoupled the socket from its consumers. New consumers come along
without Connection ever noticing.
3. Plan for reconnect from the start.
I was tempted to build the happy-path connection first and bolt
reconnect on “later”. Glad I did not. Re-subscription and failing open
callers immediately are not polish, they are half the substance.
Retrofitting them would have meant threading them through state that had
already grown.
4. What I would sharpen up: the TLS and the single
connection.
Two spots are honestly compromises. First, I do not verify the server
certificate (verify: :verify_none), because it is my own,
often self-signed server. For a third-party Electrum server that would
be wrong; there the certificate belongs pinned or properly checked.
Second, the subscription connection is a single named GenServer, a
deliberate singleton for the push stream. The bulk reconciliation of
addresses runs alongside it over short-lived connections, and I had to
throttle that to about five concurrent refreshes, because beyond that
the server answers with timeouts. A proper connection pool would be the
cleaner answer here, but it is not built yet.
If a particular spot interests you, the script-hash scheme, reconnect
with re-subscription, or how the XpubScanner hooks in, drop it in the
comments. I am happy to go deeper.
Kommentare