At Fiatbitcoin a user can paste in an xpub, the extended public key
from their hardware wallet, and then sees every address, every incoming
payment, and their full balance, in real time. Without the app ever
laying eyes on a private key. A watch-only wallet, in other words.
Three problems sit behind that. First, you have to derive the
addresses from the xpub, and that is pure cryptography. Second, those
addresses have to stay in sync, the moment something moves on-chain, not
at the next poll. Third, as a bonus, the system should recognize when
the user merely shuffled coins between their own addresses, so the tax
module does not mistake it for a sale. This article goes through the
three in order.
What an xpub gives you
An extended public key is a public key plus a chain code. That
combination allows something that looks paradoxical at first: from it
you can derive arbitrarily many child addresses, without the private key
at all. This is the basis of BIP32 and the reason watch-only wallets
work in the first place.
The convention on top is BIP44. The receiving addresses live under
the path m/0/*, the change addresses under
m/1/*. Both chains are numbered separately. Whoever holds
the xpub can compute every one of these addresses and watch their
balances, but spend nothing.
HD derivation
by hand: secp256k1 without a library
This is where it gets interesting. Deriving a child address is not
just a hash. It is real elliptic-curve arithmetic. The new public key
results from the old one plus a point on the secp256k1 curve:
defp derive_child({_x, _y} = pubkey, chain_code, index) when index < 0x80000000 do
data = compress_pubkey(pubkey) <> <<index::big-32>>
<<il::binary-32, ir::binary-32>> = :crypto.mac(:hmac, :sha512, chain_code, data)
il_int = decode_uint(il)
# child_pubkey = il_int * G + parent_pubkey
child_point = point_add(scalar_mul(il_int), pubkey)
{child_point, ir}
end
An HMAC-SHA512 over the compressed parent key and the index yields 64
bytes. The left 32 are interpreted as a number and multiplied with the
generator point G, the right 32 become the new chain code.
The il_int * G is a scalar multiplication on the curve, and
point_add adds two curve points.
The remarkable part: this curve arithmetic is implemented entirely by
hand, without a crypto library. point_add,
point_double, the modular inverse via Fermat’s little
theorem, it is all in the module. On top of that come Base58Check for
the legacy addresses and Bech32 for the SegWit addresses, also written
from scratch. The scalar multiplication is the classic
double-and-add:
defp point_mul(0, _p, acc), do: acc
defp point_mul(k, p, acc) do
acc = if rem(k, 2) == 1, do: point_add(acc, p), else: acc
point_mul(div(k, 2), point_double(p), acc)
end
It feels good to have the whole derivation standing there without a
dependency. It is also exactly the kind of code you do not want a subtle
bug in. More on that in the lessons.
The gap limit: when do I
stop deriving
If you can derive arbitrarily many addresses, the question is where
you stop. A wallet might have used its fifth address or its
five-hundredth. BIP44 answers this with the gap limit: as long as
nothing has happened over a stretch of consecutive addresses, you stop
after a fixed number of empty ones. Twenty is common.
The scanner does this eagerly rather than lazily. The moment an
address at index i shows activity, it immediately derives
up to i + gap_limit + 1, so that a sufficient buffer of
unused addresses always runs ahead:
defp maybe_extend_scan(state, change, active_index) do
required_head = active_index + state.gap_limit + 1
current_head = Map.fetch!(state.scan_head, change)
if required_head > current_head do
new_count = required_head - current_head
Logger.debug(
"[XpubScanner] Extending chain=#{change} from=#{current_head} to=#{required_head - 1}" <>
" (+#{new_count} addresses)"
)
derive_and_subscribe(state, change, current_head, required_head)
else
state
end
end
That way the window moves along with usage, without having to derive
a thousand addresses up front.
One process per wallet
that never polls
Every watched wallet gets its own GenServer, the
XpubScanner, started under a DynamicSupervisor
and findable via a Registry under the key
{user_id, xpub}. That isolates the wallets from each other:
if one scan stumbles, it does not drag the others down, and a new wallet
costs only a process.
The scanner’s state holds the derived addresses, a reverse index from
script hash to address, and the head of each chain:
state = %{
xpub: xpub,
user_id: user_id,
gap_limit: gap_limit,
# %{address => %{change: 0|1, index: non_neg_integer(), script_hash: String.t()}}
addresses: %{},
# %{script_hash => address} — reverse lookup for incoming notifications
by_script_hash: %{},
# Next index to derive for each chain (0=external, 1=change)
scan_head: %{0 => 0, 1 => 0},
tip_height: nil,
refresh_queue: [],
refreshing: MapSet.new()
}
Instead of querying the addresses every second, the scanner
subscribes each script hash to the Electrum server and then goes to
sleep. When a payment arrives, Electrum pushes a notification that
reaches the scanner over Phoenix PubSub, and it reacts. One detail there
is delicate: you have to subscribe to the PubSub topic before you send
the Electrum subscription, or you miss the initial broadcast.
# Subscribe to PubSub *before* sending Electrum subscription so we do not miss
# the initial broadcast that Connection emits when Electrum confirms.
Phoenix.PubSub.subscribe(@pubsub, "electrum:scripthash:#{script_hash}")
# Electrum subscription is sent asynchronously to avoid blocking the scanner
# while waiting for network round-trips during the initial bulk subscription.
Task.start(fn -> Client.subscribe_scripthash(address) end)
How that socket connection to Electrum works underneath, the
multiplexing and the translation of server pushes into PubSub, I
described in a separate article. Here the scanner is just a consumer of
those topics.
The push path is elegant, but not guaranteed. During a reconnect,
notifications can be lost. So as a safety net there is a full
reconciliation of the active addresses every twenty minutes. That
refresh runs through a bounded-concurrency queue, because each
individual reconciliation opens several fresh connections to
Electrum:
defp drain_refresh_queue(state) do
available = @max_concurrent_refreshes - MapSet.size(state.refreshing)
{to_run, remaining} = Enum.split(state.refresh_queue, available)
scanner_pid = self()
user_id = state.user_id
xpub_id = state.xpub_id
Enum.each(to_run, fn address ->
Task.start(fn ->
refresh_address(address, user_id, xpub_id)
send(scanner_pid, {:refresh_done, address})
end)
end)
%{state | refresh_queue: remaining,
refreshing: Enum.reduce(to_run, state.refreshing, &MapSet.put(&2, &1))}
end
More than five simultaneous reconciliations is more than the server
tolerates, beyond that you get timeouts. So it throttles, and the rest
waits in the queue.
Detecting a
consolidation: every output is mine
Whoever merges many small UTXOs into one big one sends a transaction
whose outputs all land on their own addresses. That is not a sale and
not a payment, just internal housekeeping. For the tax module the
distinction matters, because a consolidation must not reset the cost
basis.
The heuristic is deliberately simple and sharply drawn. It groups all
outgoing transfers by their transaction and keeps only the transactions
where every destination address belongs to the user themselves:
def run(user_id) do
user_addresses = load_user_addresses(user_id)
address_set = MapSet.new(user_addresses, & &1.address)
forwards = load_forwards(user_id)
# Group forwards by tx_hash to identify consolidation transactions:
# a tx where ALL outputs land on own addresses
consolidation_txs =
forwards
|> Enum.group_by(& &1.tx_hash)
|> Enum.filter(fn {_tx_hash, fwds} ->
Enum.all?(fwds, &MapSet.member?(address_set, &1.to_address))
end)
count =
Enum.reduce(consolidation_txs, 0, fn {tx_hash, fwds}, acc ->
acc + process_consolidation(tx_hash, fwds, user_id, user_addresses)
end)
{:ok, count}
end
What remains is mapping the merged target UTXO back to the source
UTXOs. If there is only one candidate, the matter is clear. With
several, the one whose value comes closest to the sum of the incoming
amounts wins, with a tolerance for the miner fee:
defp find_matching_target(_source_utxo_id, [single], _fwds), do: single
defp find_matching_target(_source_utxo_id, targets, fwds) do
# When multiple target UTXOs exist, use value proximity to the forwarded amounts
total_forwarded = fwds |> Enum.map(& &1.value_satoshi) |> Enum.sum()
Enum.find(targets, fn t ->
abs(t.value_satoshi - total_forwarded) <= @fee_tolerance_sat
end)
end
The tolerance is 10,000 satoshi, roughly 0.0001 BTC.
Lessons learned
Three things that paid off, and one where I would start
differently.
1. Watch-only is pure public-key arithmetic.
It is a surprisingly reassuring thought that the whole wallet works
without a single secret. No private key, no signatures, no material you
could lose. From a public key and a chain code, derivation produces a
complete picture of the wallet, and watching is all the app does.
2. Push beats poll, but only with a safety net.
Reacting to server pushes instead of polling is the difference
between a wallet that wakes up instantly and one that checks back every
minute. But the push path is lossy the moment the connection wobbles.
The periodic reconciliation is therefore not a convenience but the
condition under which nothing slips through for good.
3. The consolidation heuristic is deliberately
narrow.
“Every output is mine” only catches the clean, complete
consolidation. A transaction that also has an external output on the
side falls through the cracks. That is a conscious choice for few false
positives over a high hit rate. For tax it is the safe direction: better
to miss a consolidation than to wrongly write off a payment as internal
shuffling.
4. What I would do differently: secp256k1 by
hand.
The hand-written curve arithmetic is dependency-free and was fun, but
it is the riskiest spot in the whole codebase. A subtle bug in
point_add or in the Bech32 encoding silently derives wrong
addresses, and you only notice once money fails to show up. On top of
that there is the address type: if version detection falls back to the
default :p2pkh, it derives legacy addresses, even though
some hardware wallets like Ledger export native SegWit accounts with
xpub version bytes. There is a function to correct that,
but it does not kick in automatically. For the core of the curve math a
vetted library would be the more sensible choice, especially since we
already use one for the Bech32 decoding elsewhere.
If a particular spot interests you, the scalar multiplication on the
curve, the gap-limit window, or how purchases are reconstructed from the
UTXOs in the end, drop it in the comments. I am happy to go deeper.
Kommentare