NodePulse: so your Lightning node speaks up before it gets expensive

 

A Lightning node is an odd kind of device. It sits on a shelf somewhere, a Raspberry Pi with an SSD, and as long as everything works you don’t look at it for weeks. That is exactly the problem. When LND hangs, the SSD fills up or a peer has been offline for days, nobody tells you. You only notice when a payment fails. Or worse: when an HTLC is about to expire and nobody reacts.

RaspiBlitz comes with a lot, but not with alerting that actively nudges me. The ready-made solutions I looked at were either too big (Prometheus, Grafana and Alertmanager on a Pi that is supposed to run a Bitcoin node) or they ran as a service somewhere else, with access to my node data. I didn’t want either.

So I built NodePulse. A small program that runs directly on the node, checks every minute whether everything is fine, and sends me a Telegram message when it isn’t. And another one once the problem is gone. On top of that there is a dashboard in the browser. In this article I explain what NodePulse checks, how it works internally and how to install it on a RaspiBlitz.

Let me say this upfront so nobody is disappointed: NodePulse is still young. It is built and tested on RaspiBlitz 1.12 with LND (Debian 12, arm64). Umbrel, Start9 or Core Lightning don’t work at the moment.

What NodePulse checks

Every minute NodePulse takes a snapshot of the node and evaluates it. The list grew over time, the more I thought about what can go wrong:

Check	Warning	Critical
LND reachable	–	REST API not responding
LND synced to chain	–	synced_to_chain is false
Disk space (per path)	from 80 %	from 90 %
Copy of the channel backup (optional)	–	missing or outdated
Pending HTLCs	expires in ≤ 72 blocks (~12 h)	≤ 24 blocks (~4 h)
Channel closing	closing transaction unconfirmed or funds time-locked	–
Channel closed (last 144 blocks)	force close	breach close
Channel active (per channel)	inactive, with duration	–
Peer offline (per channel)	for ≥ 7 days	–
Raspberry Pi power supply	throttled now or since boot	under-voltage now
Raspberry Pi temperature	from 75 °C	from 80 °C
Bitcoin Core (optional)	< 4 peers, no block for 90 min	unreachable, not synced, 0 peers, no block for 3 h
On-chain fees	≥ 50 sat/vB	–

The two most important rows are the ones about HTLCs and force closes. That’s where real money is at stake. An HTLC that is still open shortly before it expires ends, in the worst case, in a force close, and with high fees that gets really expensive. That’s also why the fee warning is on the list. It doesn’t say “something is broken”, it says “now would be a bad moment for a force close”.

Liquidity per channel is shown on the dashboard only, as a bar. There is deliberately no alert for it. A channel sitting 95 % on one side is not an emergency, and a notification you get every day and ignore is worse than none at all.

The dashboard on my own node. Marked in orange is a force close with 2132 sat still time-locked for 133 blocks. I blurred the channel IDs and the transaction.

How the notifications work

With alerting, one thing mattered more to me than anything else: it must not be annoying. Otherwise you mute it after a week and you’re back to nothing.

So these are the rules:

• A problem has to persist for two checks in a row before a message goes out. A single hiccup of LND after a restart triggers nothing.
• There is at most one Telegram message per check run, containing everything.
• A new message only comes when the severity changes, from warning to critical or back.
• Once the problem is gone, you get a RECOVERED.

This is what it looks like in Telegram:

NodePulse
🔴 CRITICAL: LND is not reachable: connection refused
🟡 WARNING: Channel 123456789012345678 (02a1b2c3d4) is inactive (peer offline?)
✅ RECOVERED: Disk /mnt/disk_storage is 72% full

NodePulse stores which alerts have already been sent in its SQLite database. When the service restarts, for example after an update, you don’t get the same warning again, and the recovery message doesn’t get lost either.

Every morning at eight you also get a short summary, so you know NodePulse is still alive:

NodePulse daily summary
✅ All systems healthy

• LND is synced at block 974400
• Bitcoin Core is synced at block 974400
• 1.0 sat/vB for the next blocks
• CPU at 67 °C
• 2 of 2 channels active
• Disk /mnt/disk_storage is 58% full

In between you can ask the bot for the current state with /status. It only answers in the one chat that is configured and ignores everybody else. NodePulse fetches the commands from Telegram via long polling, so no port has to be opened to the outside.

The problem with a dead node

There is one thing NodePulse can’t do by design: tell you that the node is off. If the Pi has no power, NodePulse isn’t running either.

That’s what the heartbeat is for. After every check NodePulse sends an empty ping to an external service, for example healthchecks.io. If the pings stop, the service raises the alarm. It doesn’t receive any data from the node, just the signal “I’m still alive”. It’s optional, but I’d recommend it to everyone. Only with it does the whole thing really come full circle.

Under the hood

NodePulse is a Phoenix application written in Elixir. You don’t need to know anything about that to run it, but for those who are interested, here is a quick sketch.

At its core is a GenServer, the Monitor. Every minute it fetches data from LND, Bitcoin Core, the disks and the Pi, evaluates it and distributes the result via Phoenix PubSub:

results = Checks.run(snapshot)
Phoenix.PubSub.broadcast(Nodepulse.PubSub, @topic, {:status_updated, results})
Process.send_after(self(), :poll, state.interval)

Checks.run/1 is a pure function. Snapshot in, list of results out. That makes it trivial to test without a real node running anywhere. Two consumers listen on the topic: the Alerter, which decides whether Telegram gets a message, and the dashboard’s LiveView, which updates immediately. Neither knows about the other.

There’s one detail about the connection to LND worth mentioning. LND uses a self-signed TLS certificate. The convenient solution would be to switch off certificate verification. Instead, NodePulse pins the certificate from tls.cert: the connection is only accepted if LND presents exactly that certificate.

Security

A program running on a node with real money on it has to put up with a few uncomfortable questions. Here are the answers:

• Read-only access. NodePulse uses LND’s readonly.macaroon. It can’t move funds, open or close channels or change any configuration.
• Dedicated system user. The service runs as nodepulse without a login shell. The systemd unit mounts the filesystem read-only, except for NodePulse’s own data directory.
• Dedicated RPC user for Bitcoin Core. If you use the Bitcoin Core checks, you create a user that may call exactly three methods: getblockchaininfo, getnetworkinfo and estimatesmartfee. NodePulse doesn’t touch LND’s RPC user.
• No login on the dashboard. The dashboard shows channel IDs and balances. So it belongs in your LAN or behind a VPN like Tailscale. Never forward port 4000 on your router.

Installation on a RaspiBlitz

All you need is SSH access. NodePulse comes as a prebuilt release of about 13 MB, so Erlang or Elixir don’t have to be installed on the node. At runtime it needs around 210 MB of RAM.

1. Get the code

ssh admin@<node-ip>
git clone https://gitlab.com/public_elixir/nodepulse.git
cd nodepulse

The scripts for setup and installation live in deploy/ in the repository.

2. One-time setup

sh deploy/setup.sh

The script creates the user nodepulse, generates the configuration file /etc/nodepulse.env, installs the systemd service and opens port 4000 in the firewall, on the Tailscale interface only. It’s safe to run it several times. An existing configuration is never overwritten.

If you don’t use Tailscale, allow the dashboard from your own LAN instead (adjust the network):

sudo ufw allow from 192.168.0.0/16 to any port 4000 proto tcp comment "NodePulse dashboard (LAN)"

3. Create a Telegram bot

1. In Telegram, open @BotFather, send /newbot and follow the instructions. At the end you get a token like 123456789:AAH.... Treat it like a password.

2. Open the chat with your new bot, press Start and send any message.

3. Get the chat ID:

   curl -s https://api.telegram.org/bot<TOKEN>/getUpdates

   You’re looking for "chat":{"id":123456789,...}. If all you get is "result":[], the bot hasn’t received a message yet, so repeat step 2.

4. Configure

sudo nano /etc/nodepulse.env

The defaults fit a RaspiBlitz on mainnet. All you have to add are the token and the chat ID:

TELEGRAM_BOT_TOKEN=123456789:AAH...
TELEGRAM_CHAT_ID=123456789

Optional but recommended: the ping URL from healthchecks.io (create a check there with Period 1 minute and Grace 5 minutes) and the time zone for the daily summary:

HEARTBEAT_URL=https://hc-ping.com/<uuid>
TZ=Europe/Berlin

5. Install

sh deploy/install.sh

The script downloads the latest release from GitLab, verifies its SHA-256 checksum, installs it to /home/nodepulse/app, restarts the service and waits until the dashboard responds. At the end it prints NodePulse is up. On the Pi, startup takes about ten seconds.

6. Open the dashboard

Open http://<node-ip>:4000 in your browser. Really http, not https. Browsers like to add the s on their own, and then the page won’t load.

Optional extras

Two extensions are worth it. Both are described in detail in the README, so here is just the gist.

Channel backup on a USB stick. LND’s static channel backup is what gets you back to your channel funds after a total failure. RaspiBlitz can copy it to a USB stick automatically (note: the stick must be between 1 and 31 GB and will be formatted with FAT32). Once you add SCB_BACKUP_PATH=/mnt/backup/channel.backup, NodePulse checks whether the copy on the stick exists and matches the original. If the stick is pulled out or the copy is outdated for more than ten minutes, you get a critical alert.

Bitcoin Core. This needs the restricted RPC user mentioned above in bitcoin.conf. One line is crucial:

rpcwhitelistdefault=0

Without it, all other RPC users lose their permissions as soon as any rpcwhitelist exists. LND included. Also, on a RaspiBlitz restarting Bitcoin Core restarts LND as well, and without auto-unlock you have to unlock the wallet again afterwards. So do this when you’re sitting at the node.

Updates

cd nodepulse
git pull
sh deploy/install.sh

Your configuration is kept, and database migrations run automatically on start. By the way, when a new version is available, NodePulse tells you itself, on the dashboard and in the daily summary.

What I learned along the way

1. Fewer alerts are better.

It’s tempting to report everything you can measure. For every check I asked myself whether I’d get out of bed at night for it. For liquidity the answer is no, so all you get is the bar on the dashboard. Together with the “two checks in a row” rule, what’s left is what I take seriously when my phone buzzes.

2. What happens on a restart decides whether you trust it.

At first the alert state only lived in memory. After every update the same warnings came again, or a recovery message fell through the cracks. Since the state lives in SQLite, that’s over. Sounds like a detail, but it’s exactly these details that decide whether you believe your monitoring.

3. Without an external heartbeat, there’s a gap.

Monitoring that runs on the device it watches is blind to the most important case. That’s why the heartbeat was part of the plan from the start: everything important runs on the node, and exactly one external service makes sure the node itself is still there. No more infrastructure needed.

4. What’s still missing.

Honestly, quite a bit. So far NodePulse only knows RaspiBlitz and LND. Umbrel, Start9 and Core Lightning are on the list but not done yet. And the dashboard has no login, which is fine as long as it stays inside your own network, but it is a limitation.

The code is MIT licensed and on GitLab. If you try it on your node, run into a bug or miss a check, let me know in the comments or open an issue.