Skip to content

Zmq blockhash notifications#150

Open
w3irdrobot wants to merge 17 commits intobennyhodl:masterfrom
w3irdrobot:zmq-blockhash-notifications
Open

Zmq blockhash notifications#150
w3irdrobot wants to merge 17 commits intobennyhodl:masterfrom
w3irdrobot:zmq-blockhash-notifications

Conversation

@w3irdrobot
Copy link
Copy Markdown

This PR adds the ability for a node to listen on a bitcoind blockhash ZeroMQ endpoint and force a wallet sync and contract monitor check.

Current setup

The wallet syncs and contract monitor checks are executed on a set interval; 60 seconds for the wallet syncs and 30 seconds for contract checks. This is fine and functions well. However, if a block (or multiple) are found in that time frame, the node won't pick it up until at least that interval has completed again. This results in an unnecessary lag between block mining and node state updating.

New ZMQ integration

This PR adds the ability for a node to configure a ZMQ blockhash notification endpoint on a local bitcoind node. This allows the software to know immediately when a new block is found and sync right when that notification is seen. In local testing with a regtest node, the notification of a new block was immediate, no lag.

This could be implemented with a feature flag in Cargo. I figured doing it without one is okay for now since that feature flag would infect the code base quite a bit. If it's desired, I can add that in.

Configuration updates

The DDK builder now has a set_zmq_blockhash_endpoint for setting this endpoint. In addition, the ddk-node has a zmq-blockhash-endpoint flag that ends up setting the DDK builder flag under the hood. node-one in the justfile has been updated to have this turned on. Because we are using the zmqpubhashblock and not zmqpubrawblock, I needed to add that flag to the bitcoind node's configuration in the docker-compose.yaml file.

Note: zmqpubhashblock was chosen instead of zmqpubrawblock since this particular story was more concerned with the timing of the syncing and less removing HTTP requests entirely in the syncing process. Receiving and parsing an entire block seemed wasteful when we really only wanted to know the new block was received. There is a future where the zmqpubrawblock is used entirely and that block is propagated through the system. However, due to the EsploraClient's internal clients leaking out in certain areas, that would be a much larger refactor. I started down that road and retreated.

Fallback

The default is the existing behavior. The ZMQ stuff is only used when configured to be on. When the feature is being used, the polling is still there. However, the polling times are much larger in-between polls and are there as a backup in-case something happened with the ZMQ connection. That said, ZMQ is supposed to be self-healing as I understand it.

Testing

Local testing was done with the following process:

  1. Run just deps
  2. Login to the local database and run CREATE DATABASE ddk; CREATE DATABASE ddk_one; CREATE DATABASE ddk_two;. These databases weren't autocreated.
  3. Create the wallet with just bc createwallet ddk
  4. Mine some blocks with just bc generatetoaddress 101 $(just bc getnewaddress)
  5. Spin up node one with just node-one.
  6. In another terminal, mine another block and observe the logs : just bc generatetoaddress 1 $(just bc getnewaddress)

Note: I used log_info when testing since the debug logs didn't seem to be coming out, and I didn't spend much time debugging that. I changed most of those logs back to log_debug once I confirmed things work.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants