Skip to main content

Debug RPC and Debug Pages

Recently, we’ve added a set of debug pages, that allow you to see more details about how your node works.

Enabling debug page

Debug support must be enabled in the config.json (auto-generated by neard with init command)

cd ~/path_to_nearcore/nearcore
cargo run --package neard -- init # generates config.json on ~/.near/config.json
vim ~/.near/config.json # update "enable_debug_rpc"

The file should be generated on the path ~/.near/config.json with these default values.

"rpc": {
...
"enable_debug_rpc": true
}

And restarting your node.

After this, you can go to "/debug" page to see links to all the sub-pages.

Important: If you’re a mainnet validator, we’d suggest you to firewall-protect access to debug pages & APIs. Some debug API calls are quite expensive and if someone spams them, they can overload the system and impact your block production speed.

Information on debug pages

Most of the debug pages should be self-explanatory, but here’s a quick overview of what they offer:

Last blocks:

  • quick overview of the most recent blocks - including their gas usage, time spent, who was responsible for producing the block/chunk etc.

Network info

  • Shows you information about the currently reachable validators, about the state of your peers, and the routing of messages that are sent towards the validator

Epoch info

  • shows you information about current (and past) epochs - when did they start, how many validators etc

  • information about validator stakes, upcoming proposals & kickouts

  • the history of validators in the previous epochs

  • and shard sizes

Chain & chunk info page

  • shows you information about the recent blocks & chunks:

    • when chunks were requested, what parts are we waiting for, how long did download take etc etc.

Sync page

shows the status of the sync (header sync, block sync etc) - and how many blocks are remaining

it also show the status of the catchup - when you’re requesting to track a new shard

Validator page

  • this page is the most complex one - as it shows the consensus layer:

    • when your node sent approvals
    • when your node is about to produce a block - it shows received approvals, chunk headers

Thanks to this page, you can debug why a given block wasn’t produced, or why it didn’t contain a chunk.

In the image above, the block 3253009 didn’t contain the chunk for shard 3, let’s understand why:

  • This node was a block producer for this block (otherwise this row would be empty) - at at some time "F", it received a first approval from one of the other validators.

  • Then more approvals and chunks started coming (for example munris.factory has sent chunk0 at time F + 62ms and boot2.node sent the approval at time F + 528 ms

  • At one moment, this node received the 2/3 of the approvals needed – this happened exactly at F + 617 ms - which we’d call the time "T".

  • At this time, it was still missing chunk from shard 3 - so it decided to wait a little.

  • 600ms later (so at time T + 643 ms) - it got tired of waiting and it produced the block WITHOUT chunk3

  • In this example, we can see that verse2 was able to deliver the chunk eventually – but that was far too late (as it arrived more than 1.3 seconds after time T).

For the block 3253014 - the situation was similar, but producer of chunk 4, was able to deliver it before the 600 ms cutoff (block arrived at T + 430ms) - which allowed this block producer to create the block 1 ms later (as all chunks were present).

Approval history:

At the bottom of the validator page, you can see “approval history” table:

This shows all the approvals that your node has sent.

When everything is running fine, you’ll see that your node is regularly sending approvals for the next blocks (from block X to block X+1).

Once there are some missing blocks etc, the node might decide to start sending ‘skip’ requests (basically saying - please ignore the blocks in the middle and build a new block Y on top of block X )

This table becomes very useful during debugging reasons why blocks weren’t produced - as sometimes the block producer might not get the approvals. And using this page, we can check whether approval was generated and at what time.