Skip to main content

Debug RPC and Debug Pages

Recently, we’ve added a set of debug pages, that allow you to see more details about how your node works.

Enabling debug page​

Debug support must be enabled in the config.json (auto-generated by neard with init command)

cd ~/path_to_nearcore/nearcore
cargo run --package neard -- init # generates config.json on ~/.near/config.json
vim ~/.near/config.json # update "enable_debug_rpc"

The file should be generated on the path ~/.near/config.json with these default values.

"rpc": {
   ...
  "enable_debug_rpc": true
}

And restarting your node.

After this, you can go to "/debug" page to see links to all the sub-pages.

Important: If you’re a mainnet validator, we’d suggest you to firewall-protect access to debug pages & APIs. Some debug API calls are quite expensive and if someone spams them, they can overload the system and impact your block production speed.

Information on debug pages​

Most of the debug pages should be self-explanatory, but here’s a quick overview of what they offer:

Last blocks:​

  • quick overview of the most recent blocks - including their gas usage, time spent, who was responsible for producing the block/chunk etc.

Network info​

  • Shows you information about the currently reachable validators, about the state of your peers, and the routing of messages that are sent towards the validator

Epoch info​

  • shows you information about current (and past) epochs - when did they start, how many validators etc

  • information about validator stakes, upcoming proposals & kickouts

  • the history of validators in the previous epochs

  • and shard sizes

Chain & chunk info page​

  • shows you information about the recent blocks & chunks:

    • when chunks were requested, what parts are we waiting for, how long did download take etc etc.

Sync page​

shows the status of the sync (header sync, block sync etc) - and how many blocks are remaining

it also show the status of the catchup - when you’re requesting to track a new shard

Validator page​

  • this page is the most complex one - as it shows the consensus layer:

    • when your node sent approvals
    • when your node is about to produce a block - it shows received approvals, chunk headers

Thanks to this page, you can debug why a given block wasn’t produced, or why it didn’t contain a chunk.

In the image above, the block 3253009 didn’t contain the chunk for shard 3, let’s understand why:

  • This node was a block producer for this block (otherwise this row would be empty) - at at some time "F", it received a first approval from one of the other validators.

  • Then more approvals and chunks started coming (for example munris.factory has sent chunk0 at time F + 62ms and boot2.node sent the approval at time F + 528 ms

  • At one moment, this node received the 2/3 of the approvals needed – this happened exactly at F + 617 ms - which we’d call the time "T".

  • At this time, it was still missing chunk from shard 3 - so it decided to wait a little.

  • 600ms later (so at time T + 643 ms) - it got tired of waiting and it produced the block WITHOUT chunk3

  • In this example, we can see that verse2 was able to deliver the chunk eventually – but that was far too late (as it arrived more than 1.3 seconds after time T).

For the block 3253014 - the situation was similar, but producer of chunk 4, was able to deliver it before the 600 ms cutoff (block arrived at T + 430ms) - which allowed this block producer to create the block 1 ms later (as all chunks were present).

Approval history:​

At the bottom of the validator page, you can see “approval history” table:

This shows all the approvals that your node has sent.

When everything is running fine, you’ll see that your node is regularly sending approvals for the next blocks (from block X to block X+1).

Once there are some missing blocks etc, the node might decide to start sending ‘skip’ requests (basically saying - please ignore the blocks in the middle and build a new block Y on top of block X )

This table becomes very useful during debugging reasons why blocks weren’t produced - as sometimes the block producer might not get the approvals. And using this page, we can check whether approval was generated and at what time.