Debug RPC and Debug Pages
Recently, we’ve added a set of debug pages, that allow you to see more details about how your node works.
Enabling debug page
Debug support must be enabled in the
config.json (auto-generated by
cargo run --package neard -- init # generates config.json on ~/.near/config.json
vim ~/.near/config.json # update "enable_debug_rpc"
The file should be generated on the path
~/.near/config.json with these default values.
And restarting your node.
After this, you can go to
"/debug" page to see links to all the sub-pages.
Important: If you’re a mainnet validator, we’d suggest you to firewall-protect access to debug pages & APIs. Some debug API calls are quite expensive and if someone spams them, they can overload the system and impact your block production speed.
Information on debug pages
Most of the debug pages should be self-explanatory, but here’s a quick overview of what they offer:
- quick overview of the most recent blocks - including their gas usage, time spent, who was responsible for producing the block/chunk etc.
- Shows you information about the currently reachable validators, about the state of your peers, and the routing of messages that are sent towards the validator
shows you information about current (and past) epochs - when did they start, how many validators etc
information about validator stakes, upcoming proposals & kickouts
the history of validators in the previous epochs
and shard sizes
Chain & chunk info page
shows you information about the recent blocks & chunks:
- when chunks were requested, what parts are we waiting for, how long did download take etc etc.
shows the status of the sync (header sync, block sync etc) - and how many blocks are remaining
it also show the status of the catchup - when you’re requesting to track a new shard
this page is the most complex one - as it shows the consensus layer:
- when your node sent approvals
- when your node is about to produce a block - it shows received approvals, chunk headers
Thanks to this page, you can debug why a given block wasn’t produced, or why it didn’t contain a chunk.
In the image above, the block 3253009 didn’t contain the chunk for shard 3, let’s understand why:
This node was a block producer for this block (otherwise this row would be empty) - at at some time "F", it received a first approval from one of the other validators.
Then more approvals and chunks started coming (for example munris.factory has sent chunk0 at time
F + 62msand
boot2.nodesent the approval at time
F + 528 ms
At one moment, this node received the 2/3 of the approvals needed – this happened exactly at
F + 617 ms- which we’d call the time "T".
At this time, it was still missing chunk from shard 3 - so it decided to wait a little.
600ms later (so at time T + 643 ms) - it got tired of waiting and it produced the block WITHOUT chunk3
In this example, we can see that verse2 was able to deliver the chunk eventually – but that was far too late (as it arrived more than 1.3 seconds after time T).
For the block 3253014 - the situation was similar, but producer of chunk 4, was able to deliver it before the 600 ms cutoff (block arrived at
T + 430ms) - which allowed this block producer to create the block 1 ms later (as all chunks were present).
At the bottom of the validator page, you can see “approval history” table:
This shows all the approvals that your node has sent.
When everything is running fine, you’ll see that your node is regularly sending approvals for the next blocks (from block X to block X+1).
Once there are some missing blocks etc, the node might decide to start sending ‘skip’ requests (basically saying - please ignore the blocks in the middle and build a new block Y on top of block X )
This table becomes very useful during debugging reasons why blocks weren’t produced - as sometimes the block producer might not get the approvals. And using this page, we can check whether approval was generated and at what time.