NEAR Validator Bootcamp π
Validator Onboarding FAQ'sβ
Whatβs the current protocol upgrade that will increase the number of validators on Mainnet?
The next upgrade to increase the number of mainnet validators will introduce Chunk-Only Producers, and is currently slated for Q3 2022.
How do I join NEAR as a validator on the Mainnet? What steps do I need to take?
- Find out more about how to become a validator, head to https://near-nodes.io/validator
- Join the Open Shards Alliance Server to find out more on how to run nodes and participate on the guildnet.
What are the future plans for NEAR Protocol?
Learn about the protocol roadmap here https://near.org/blog/near-launches-simple-nightshade-the-first-step-towards-a-sharded-blockchain/.
LESSON 1 - OVERVIEW OF NEARβ
Blockchain Technologyβ
Blockchain is a new technology originally developed by Bitcoin. The term most often used to describe blockchain technology is a βpublic ledgerβ. An easy way to describe it, is that itβs similar to your banking statement that keeps track of balances, debits, and credits, with several key differences:
- Limitless β Transactions can be added as long as the network is up and running
- Unchangeable β Immutable once written (validated), it can not be modified
- Decentralized β Maintained by individual nodes called validators worldwide
- Verifiable β Transactions are verified via consensus by decentralized validators via a Proof of Stake (POS) or Proof of Work (POW) algorithm
Smart Contractsβ
Not all blockchains are equal. While some only allow the processing of transactions (balances, debits, and credits), blockchains like NEAR and Ethereum offer another layer of functionality that enables programs to run on the blockchain known as smart contracts.
An easy way to think about it is that βsmart contractβ enabled blockchains are like supercomputers. They allow for users and developers to pay a fee to use CPU/Memory (resources), and Disk (storage) on a global scale.
However, it does not make sense to store everything on the blockchain as it would be too expensive. Web front-ends (HTML, CSS, and JS) are served from web hosting, Github, or Skynet, and images are often stored on InterPlanetary File Systems (IPFS) like Filecoin or Arweave.
Shardingβ
As the founding blockchains, Bitcoin and Ethereum, began to grow, limitations presented themselves. They could only manage to process a specific number of transactions in a block/space of time. This created severe bottlenecks with processing, wait times, and also significantly increased the transaction fee required to have transactions processed.
Sharding is a technological advancement that allows blockchains to scale and to provide more throughput, while decreasing wait times and keeping transaction fees low. It accomplishes this by creating additional shards as the use of the network grows.
About NEARβ
NEAR Protocol was designed with a best-of-breed approach in mind. It focuses on fixing the bottlenecks of the earlier blockchains while also enhancing the user experience to enable broader adoption of blockchain technology by existing Web2 users and developers. NEAR is a sharded Proof of Stake (POS) blockchain.
Key Featuresβ
- Award-Winning Team β Top Programmers in the world
- Proof of Stake (POS)
- Unlimited Shards
- User Experience (UX)
- Developer Experience (DX)
- Rainbow Bridge to ETH β Bridge assets to and from ETH
- Aurora β Run native ETH apps built-in solidity
Technology Stackβ
- Rust β Primary Smart Contract Language
- Assembly Script β Alternant Smart Contract Language (similar to TypeScript)
- NodeJS β Tooling
- Javascript / React / Angular β Frontends
LESSON 2 - NEAR-CLIβ
NEAR-CLI is a command-line interface that communicates with the NEAR blockchain via remote procedure calls (RPC):
- Setup and Installation NEAR CLI
- View Validator Stats
Note: For security reasons, it is recommended that NEAR-CLI be installed on a different computer than your validator node and that no full access keys be kept on your validator node.
Setup NEAR-CLIβ
First, let's make sure the Debian machine is up-to-date.
sudo apt update && sudo apt upgrade -y
Install developer tools, Node.js, and npmβ
First, we will start with installing Node.js and npm:
curl -sL https://deb.nodesource.com/setup_17.x | sudo -E bash -
sudo apt install build-essential nodejs
PATH="$PATH"
Check Node.js and npm version:
node -v
v17.x.x
npm -v
8.x.x
Install NEAR-CLIβ
Here's the Github Repository for NEAR CLI.: https://github.com/near/near-cli. To install NEAR-CLI, unless you are logged in as root, which is not recommended you will need to use sudo to install NEAR-CLI so that the near binary is saved to /usr/local/bin
sudo npm install -g near-cli
Validator Statsβ
Now that NEAR-CLI is installed, let's test out the CLI and use the following commands to interact with the blockchain as well as to view validator stats. There are three reports used to monitor validator status:
Environmentβ
The environment will need to be set each time a new shell is launched to select the correct network.
Networks:
- GuildNet
- TestNet
- MainNet
Command:
export NEAR_ENV=<network> (use guildnet / testnet / mainnet)
Proposalsβ
A proposal by a validator indicates they would like to enter the validator set, in order for a proposal to be accepted it must meet the minimum seat price.
Command:
near validators proposals
Validators Currentβ
This shows a list of active validators in the current epoch, the number of blocks produced, number of blocks expected, and online rate. Used to monitor if a validator is having issues.
Command:
near validators current
Validators Nextβ
This shows validators whose proposal was accepted one epoch ago, and that will enter the validator set in the next epoch.
Command:
near validators next
LESSON 3 - SETUP UP A VALIDATORβ
In this lesson you will learn about:
- Setting up a Node
- Difference between MainNet and TestNet
- Compiling Nearcore for MainNet
- Genesis file (genesis.json)
- Config file (config.json)
Server Requirementsβ
For Block Producing Validators, please refer to the Validator Hardware
For Chunk-Only Producers (an upcoming role on NEAR), please see the hardware requirement below:
| Hardware | Chunk-Only Producer Specifications |
|---|---|
| CPU | 4-Core CPU with AVX support |
| RAM | 8GB DDR4 |
| Storage | 500GB SSD |
Install required software & set the configurationβ
Prerequisites:β
Before you start, you may want to confirm that your machine has the right CPU features. For more hardware specific information, please take a look of the Hardware requirement.
for f in ssse3 sse4_1 sse4_2 popcnt avx fma bmi1 bmi2 abm movbe pclmulqdq; do
grep -qw "$f" /proc/cpuinfo || echo "Missing: $f"
done
If the command prints nothing, the CPU has every instruction that the neard release build needs. Linux lists the LZCNT instruction as the abm flag.
Next, let's make sure the Debian machine is up-to-date.
sudo apt update && sudo apt upgrade -y
Install developer tools:β
sudo apt install -y git binutils-dev libcurl4-openssl-dev zlib1g-dev libdw-dev libiberty-dev cmake gcc g++ python docker.io protobuf-compiler libssl-dev pkg-config clang llvm cargo jq
Install Python pip:β
sudo apt install python3-pip
Set the configuration:β
USER_BASE_BIN=$(python3 -m site --user-base)/bin
export PATH="$USER_BASE_BIN:$PATH"
Install Building envβ
sudo apt install clang build-essential make
Install Rust & Cargoβ
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
You will see the following:

Press 1 and press enter.
Source the environmentβ
source $HOME/.cargo/env
Clone nearcore project from GitHubβ
First, clone the nearcore repository.
git clone https://github.com/near/nearcore
cd nearcore
git fetch origin --tags
Checkout to the branch you need. Latest unstable release is recommended if you are running on testnet and latest stable version is recommended if you are running on mainnet. Please check the releases page on GitHub.
git checkout tags/<version> -b mynode
Compile nearcore binaryβ
In the nearcore folder run the following commands:
make neard
The binary path is target/release/neard. If you are seeing issues, it is possible that cargo command is not found. Make sure sudo apt install cargo. Compiling nearcore binary may take a little while.
Initialize working directoryβ
In order to work properly, the NEAR node requires a working directory and a couple of configuration files. Generate the initial required working directory by running:
./target/release/neard --home ~/.near init --chain-id <network> --download-genesis

This command will create the directory structure and will generate config.json, node_key.json, and genesis.json on the network you have passed. The genesis file for testnet is big (6GB +) so this command will be running for a while and no progress will be shown.
config.json- Configuration parameters which are responsive for how the node will work. The config.json contains needed information for a node to run on the network, how to communicate with peers, and how to reach consensus. Although some options are configurable. In general validators have opted to use the default config.json provided.genesis.json- A file with all the data the network started with at genesis. This contains initial accounts, contracts, access keys, and other records which represents the initial state of the blockchain. The genesis.json file is a snapshot of the network state at a point in time. In contacts accounts, balances, active validators, and other information about the network. On MainNet, it is the state of what the network looked like at launch. On testnet or guildnet, it can include a more intermediate state if the network was hard-forked at one point.node_key.json- The node's network key. Peers use it to identify the node. It is not the validator key.validator_key.json- The key and account that the node validates with. Itsaccount_idmust be your staking pool account.neard initcreates it only when you pass--account-id.data/- A folder in which a NEAR node will write it's state.
Replace the config.jsonβ
rm ~/.near/config.json
wget -O ~/.near/config.json https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/<network>/validator/config.json
Add boot nodesβ
The reference validator config.json has no boot nodes. Follow Update an existing configuration, choosing CHAIN_ID=testnet or CHAIN_ID=mainnet to match <network>. Use CONFIG_PATH=~/.near/config.json for this setup.
Sync with the networkβ
The node is ready to be started. When it starts with an empty data folder, it syncs with the network using Epoch Sync. It does not need a snapshot.
Run the nodeβ
To start your node simply run the following command:
cd nearcore
./target/release/neard --home ~/.near run
The node is now running you can see log outputs in your console. Your node should be find peers, download headers to 100%, and then download blocks.
Create a walletβ
- MainNet: https://wallet.near.org/
- TestNet: https://wallet.testnet.near.org/
- GuildNet:
https://wallet.openshards.io/
Authorize Wallet Locallyβ
A full access key needs to be installed locally to be able transactions via NEAR-CLI.
For Guildnet
export NEAR_ENV=testnet
- You need to run this command:
near login
Note: This command launches a web browser allowing for the authorization of a full access key to be copied locally.
1 β Copy the link in your browser

2 β Grant Access to Near CLI

3 β After Grant, you will see a page like this, go back to console

4 β Enter your wallet and press Enter

For Guildnet
- Download the latest genesis and config files:
cd ~/.near/guildnet
wget -c https://s3.us-east-2.amazonaws.com/build.openshards.io/nearcore-deploy/guildnet/config.json
wget -c https://s3.us-east-2.amazonaws.com/build.openshards.io/nearcore-deploy/guildnet/genesis.json
- Launch this command so set the Near guildnet Environment:
export NEAR_ENV=guildnet
You can also run this command to set the Near guildnet Environment persistent:
echo 'export NEAR_ENV=guildnet' >> ~/.bashrc
For Testnet
- Download the latest genesis, config files:
cd ~/.near/testnet
wget -c https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/testnet/validator/config.json
wget -c https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/testnet/genesis.json
Add boot nodes using Update an existing configuration with
CHAIN_ID=testnetandCONFIG_PATH=~/.near/testnet/config.json. The node syncs with Epoch Sync, so it does not need a snapshot.Launch this command so set the Near testnet Environment:
export NEAR_ENV=testnet
- You can also run this command to set the Near testnet Environment persistent:
echo 'export NEAR_ENV=testnet' >> ~/.bashrc
Note: This is the Chicken and the Egg. You have not created the staking pool, but need to provide the name.
Check the validator_key.jsonβ
- Run the following command:
cat ~/.near/testnet/validator_key.json
Note: If the file exists, keep it. Its
account_idmust be your full staking pool account, such asxx.pool.f863973.m0. Use itspublic_keyas thestake_public_keywhen you create the staking pool.
Note: If a validator_key.json is not present, follow these steps to create one
Create a validator_key.json for Testnet
- Generate the Key file:
near generate-key <pool_id>
- Copy the generated file to
~/.near/testnet/validator_key.json. WithNEAR_ENV=testnet,near generate-keysaves the key in~/.near-credentials/testnet/<pool_id>.json.
cp ~/.near-credentials/testnet/<pool_id>.json ~/.near/testnet/validator_key.json
- Edit βaccount_idβ => xx.pool.f863973.m0, where xx is your PoolName
- Change
private_keytosecret_key
Note: The account_id must match the staking pool contract name or you will not be able to sign blocks.\
File content must be in the following pattern:
{
"account_id": "xx.pool.f863973.m0",
"public_key": "ed25519:HeaBJ3xLgvZacQWmEctTeUqyfSU4SDEnEwckWxd92W2G",
"secret_key": "ed25519:****"
}
- Use the
public_keyof this file as thestake_public_keywhen you create the staking pool.
Step 8 β Check all files were generatedβ
Command for Guildnet
ls ~/.near/guildnet
Command for Testnet
ls ~/.near/testnet
You should have: validator_key.json node_key.json config.json data genesis.json
Setup using NEARCore (MainNet)
RECOMMENDED FOR MAINNET
Step 1 β Installation required software & set the configurationβ
- Before you start, you might want to ensure your system is up to date.
sudo apt update && sudo apt upgrade -y
- Install Python
sudo apt install python3 git curl jq
- Install Building env
sudo apt install clang build-essential make
- Install Rust & Cargo
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Press 1 and press enter

- Source the environment
source $HOME/.cargo/env
- Clone the NEARCore Repo
git clone https://github.com/nearprotocol/nearcore.git
- Set environment to the latest release tag. For the latest release tag, please check here: https://github.com/near/nearcore/releases. Note: RC tags are for Testnet only. The following command reads the latest stable release:
export NEAR_RELEASE_VERSION=$(curl -s https://api.github.com/repos/near/nearcore/releases/latest | grep '"tag_name"' | cut -d '"' -f 4)
cd nearcore
git checkout "tags/${NEAR_RELEASE_VERSION:?}" && make release
- Install Nodejs and NPM
curl -sL https://deb.nodesource.com/setup_17.x | sudo -E bash -
sudo apt install build-essential nodejs
PATH="$PATH"
- Install Near CLI
Once NodeJs and NPM are installed you can now install NEAR-Cli.
Unless you are logged in as root, which is not recommended you will need to use sudo to install NEAR-Cli so that the near binary to /usr/local/bin
sudo npm install -g near-cli
- Launch this command so set the Near Mainnet Environment:
export NEAR_ENV=mainnet
- You can also run this command to set the Near testnet Environment persistent:
echo 'export NEAR_ENV=mainnet' >> ~/.bashrc
Step 2 β Create a walletβ
MainNet: https://wallet.near.org/
Step 3 β Authorize Wallet Locallyβ
A full access key needs to be installed locally to be able transactions via NEAR-Cli.
- You need to run this command:
near login
Note: This command launches a web browser allowing for the authorization of a full access key to be copied locally.
1 β Copy the link in your browser

2 β Grant Access to Near CLI

3 β After Grant, you will see a page like this, go back to console

4 β Enter your wallet and press Enter

Step 4 β Initialize & Start the Nodeβ
- Download the latest genesis and config files:
mkdir ~/.near
cd ~/.near
wget -c https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/mainnet/genesis.json
wget -c https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/mainnet/validator/config.json
Add boot nodes using Update an existing configuration with
CHAIN_ID=mainnetandCONFIG_PATH=~/.near/config.json.The node syncs with Epoch Sync when it starts with an empty
datafolder, so it does not need a snapshot.Initialize NEAR:
target/release/neard init --chain-id="mainnet" --account-id=<pool_id>.poolv1.near
Check validator_key.jsonβ
neard init --account-id=<pool_id>.poolv1.near in the previous step created ~/.near/validator_key.json with a new key. This is the file that neard reads. Keep this key, and show its public key:
grep public_key ~/.near/validator_key.json
Note: The
account_idin this file must be your full staking pool account, such asxx.poolv1.near.neard initwrites the value of--account-idthere. If you passed only the pool name, editaccount_idbefore you start the node.
Note: Use this
public_keyas thestake_public_keywhen you create the staking pool. If the two keys differ, the node stops with a "validator key mismatch" error once the pool is in the validator set.
Optional: use a different keyβ
Follow these steps only if you want to use a different key than the one that neard init created. They replace ~/.near/validator_key.json.
- Generate the Key file:
near generate-key <pool_id>
- Copy the generated file to
~/.near/validator_key.json. WithNEAR_ENV=mainnet,near generate-keysaves the key in~/.near-credentials/mainnet/<pool_id>.json.
cp ~/.near-credentials/mainnet/<pool_id>.json ~/.near/validator_key.json
- Edit βaccount_idβ => xx.poolv1.near, where xx is your PoolName
- Change βprivate_keyβ to βsecret_keyβ
Note: The account_id must match the staking pool contract name or you will not be able to sign blocks. File content must be something like : { "account_id": "xx.poolv1.near", "public_key": "ed25519:HeaBJ3xLgvZacQWmEctTeUqyfSU4SDEnEwckWxd92W2G", "secret_key": "ed25519:****" }
- Use the
public_keyof this file as thestake_public_keywhen you create the staking pool.
Start the nodeβ
- Start the Node
target/release/neard run
- Setup Systemd Command:
sudo vi /etc/systemd/system/neard.service
Paste:
Description=NEARd Daemon Service
[Service]
Type=simple
User=<USER>
#Group=near
WorkingDirectory=/home/<USER>/.near
ExecStart=/home/<USER>/nearcore/target/release/neard run
StandardOutput=file:/home/<USER>/.near/neard.log
StandardError=file:/home/<USER>/.near/nearderror.log
Restart=on-failure
RestartSec=30
KillSignal=SIGINT
TimeoutStopSec=45
KillMode=mixed
[Install]
WantedBy=multi-user.target
Note: Change USER to your paths
Command:
sudo systemctl enable neard
Command:
sudo systemctl start neard
If you need to make a change to service because of an error in the file. It has to be reloaded:
sudo systemctl reload neard
Watch logsβ
Command:
journalctl -n 100 -f -u neard
Make log output in pretty print
Command:
sudo apt install ccze
View Logs with color
Command:
journalctl -n 100 -f -u neard | ccze -A
Becoming a Validatorβ
In order to become a validator and enter the validator set, a minimum set of success criteria must be met.
- The node must be fully synced
- The
validator_key.jsonmust be in place - The contract must be initialized with the public_key in
validator_key.json - The account_id must be set to the staking pool contract id
- There must be enough delegations to meet the minimum seat price. See the seat price here.
- A proposal must be submitted by pinging the contract
- A proposal sent during an epoch is used to select the validators of the epoch after the next one. If it is accepted, the validator joins at the start of that epoch. So it waits between one and two epochs.
- Once in the validator set, the validator must produce at least 80% of its expected blocks and at least 80% of its expected chunks. A validator that only validates chunks must send at least 70% of its expected chunk endorsements. A validator below these thresholds is kicked out of the validator set.
Check running status of validator node. If βValidatorβ is showing up, your pool is selected in the current validators list.
Submitting Pool Information (Mainnet Only)β
Adding pool information helps delegators and also helps with outreach for upgrades and other important announcements: https://github.com/zavodil/near-pool-details. The available fields to add are: https://github.com/zavodil/near-pool-details/blob/master/FIELDS.md.
The identifying information that we ask the validators:to provide are:
- Name
- Description
- URL
- Country and country code
- Email (for support)
- Telegram, Discord, or Twitter
Command:
near call name.near update_field '{"pool_id": "<pool_id>.poolv1.near", "name": "url", "value": "https://yoururl.com"}' --accountId=<accountId>.near --gas=200000000000000
near call name.near update_field '{"pool_id": "<pool_id>.poolv1.near", "name": "twitter", "value": "<twitter>"}' --accountId=<account id>.near --gas=200000000000000
near view name.near get_all_fields '{"from_index": 0, "limit": 3}'
near view name.near get_fields_by_pool '{"pool_id": "<pool_id>.poolv1.near"}'
LESSON 4 - STAKING POOLSβ
NEAR uses a staking pool factory with a whitelisted staking contract to ensure delegatorsβ funds are safe. In order to run a validator on NEAR, a staking pool must be deployed to a NEAR account and integrated into a NEAR validator node. Delegators must use a UI or the command line to stake to the pool. A staking pool is a smart contract that is deployed to a NEAR account.
Deploy a Staking Pool Contractβ
Deploy a Staking Poolβ
Calls the staking pool factory, creates a new staking pool with the specified name, and deploys it to the indicated accountId.
For Guildnet
near call stake.guildnet create_staking_pool '{"staking_pool_id": "<pool id>", "owner_id": "<accountId>", "stake_public_key": "<public key>", "reward_fee_fraction": {"numerator": 5, "denominator": 100}}' --accountId="<accountId>" --amount=30 --gas=300000000000000
For Testnet
near call pool.f863973.m0 create_staking_pool '{"staking_pool_id": "<pool id>", "owner_id": "<accountId>", "stake_public_key": "<public key>", "reward_fee_fraction": {"numerator": 5, "denominator": 100}}' --accountId="<accountId>" --amount=30 --gas=300000000000000
For Mainnet
near call poolv1.near create_staking_pool '{"staking_pool_id": "<pool id>", "owner_id": "<accountId>", "stake_public_key": "<public key>", "reward_fee_fraction": {"numerator": 5, "denominator": 100}}' --accountId="<accountId>" --amount=30 --gas=300000000000000
From the example above, you need to replace:
- Pool ID: Staking pool name, the factory automatically adds its name to this parameter, creating {pool_id}.{staking_pool_factory} Examples:
nearkat.stake.guildnetfor guildnetnearkat.pool.f863973.m0for testnetnearkat.poolv1.nearfor mainnet
- Owner ID: The NEAR account that will manage the staking pool. Usually your main NEAR account.
- Public Key: The public key in your validator_key.json file.
- 5: The fee the pool will charge (e.g. in this case 5 over 100 is 5% of fees).
- Account Id: The NEAR account deploying the staking pool.
Be sure to have at least 30 NEAR available, it is the minimum required for storage.
To change the pool parameters, such as changing the amount of commission charged to 1% in the example below, use this command:
near call <pool_name> update_reward_fee_fraction '{"reward_fee_fraction": {"numerator": 1, "denominator": 100}}' --accountId <account_id> --gas=300000000000000
You will see something like this:

If there is a βTrueβ at the End. Your pool is created.
You have now configure your Staking pool.
For Guildnet & Testnet You can go to OSA Discord to ask for some guildnet or testnet token to be a validator, https://discord.gg/GrBqK3ZJ2T
Configure your staking pool contractβ
Replace:
- Pool Name β pool_id.staking_pool_factory
- Owner Id
- Public Key
- Reward Fraction
- Account Id
near call <pool name> new '{"owner_id": "<owner id>", "<public key>": "<public key>", "reward_fee_fraction": {"numerator": <reward fraction>, "denominator": 100}}' --accountId <accountId>
Manage your staking pool contractβ
HINT: Copy/Paste everything after this line into a text editor and use search and replace. Once your pool is deployed, you can issue the commands below:
Retrieve the owner ID of the staking poolβ
Command:
near view {pool_id}.{staking_pool_factory} get_owner_id '{}'
Issue this command to retrieve the public key the network has for your validatorβ
Command:
near view {pool_id}.{staking_pool_factory} get_staking_key '{}'
If the public key does not match you can update the staking key like this (replace the pubkey below with the key in your validator.json file)β
near call {pool_id}.{staking_pool_factory} update_staking_key '{"stake_public_key": "<public key>"}' --accountId <accountId>
Working with Staking Poolsβ
NOTE: Your validator must be fully synced before issuing a proposal or depositing funds.
Proposalsβ
In order to get a validator seat you must first submit a proposal with an appropriate amount of stake. Proposals are sent for epoch +2. If you send a proposal now and it is accepted, you get the seat at the start of the epoch after the next one. A validator that is already in the validator set stays a candidate in later epochs without a new proposal. It must not be kicked out, and its stake must stay above the seat price. To send a proposal we use the ping command. A proposal is also sent if a stake or unstake command is sent to the staking pool contract.
To note, a ping also updates the staking balances for your delegators. A ping should be issued each epoch to keep reported rewards current on the pool contract. You could set up a ping using a cron job.
Staking Pools Factories for each network:
- GuildNet: stake.guildnet
- TestNet: pool.f863973.m0
- MainNet: poolv1.near
Transactionsβ
Deposit and Stake NEARβ
Command:
near call <staking_pool_id> deposit_and_stake --amount <amount> --accountId <accountId> --gas=300000000000000
Unstake NEARβ
Amount in yoctoNEAR.
Run the following command to unstake:
near call <staking_pool_id> unstake '{"amount": "<amount yoctoNEAR>"}' --accountId <accountId> --gas=300000000000000
To unstake all you can run this one:
near call <staking_pool_id> unstake_all --accountId <accountId> --gas=300000000000000
Withdrawβ
Unstaking takes 2-3 epochs to complete, after that period you can withdraw in YoctoNEAR from pool.
Command:
near call <staking_pool_id> withdraw '{"amount": "<amount yoctoNEAR>"}' --accountId <accountId> --gas=300000000000000
Command to withdraw all:
near call <staking_pool_id> withdraw_all --accountId <accountId> --gas=300000000000000
Pingβ
A ping issues a new proposal and updates the staking balances for your delegators. A ping should be issued each epoch to keep reported rewards current.
Command:
near call <staking_pool_id> ping '{}' --accountId <accountId> --gas=300000000000000
Balances Total Balance Command:
near view <staking_pool_id> get_account_total_balance '{"account_id": "<accountId>"}' --gas=300000000000000
Staked Balanceβ
Command:
near view <staking_pool_id> get_account_staked_balance '{"account_id": "<accountId>"}'
Unstaked Balanceβ
Command:
near view <staking_pool_id> get_account_unstaked_balance '{"account_id": "<accountId>"}'
Available for Withdrawalβ
You can only withdraw funds from a contract if they are unlocked.
Command:
near view <staking_pool_id> is_account_unstaked_balance_available '{"account_id": "<accountId>"}'
Pause / Resume Stakingβ
Pauseβ
Command:
near call <staking_pool_id> pause_staking '{}' --accountId <accountId>
Resumeβ
Command:
near call <staking_pool_id> resume_staking '{}' --accountId <accountId>
LESSON 5 - MONITORINGβ
Log Filesβ
The log file is stored in systemd depending on your setup.
Systemd Command:
journalctl -n 100 -f -u neard | ccze -A
Log file sample:
Validator | 1 validator
INFO stats: #85079829 H1GUabkB7TW2K2yhZqZ7G47gnpS7ESqicDMNyb9EE6tf Validator 73 validators 30 peers β¬ 506.1kiB/s β¬ 428.3kiB/s 1.20 bps 62.08 Tgas/s CPU: 23%, Mem: 7.4 GiB
- Validator: A βValidatorβ will indicate you are an active validator
- 73 validators: Total 73 validators on the network
- 30 peers: You current have 30 peers. You need at least 3 peers to reach consensus and start validating
- #46199418: block β Look to ensure blocks are moving
RPCβ
Any node within the network offers RPC services on port 3030 as long as the port is open in the nodes firewall. The NEAR-CLI uses RPC calls behind the scenes. Common uses for RPC are to check on validator stats, node version and to see delegator stake, although it can be used to interact with the blockchain, accounts and contracts overall.
Find many commands and how to use them in more detail here:
Command:
sudo apt install curl jq
Common Commands:β
Check your node version:β
Command:
curl -s http://127.0.0.1:3030/status | jq .version
Check Delegators and Stakeβ
Command:
near view <your pool>.stake.guildnet get_accounts '{"from_index": 0, "limit": 10}' --accountId <accountId>.guildnet
Check Reason Validator Kickedβ
Command:
curl -s -d '{"jsonrpc": "2.0", "method": "validators", "id": "dontcare", "params": [null]}' -H 'Content-Type: application/json' https://rpc.openshards.io | jq -c ".result.prev_epoch_kickout[] | select(.account_id | contains ("<POOL_ID>"))" | jq .reason
Check Blocks Produced / Expectedβ
Command:
curl -s -d '{"jsonrpc": "2.0", "method": "validators", "id": "dontcare", "params": [null]}' -H 'Content-Type: application/json' http://localhost:3030/ | jq -c ".result.current_validators[] | select(.account_id | contains ("POOL_ID"))"
Prometheusβ
Monitoring disk, CPU, memory, network io, missed blocks, and peers is critically important to a healthy node. Prometheus and Grafana combined provide monitoring and visual reporting tools. Please note that Prometheus is best set up on another machine due the storage requirement for logs.
Installationβ
Command:
sudo apt-get update
Command:
sudo apt-get install prometheus prometheus-node-exporter prometheus-pushgateway
prometheus-alertmanager
Check that Prometheus was installed and is in your path.
prometheus --version
Start Servicesβ
Command:
sudo systemctl status prometheus
Command:
sudo systemctl status prometheus-node-exporter
Command:
sudo vi /etc/prometheus/prometheus.yml
Update the targets and saveβ
targets: [βlocalhost:9093β, βlocalhost:3030β]
Reference the rules fileβ
Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "rules.yml"
Setup Postfix emailβ
Command:
sudo apt-get install mailutils
Setup Responsesβ
- internet site
- enter a domain name (Used for the from email address)
Command
sudo vi /etc/postfix/main.cf
Update and save the config fileβ
- inet_interfaces = all to inet_interfaces = localhost
- inet_protocols = all to inet_protocols = ipv4
Restart postfixβ
Command:
sudo service postfix restart
Add the hostname used with Postfixβ
Command:
sudo vi /etc/hostname
Command
sudo nano /etc/hosts
Send a test emailβ
echo "This is the body of the email" | mail -s "This is the subject line" user@example.com
Update rules.ymlβ
Command:
sudo vi /etc/prometheus/rules.yml
groups:
- name: near
rules:
- alert: InstanceDown
expr: up == 0
for: 1m
labels:
severity: "critical"
annotations:
summary: "Endpoint {{ $labels.instance }} down"
description: "{{ $labels.instance }} of job {{ $labels.job }} "
- alert: NearVersionBuildNotMatched
expr: near_version_build{instance="yournode.io", job="near"} != near_dev_version_build{instance="yournode.io", job="near"}
for: 5m
labels:
severity: critical
annotations:
summary: "Near Node Version needs updated."
description: "Your version is out of date and you risk getting kicked."
- alert: StakeBelowSeatPrice
expr: abs((near_current_stake / near_seat_price) * 100) < 100
for: 2m
labels:
severity: critical
annotations:
description: 'Pool is below the current seat price'
Restart servicesβ
Command:
sudo systemctl restart prometheus
Command:
sudo systemctl status prometheus
sudo systemctl status prometheus
Grafanaβ
Download and add the packageβ
Command:
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
Command:
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
Command:
sudo apt-get install grafana
Setup Rulesβ
Command:
sudo vi /etc/prometheus/rules.yml
- job_name: validator
- static_configs:
- targets: ['localhost:9093']
Update alert managerβ
Command:
sudo vi /etc/init.d/prometheus-alertmanager
NAME=prometheus-alertmanager
Set the command-line argumentsβ
Command:
sudo vi /etc/default/prometheus-alertmanager
ARGS="--cluster.listen-address="
Start & Reload servicesβ
Command:
sudo systemctl daemon-reload
sudo systemctl enable grafana-server.service
sudo systemctl start grafana-server
sudo service grafana-server status
Install additional pluginsβ
sudo grafana-cli plugins install simpod-json-datasource
sudo grafana-cli plugins install ryantxu-ajax-panel
Restart service and update passwordβ
service grafana-server restart
sudo grafana-cli admin reset-admin-password admin
Open Ports to specific IPβ
Command:
sudo iptables -L
Command:
sudo iptables -A INPUT -p tcp --dport 3000 -s 66.73.0.194 -j ACCEPT
sudo netfilter-persistent save
sudo netfilter-persistent reload
Create Dashboardβ
- Add datasource Prometheus
- Add Notification channel email
- Config grafana email
Command
sudo vi /etc/grafana/grafana.ini
enabled = true
host = localhost:25
skip_verify = true
from_address = moinitor@yournode.com
from_name = Validator
Restart serviceβ
service grafana-server restart
Watch logsβ
sudo tail -f /var/log/grafana/grafana.log
Install Prometheus Exporterβ
Command:
sudo apt install golang-go
git clone https://github.com/masknetgoal634/near-prometheus-exporter.git
cd near-prometheus-exporter/
go build -a -installsuffix cgo -ldflags="-w -s" -o main .
Start exporterβ
Command:
./main -accountId <contract account id>
netstat -an | grep 9333
Update Prometheusβ
Command:
sudo vi /etc/prometheus/prometheus.yml
- job_name: near-node
scrape_interval: 15s
static_configs:
- targets: ['<NODE_IP_ADDRESS>:3030']
Update AlertManagerβ
sudo vi /etc/prometheus/alertmanager.yml
postqueue -p
amtool alert
promtool check rules /etc/prometheus/rules.yml
promtool check config /etc/prometheus/prometheus.yml
amtool check-config /etc/prometheus/alertmanager.yml
cd near-prometheus-exporter/
mv main near-exporter
sudo vi /lib/systemd/system/near-exporter.service
[Unit]
Description=NEAR Prometheus Exporter
[Service]
Restart=always
User=prometheus
EnvironmentFile=/etc/default/near-exporter
ExecStart=/opt/near-prometheus-exporter/near-exporter $ARGS
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
Update exporterβ
Command:
sudo vi /etc/default/near-exporter
# Set the command-line arguments to pass to the server.
# Set you contract name
ARGS="-accountId yournode"
Copy config and start serviceβ
Command:
sudo cp /lib/systemd/system/near-exporter.service /etc/systemd/system/near-exporter.service
sudo chmod 644 /etc/systemd/system/near-exporter.service
sudo systemctl start near-exporter
sudo systemctl status near-exporter
ps -elf | grep near-exporter
netstat -an | grep 9333
sudo systemctl enable near-exporter
LESSON 6 - TROUBLESHOOTINGβ
In this lesson you will learn about:
- Keys
- Common Errors
Keysβ
NEAR uses cryptographic keys to secure accounts and validators, each key has a public and matching private key pair.
Find more detailed information about NEAR and keys: https://docs.near.org/protocol/accounts-contracts/access-keys
Validator Keys:β
To manage and sign transactions a node_key and validator_key.
Node Key:β
Used to communicate with other peers and is primarily responsible for syncing the blockchain.
~/.near/<network>/node_key.json
Common Issue:
In a failure situation, copy the validator_key to the backup node. You do not need to copy the node_key. See LESSON 7 - NODE FAILOVER.
Validator Key:β
Used to sign and validate blocks.
~/.near/<network>/validator_key.json
Common Issues:
- The validator key used to initialize the staking contract is not the one listed in validator_key.json
- The account Id used when the node was initialized is not the same as the one in validator_key.json
Common Errors & Solutionsβ
Submitting a proposal before the validator is syncedβ
A validator must be fully synced before submitting a proposal to enter the validator set. If you are not fully synced and you entered the validator set your log will be filled with errors.
Resolution Wait until 4 epochs until you have been kicked from the validator set. Delete the data directory and resync. Be sure to be fully synced before staking actions or pings go to your pool.
1. Stop the node
2. rm -Rf ~/.near/<NETWORK_ID>/data
3. Start the node
Starting the node when another instance is runningβ
On occasion, a process will get disconnected from the shell. The common error seen when trying to start the node will be:
Err value: Os { code: 98, kind: AddrInUse, message: "Address already in use" }',
Resolution Find the pid of the running process
ps -elf | grep neard
Kill the hung pid
kill -9 <PID>
Running out of Disk spaceβ
If you find no space left on device errors in the validator log files then you likely ran out of disk space.
1. Stop the node
2. Failover to a backup node
3. Cleanup files or resize disk
4. Fail back over.
Not enough stake to obtain a validator seatβ
You can check that your validator proposal was accepted by checking
near proposals
Not producing blocksβ
Your validator shows zero produced blocks in near validators current, while its expected blocks are above zero.
On mainnet, only validators that hold one of the 100 block producer seats are expected to produce blocks. Other validators only validate chunks. For them, zero blocks and zero chunks is normal. Check num_produced_endorsements against num_expected_endorsements in the validators RPC response instead.
validator_key.json must contain your staking pool id as account_id. It must also contain the public key that is registered for your pool in the current epoch. If the account_id is in the current validator set but the public key differs from the registered key, neard stops with an error that starts with validator key mismatch for <account>. The node checks this once per epoch after it is synced. If the account_id is not in the current validator set, the node does not act as a validator and produces nothing.
Two keys matter here. The key that is active for your pool in the current epoch must be in validator_key.json. The key stored in the staking pool contract is the key that the pool uses in its next stake proposal. The two keys are normally the same.
Get the key that is active in the current epoch
curl -s -d '{"jsonrpc": "2.0", "method": "validators", "id": "dontcare", "params": [null]}' -H 'Content-Type: application/json' http://localhost:3030/ | jq -r '.result.current_validators[] | select(.account_id == "<POOL_ID>") | .public_key'
If neard has stopped, send this request to any synced RPC node instead of localhost.
Get the key in validator_key.json
cat ~/.near/<network>/validator_key.json | grep public_key
Note: These two keys must match. Otherwise the node stops with the error above.
Get the key stored in the staking pool contract
near view <POOL_ID> get_staking_key '{}'
Note: When you call
update_staking_key, the contract stores the new key at once. The new key becomes active two epochs later. During those two epochs, the contract key and the active key differ, which is expected. Keep the active key invalidator_key.jsonuntil the new key becomes active.
To rotate the key on purpose, follow the steps in Can Keys be Changed?. The command that registers the new key in the pool is:
Update staking pool key
near call <POOL_ID> update_staking_key '{"stake_public_key": "<PUBLIC_KEY>"}' --accountId <ACCOUNT_ID>
Check the account_id is set to the staking pool contract
vi ~/.near/<network>/validator_key.json
"account_id" = <staking pool id>
Note: if it does now match the staking pool id upate it and restart your node.
Not producing enough blocks or chunksβ
A validator is kicked out of the validator set when it produces too few blocks, chunks, or chunk endorsements. On mainnet, block and chunk producers are kicked out below 80% of their expected blocks or chunks. Validators with no expected blocks and no expected chunks (chunk validators only) are kicked out below 70% of their expected chunk endorsements, with reason NotEnoughChunkEndorsements. You can check the expected and produced numbers via NEAR-Cli and RPC. For endorsements, compare num_produced_endorsements with num_expected_endorsements.
NEAR-CLI
near validators current | grep <pool id>
RPC
curl -s -d '{"jsonrpc": "2.0", "method": "validators", "id": "dontcare", "params": [null]}' -H 'Content-Type: application/json' http://localhost:3030/ | jq -c ".result.current_validators[] | select(.account_id | contains ("<POOL_ID>"))"
Check the reason kicked
curl -s -d '{"jsonrpc": "2.0", "method": "validators", "id": "dontcare", "params": [null]}' -H 'Content-Type: application/json' https://rpc.openshards.io | jq -c ".result.prev_epoch_kickout[] | select(.account_id | contains ("<POOL_ID>"))" | jq .reason
Not exporting the correct environmentβ
Be sure that you are using the correct environment each time you run NEAR-Cli
export NEAR_ENV=<mainnet,testnet,guildnet>
Incompatible CPU without AVX supportβ
neard needs the CPU instructions listed in the CPU check under Prerequisites in Lesson 3. On a CPU without them, neard crashes with an illegal instruction error.
Not running on SSD drivesβ
NEAR Requires a 1 second block time and HDD disks are just not fast enough. SDD drives are required and SSD NVME drives are recommended.
Inconsistent Internet connectionβ
NEAR requires a 1 second block time, so any latency from your internet provider can cause you to miss blocks.
Inability to gain enough peers for consensusβ
If you are unable to gain any peers a restart can help. The node also has no peers to start from when network.boot_nodes in config.json is empty. The reference config.json files have no boot nodes.
Logs like that :
INFO stats: #42376888 Waiting for peers 0/0/40 peers β¬ 0 B/s β¬ 0 B/s 0.00 bps 0 gas/s CPU: 2%, Mem: 91.3 MiB
Resolution
Check the boot nodes in the node's config.json. With the default home directory, run:
jq -r '.network.boot_nodes' ~/.near/config.json
If it prints an empty line, add boot nodes as described in Add boot nodes, and restart the node. Downloading the reference config.json again does not help, because it has no boot nodes.
LESSON 7 - NODE FAILOVERβ
It is an unspoken requirement to maintain a secondary node in a different location in the event of a hardware or network failure on your primary node.
You will need to set up a standard node (without a validator key) and a different node_key. The node will be set up using the normal process see Setup a Validator Node.
The Failover/backup node instruction page describes two ways to set up the backup node and fail over. The steps below follow its first option. In that option, the backup node restarts to take over. That page also gives the store.load_mem_tries_for_tracked_shards setting for each step.
While the backup node is on standby, it tracks all shards without memtries. Set this in its config.json:
"tracked_shards_config": "AllShards",
and in its store section:
"load_mem_tries_for_tracked_shards": false,
Use tracked_shards_config, not the deprecated tracked_shards key. A config.json written by neard init already has tracked_shards_config. The node does not start when both keys are set.
Failing Overβ
- Copy
validator_key.jsonto the backup node. - In the backup node's
config.json, settracked_shards_configto"NoShards"andstore.load_mem_tries_for_tracked_shardstotrue. On mainnet and testnet, a node with the key of a current validator stops at startup if it tracks all shards. - Stop the primary node.
- Restart the backup node.
You do not need to copy node_key.json. To fail back to the primary node, stop the backup node and remove validator_key.json from it. Then start the primary node. The failover/backup node instruction page notes that a node that has run as a validator and tracked a single shard cannot go back to tracking all shards. To make it a standby again, resync its data.