Categories
Getting Started

Validator Automation

Running your validator and keeping it updated can be completely automated. In this guide, we will perform this in two parts.

Part 1 – Automating the start up and restart process

Running your validator as a systemd unit is an easy way to manage running your validator in the background and have it automatically start up after your system is restarted or after a panic error.

We will be using nano as the file editor throughout this guide. When you have finished editing the file simply hit CTRL+X to save and exit. Or you can use your own favourite editor.

Make sure you know the USERNAME that you used when installing the validator. If you get this bit wrong it will not work.

Let’s create the validator systemd unit config file:

sudo nano /etc/systemd/system/validator.service

Paste the following. Replace the TWO instances of elvis in the User and ExecStart sections (below) with the username that you used to install your validator (unless you did actually use elvispresley). This is the only area you need to change for your own username:

[Unit]
Description=Safecoin Validator
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=15
User=elvis
LimitNOFILE=1000000
Environment="PATH=/bin:/usr/bin"
ExecStart=/home/elvis/start.sh

[Install]
WantedBy=multi-user.target

Your startup line may differ however, if you have followed the official instructions (or the guide from this site) then the following startup script is sufficient for a pruned node

We will call the startup script start.sh, edit the script as follows:

nano ~/start.sh

Paste in the following startup code (put in your own customisations if required). Replace the –limit-ledger-size with –no-snapshot-fetch if you are running a full history node:

#!/bin/bash

cd ~

NDEBUG=1 ~/Safecoin/target/release/safecoin-validator \
        --limit-ledger-size \
        --identity ~/ledger/validator-identity.json \
        --ledger ~/ledger/validator-ledger \
        --expected-genesis-hash HfHmB9nh7EjpqoCL2DDZ559SJW4xN52gxheWPER782jW \
        --wal-recovery-mode skip_any_corrupted_record  \
        --known-validator 3USWzeHtWnMjzFCgmY5sUGWjn2J6KvM6xEgxegkrD16P \
        --known-validator 4BRXtL6nEKDVJdYPEwgGCfAAvEHT4C4Sj5peQz8kHGZu \
        --known-validator 4aWxVu4ZjKV9EtuCZi4b8Z1XwsvDWvyyfA7LRXjhDDsA \
        --vote-account ~/ledger/validator-vote-account.json \
        --authorized-voter ~/ledger/validator-identity.json \
        --entrypoint entrypoint.mainnet-beta.safecoin.org:10015 \
        --entrypoint entrypoint2.mainnet-beta.safecoin.org:10015 \
        --entrypoint entrypoint3.mainnet-beta.safecoin.org:10015 \
        --rpc-port 8328 --enable-rpc-transaction-history --no-untrusted-rpc

Let’s make the start.sh script executable:

sudo chmod +x ~/start.sh

If running then now it’s time to stop your existing validator and then enable systemd unit which will auto start on reboot or panic errors. If you are running a validator script then stop it and kill the process:

sudo killall -9 safecoin-validator

Now let’s activate systemd unit for your validator script:

sudo systemctl enable --now validator

Your validator will now start up. In the event your VPS is restarted/rebooted, it will automatically start up after 15 seconds.

To stop your validator:

sudo systemctl stop validator

To check your validator status:

sudo systemctl status validator

That’s all there is to part 1. Your validator startup and restarts are now completely automated.

Test your validator by restarting your VPS, it should auto restart. If it does not then go back to the beginning and check each step carefully. Please allow up to 5 minutes for your validator to restart.

You can review the validator system log like this:

tail -f ~/safecoin*.log

Allow 5 minutes. If it works – you can move on to Part 2.

Part 2 – Automatic rebuild, restart and log management

Occasionally there will be codebase updates that will require a manual pull from github and a rebuild and a restart. We also need to keep the log files in check. This is a manual task that we can completely automate.

With a short bash script and with the use of the crond service it’s actually very easy to do. Some staggering is added to ensure not all validators are updated at exactly the same time.

First we create the cron.sh file using nano, our favourite text editor. 

nano ~/cron.sh

Then paste in the following code block, remember to press CTRL+X afterwards to save and exit:

#!/bin/bash

# check and apply codebase updates
cd ~/Safecoin
SAFE=`git pull| grep Already | wc -l`
if [ $SAFE -ne 1 ];then
 cargo build --release
 sleep $((1 + $RANDOM % 1800))
 systemctl restart validator
fi

# truncate the log
truncate -s 0 ~/safecoin-validator-*.log

Let’s make the cron.sh executable:

chmod +x ~/cron.sh

Now let’s add cron.sh to the crontab:

crontab -e

Using nano (default easiest option), press CTRL+V to go to the end and paste this then hit CTRL+X to save and exit:

00 * * * * ~/cron.sh

CTRL+X to save and exit. Should see “crontab: installing new crontab” if your save was successful.

Run the cron.sh manually to ensure there are no obvious errors:

~/cron.sh

If no errors, then you are finished. Your validator will now automatically restart on reboot, download updates, rebuild, restart and keep your validator log files in check.

Troubleshooting

In the event that you are all setup and you have been running successfully for sometime and now for an unknown reason your validator is delinquent for more than an hour, and it never catches up then check out the log file for clues:

tail -f ~/safecoin*.log

Got an “assertion failed”? Then manual action will be required, check out the #validators channel on Discord for further instructions. If there’s no other clues then the only option is to remove the ledger and restart, you can do it like this:

sudo systemctl stop validator

sudo rm -rf ~/ledger/validator-ledger

sudo systemctl start validator