Maintenance

Monthly:

1. Check GitHub to see if there are any new blockchain node releases and determine if an upgrade is needed. Most importantly, check to see there is a fork coming up. If so follow directions here
   1. Primarily look for forks
2. Make sure have a recent, valid snapshot for each chain
3. Consider making a Gitbook sheet to track monthly maintenance checks

Monitoring and Alerts

Cointainers

Alerts when storage volume utilization exceeds 80%. If increase is needed follow directions here

Watchtower

Watchtower block height monitors track the difference between the WT ingester height and the node height. If they are different by more than a certain amount, an alert goes out to OpsGenie Coins Team rotation and Discord #critical

Monitors are configured here: https://app.datadoghq.com/monitors/manage?q=watchtower prod

Dashboard with status for each chain:

https://app.datadoghq.com/dashboard/n39-z2z-fbu/watchtower-block-heights-prod?from_ts=1632764971973&to_ts=1632851371973&live=true

Maintenance Tasks

Increase storage volume

There are alerts whenever more than 80% of volume is used. To increase: