How to update your etcd space alerts for better etcd monitoring

Follow
Table of Contents

Task

An alert you may have seen is Database usage close to the quota 500M in your Rancher 2.x cluster.

This is a default etcd alert built into Rancher, you can find more info on the default alerts here: Rancher v2.x Default Alerts

Upon further examination of the alert, you see the description of

A warning alert is triggered when the size of etcd exceeds 500M.

This alert is somewhat misleading as the default etcd size is 2GB. This alert is also at a severity level of Warning. Below, we suggest configuring your etcd alerts to better utilize the alert thresholds and to avoid the default alert constantly going off.

Pre-requisites

  • Running Rancher v2.x

Resolution

You can access the alerts by going to Tools -> Alerts at the cluster level. From here, clone the default alert three times. Once the 3 clones are created, disable the default alert so you always retain a clean reference alert.

Alert for 1GB usage - INFO

Alert for 1.5GB usage - WARNING

Alert for 1.75GB usage - CRITICAL

If you are running out of space, please reference the following two links. There will be an upcoming KB article walking through the resize process for etcd.

Rancher v2.x etcd options

etcd space quota documentation

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.