Some users run into an issue where fluentd buffers grow indefinetely and fill up the disk on nodes. This documentation explains how to limit this size in V2 Logging.
- Rancher 2.5.x and above
- Logging App installed with Cluster Explorer
When you're ready to configure your Output (the location that the logs are sent to) you can get there with the following steps after you have Logging installed:
- Cluster Explorer
- Dropdown in the top left
From here, you can edit the Output as YAML to access the buffer configurations. Using ElasticSearch as an example you can fill out the form easily, but then Edit as YAML:
apiVersion: logging.banzaicloud.io/v1beta1 kind: ClusterOutput metadata: name: "elasticsearch-output" namespace: "cattle-logging-system" elasticsearch: host: 18.104.22.168 index_name: some-index port: 9200 scheme: http buffer: type: file total_limit_size: 2GB
As far as the buffer size is concerned, the
total_limit_size is the important parameter to change. 2GB is a good starting point and is unlikely to need to be changed, but you can adjust this based on your needs.
As stated in the pre-reqs, this is only possible on the V2 Logging App which is available in Rancher 2.5. This uses the Banzai Cloud logging operator, which allows us to use CRDs to configure it.