Missing container metrics in Rancher monitoring for AKS clusters in Rancher v2.2 - v2.4

Issue

There is a bug affecting Rancher monitoring v1 in Rancher v2.2 - v2.4, in which some metrics for Rancher-managed AKS clusters are missing. When viewing Grafana dashboards for affected clusters, some graphs will display a Not enough data for graph or No Data message.

Pre-requisites

• A Rancher v2.2 - v2.4 instance
• A Rancher-managed AKS cluster with monitoring enabled

Root cause

This is due to the Kubelet scrape port not being properly set for Prometheus, as the cluster is not being detected as an AKS instance.

Resolution

Simply adding the exporter-kubelets.https: true answer in the advanced section of the monitoring configuration page is all that is needed.

Navigate to the cluster view in the Rancher UI for the relevant cluster and click Tools > Monitoring. Click Show advanced options in the bottom-right corner, and add exporter-kubelets.https: true under Answers, per the screenshot below. Once you hit Save container metrics should start being populated in Prometheus and will be visible in the Grafana dashboards.

It is worth noting that this is considered a "hidden answer" which means that it will not remain visible under Answers on the monitoring configuration page. As a result, you will have to re-add this answer every time there is a monitoring update or configuration change.

Further reading

• This issue is tracked in GitHub issue #27550.