Enabling project monitoring in a Rancher v2.2 cluster, in which cluster monitoring is enabled, fails with the Prometheus Pod in a CrashLoopBackOff.
prometheus container in the
prometheus-project-monitoring StatefulSet fails with an error of the following format:
panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x669c12] goroutine 437 [running]: net/http.(*Client).deadline(0x0, 0xc005381070, 0x40bb8f, 0xc0055e3600) /usr/local/go/src/net/http/client.go:187 +0x22 net/http.(*Client).do(0x0, 0xc005cdaa00, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/client.go:527 +0xab net/http.(*Client).Do(0x0, 0xc005cdaa00, 0x23, 0xc002802230, 0x9) /usr/local/go/src/net/http/client.go:509 +0x35 github.com/prometheus/prometheus/scrape.(*targetScraper).scrape(0xc0060fa960, 0x1fd4a60, 0xc00010ec60, 0x1fb2760, 0xc0002eb110, 0x0, 0x0, 0x0, 0x0) /app/scrape/scrape.go:471 +0x111 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc00616a100, 0xdf8475800, 0x2540be400, 0x0) /app/scrape/scrape.go:813 +0x487 created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync /app/scrape/scrape.go:336 +0x45d
- A cluster managed by Rancher v2.2
- Cluster monitoring enabled and Project monitoring enabled on the System project
Project monitoring is not compatible with the Rancher System project and should not be enabled in the System project. Starting with Rancher v2.3.0 monitoring of the System project is performed by cluster monitoring, when this is enabled, and the UI prevents enabling of project monitoring on the System project.