September 11, 2019
This advisory was first communicated by email, on the date mentioned above, to all Rancher customers with a then active support subscription.
Dear Rancher 2.x user,
This is an operational advisory from Rancher Support that is addressed to users of Rancher 2.x.
Users of cert-manager, who use it as part of Rancher 2.x's HA installation must take action to upgrade cert-manager in order to avoid downtime.
If you are wondering what cert-manager is
cert-manager is a utility for Kubernetes that Rancher uses to automatically generate and renew TLS certificates for HA deployments of Rancher. Those certificates can be self-signed or issued through LetsEncrypt. cert-manager is also available as an application from the App Catalog, and this version expressly uses LetsEncrypt to generate TLS certificates for ingress resources in the cluster.
LetsEncrypt recently sent emails out to certificate holders using cert-manager, announcing that they would be discontinuing support for cert-manager versions less than 0.8 on November 1, 2019. They also announced that they will continue to deprecate and expire support for non-current cert-manager versions an a regular three month rotation.
cert-manager development is overseen by Jetstack, a Kubernetes consultancy in the UK. Control of the cert-manager helm chart recently moved from the Helm Stable repo to Jetstack's private repo, which allows them to release new versions quickly and efficiently. They are currently on 0.9.1 and are about to release 0.10.
For your Rancher/RKE deployments
Our documentation demonstrates how to install the latest version of cert-manager for new Rancher installations and how to update from older versions to the latest version for existing Rancher installations. This upgrade will not affect certificates currently installed in the Kubernetes cluster, nor will it affect running workloads. It only upgrades the cert-manager engine and migrates it from the Helm Stable chart to the Jetstack chart. Once the upgrade is complete, the engine will continue to renew certificates from LetsEncrypt.
If you're using cert-manager with self-signed certificates, we still recommend that you upgrade. Jetstack's development moves quickly, and because cert-manager is a core component of the Rancher deployment, it's important that it stays up to date.
Know what's coming up
Jetstack introduced a new ACME solver configuration for certificate generation in 0.8. They will support both methods of generating certificates until at least version 1.0. While the steps to change from one format to the other are relatively easy when manually configuring cert-manager, performing the migration as part of an automated application upgrade presents unique challenges.
Rancher documentation references the data migration docs provided by cert-manager and we recommend migrating to the new format while upgrading to the latest cert-manager.
We are working to ensure that Rancher and cert-manager are always compatible, and our documentation will always reflect the latest production changes that cert-manager requires.
Please upgrade your cert-manager installations as soon as possible before the November 1, 2019 deadline from LetsEncrypt, and please include cert-manager in your production upgrade workflow for Rancher and RKE.
Simply submit a request via this portal referencing this article and we will track and respond to your question as a Support Ticket.
Watch this video on YouTube, where our fellow Rancher Adrian Goins walks you through how to upgrade Kubernetes to the latest version of Cert Manager from Jetstack. It covers moving off of the Rancher App or the Helm Stable chart, and it shows two workarounds for problems that can cause your upgrade to fail.
Rancher Support Team