October 17, 2019
This advisory was first communicated by email, on the date mentioned above, to all Rancher customers with a then active support subscription.
Dear Rancher 2.x user,
We sent you an advisory on this topic last month regarding HA installs, users must upgrade cert manager in order to keep working with LetsEncrypt. Please continue using v0.9.1 as recommended in our HA install documentation. The later versions require different updates/upgrades that we will add in within the next month or so. See original advisory email below.
As a follow-up to that advisory, we want to bring the following scenario as well to your attention:
The Rancher Application Catalog represents a curated list of applications that we use ourselves, and we've taken the steps to extend their Helm charts into the question and answer form that makes them so easy to deploy and upgrade. The App Catalog strives to be a one-click upgrade solution, where if no answers have changed, upgrades are quick and painless.
Occasionally an application makes a change that isn't backwards compatible, or the deployment architecture changes into one that requires manual steps outside of what one can do from within Helm or the Application Catalog. This happens most often with alpha software, which we all know that we aren't really supposed to be using in production. It's just so tempting, and when it works well enough, it's easy to forget that alpha software means that changes between versions may not be backwards or forwards compatible.
Jetstack has been developing cert-manager actively, rolling out new features at a pace that they have started deploying through their own Helm repository instead of the Helm stable repository.
We use cert-manager via Helm for certificate management of the Rancher Server itself, and we've also made it available in the App Catalog for installation in downstream clusters.
Because of the changes that Jetstack is rolling out between versions, it is, unfortunately, not possible to adhere to the "one-click upgrade" objectives of the App Catalog. This is not a negative reflection of Jetstack; if anything, it's a byproduct of their success. All the same, we're going to temporarily remove cert-manager from the App Catalog. You'll still be able to deploy it directly, either via Helm or through standard manifests, both of which are clearly explained in their documentation.
Jetstack also does a fantastic job of outlining the specific instructions for upgrading from each version, calling out special instructions when necessary, and you'll find their dev team in `#cert-manager` on the Kubernetes Slack.
How to Migrate
If the version that you're running is from the App Catalog, you'll need to remove the app and install the chart or manifests from Jetstack. Most of the steps are outlined in a YouTube video from Adrian Goins, but essentially consist of:
1. Make a backup of all cert-manager resources. This will back up the ClusterIssuer and Certificates.
2. On the Apps page in Rancher delete the cert manager app. This will not remove any certificates or affect any running workloads.
3. Install the latest version of cert-manager using either Helm or direct manifests.
4. Restore your backup - this will recreate the ClusterIssuer and any other resources. If you are upgrading to 0.11.0 or newer, you will need to follow the additional steps outlined here and here to update your ClusterIssuer/Issuer(s) and any annotations.
5. If necessary, perform an upgrade of cert-manager according to the instructions.
6. Test your new installation.
Take care to observe any changes between versions when you upgrade. A safe method is to upgrade through each version and test along the way, instead of making a leap over several versions.
Jetstack is rapidly approaching a 1.0 release of cert-manager, after which the pace of changes will hopefully subside. When we're able to ensure that your experience with cert-manager in the App Catalog is easy and painless, we'll look into bringing it back into the App Catalog.
Until then, you'll be totally fine working directly with the upstream version. There's nothing better for automatic certificate management than cert-manager, and the Jetstack team is truly fantastic. We're looking forward to continuing to use cert-manager long into the future.
If there are any questions, simply submit a request via this portal referencing this article and we will track and respond to your question as a Support Ticket.
Dear Rancher 2.x user,
This is an operational advisory from Rancher Support that is addressed to users of Rancher 2.x.
Users of cert-manager, who use it as part of Rancher 2.x's HA installation must take action to upgrade cert-manager in order to avoid downtime.
If you are wondering what cert-manager is
cert-manager is a utility for Kubernetes that Rancher uses to automatically generate and renew TLS certificates for HA deployments of Rancher. Those certificates can be self-signed or issued through LetsEncrypt. cert-manager is also available as an application from the App Catalog, and this version expressly uses LetsEncrypt to generate TLS certificates for ingress resources in the cluster.
LetsEncrypt recently sent emails out to certificate holders using cert-manager, announcing that they would be discontinuing support for cert-manager versions less than 0.8 on November 1, 2019. They also announced that they will continue to deprecate and expire support for non-current cert-manager versions an a regular three month rotation.
cert-manager development is overseen by Jetstack, a Kubernetes consultancy in the UK. Control of the cert-manager helm chart recently moved from the Helm Stable repo to Jetstack's private repo, which allows them to release new versions quickly and efficiently. They are currently on 0.9.1 and are about to release 0.10.
For your Rancher/RKE deployments
Our documentation demonstrates how to install the latest version of cert-manager for new Rancher installations and how to update from older versions to the latest version for existing Rancher installations. This upgrade will not affect certificates currently installed in the Kubernetes cluster, nor will it affect running workloads. It only upgrades the cert-manager engine and migrates it from the Helm Stable chart to the Jetstack chart. Once the upgrade is complete, the engine will continue to renew certificates from LetsEncrypt.
If you're using cert-manager with self-signed certificates, we still recommend that you upgrade. Jetstack's development moves quickly, and because cert-manager is a core component of the Rancher deployment, it's important that it stays up to date.
Know what's coming up
Jetstack introduced a new ACME solver configuration for certificate generation in 0.8. They will support both methods of generating certificates until at least version 1.0. While the steps to change from one format to the other are relatively easy when manually configuring cert-manager, performing the migration as part of an automated application upgrade presents unique challenges.
Rancher documentation references the data migration docs provided by cert-manager and we recommend migrating to the new format while upgrading to the latest cert-manager.
We are working to ensure that Rancher and cert-manager are always compatible, and our documentation will always reflect the latest production changes that cert-manager requires.
Please upgrade your cert-manager installations as soon as possible before the November 1, 2019 deadline from LetsEncrypt, and please include cert-manager in your production upgrade workflow for Rancher and RKE.
Simply submit a request via this portal referencing this article and we will track and respond to your question as a Support Ticket.
Watch this video on YouTube, where our fellow Rancher Adrian Goins walks you through how to upgrade Kubernetes to the latest version of Cert Manager from Jetstack. It covers moving off of the Rancher App or the Helm Stable chart, and it shows two workarounds for problems that can cause your upgrade to fail.
Rancher Support Team