In a Rancher v2.x provisioned Kubernetes cluster, with Rancher Cluster Logging configured, the
rancher-logging-fluentd Pod logs on Linux worker role only nodes show warning messages of the following format:
2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kubelet_5c47838dd4af749a7a0d1c457b04a6d7b905e680157718063c8e5d9eb61268fa.log unreadable. It is excluded and would be examined next time. 2019-12-05 10:58:27 +0000 [warn]: #0 /var/lib/rancher/rke/log/kube-proxy_d2beb2e667eefbd6d95355082af4bc61c367fc4c220d9f1d165d15a8c8be2ab1.log unreadable. It is excluded and would be examined next time.
The output of
ls /var/lib/rancher/rke/log/ on affected workers shows that these files referenced in the warning log messages are broken symlinks. In addition,
docker ps output shows the container ID for the currently running
kube-proxy containers does not match the IDs in these filenames.
- A Rancher v2.x provisioned Kubernetes cluster, using either custom nodes or nodes hosted in an infrastructure provider.
- Rancher Cluster Logging configured on the cluster, with the 'Include System Log' option set.
- Worker role only nodes in the cluster.
These warning level messages in the
rancher-logging-fluentd Pod are the result of the issue tracked in Rancher GitHub issue #22549.
The container log symlinks in
/var/lib/rancher/rke/log/ for cluster component containers (
nginx-proxy) on worker nodes in Rancher launched clusters are not cleaned up when these components are re-created, i.e. due to a Kubernetes version upgrade, or other configuration update for these components.
As a result these broken symlinks persist and cause the
log unreadable warning messages when the
rancher-logging-fluentd Pod attempts to parse files in the
This warning message itself is harmless and can be ignored.
The request to handle automatic clean-up of these log symlinks on worker nodes, in Rancher provisioned clusters, is tracked in Rancher GitHub issue #22549.