How to send logs to Amazon Web Services (AWS) CloudWatch with the new logging services available on Rancher v2.5.x

Follow
Table of Contents

Task

New logging service introduced in Rancher v2.5.x allows users to send logs to Amazon Web Services (AWS) Cloudwatch. This article details how to send logs to AWS CloudWatch with the new logging services available on Rancher v2.5.x, in Rancher Kubernetes Engine (RKE) CLI or Rancher v2.5.x provisioned Kubernetes clusters.

Pre-requisites

  • A Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes cluster with Logging in Rancher 2.5 enabled
  • Rancher v2.5.x
  • AWS IAM policy with at least the following permissions, the policy is attached to either an IAM user with credentials, or an EC2 instance profile which is attached to the nodes in the cluster
    {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                  "logs:PutLogEvents",
                  "logs:CreateLogGroup",
                  "logs:PutRetentionPolicy",
                  "logs:CreateLogStream",
                  "logs:DescribeLogGroups",
                  "logs:DescribeLogStreams"
              ],
              "Effect": "Allow",
              "Resource": "*"
          }
      ]
    }

Steps

  1. Ensure the Rancher v2.5 logging is enabled on the cluster; visit and follow the Logging section in the Rancher docs if it is not already enabled.
  2. Optional Create a secret containing the AWS Access key ID and Secret access key in cattle-logging-system namespace:
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    data:
      id: <AWS Access key ID>
      secret: <AWS Secret access key>
    kind: Secret
    metadata:
      name: aws
      namespace: cattle-logging-system
    type: Opaque
    EOF
    > Note, this step is not required if using an EC2 instance profile
  3. Create the ClusterOutput and ClusterFlow to forward the logs to the CloudWatch.
    cat <<EOF | kubectl apply -f -
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: ClusterOutput
    metadata:
      name: cloudwatch
      namespace: cattle-logging-system
    spec:
      cloudwatch:
        auto_create_stream: true  #Set to false to disable automatically create Log Stream under the Log Group
        aws_key_id:
          valueFrom:
            secretKeyRef:
              key: id
              name: aws
        aws_sec_key:
          valueFrom:
            secretKeyRef:
              key: secret
              name: aws
        buffer:
          timekey: 30s
          timekey_use_utc: true
          timekey_wait: 30s
        log_group_name: <LOG GROUP NAME ON THE CLOUDWATCH>
        log_stream_name: <LOG STREAM NAME UNDER THE LOG GROUP>
        region: <AWS REGION>
    ---
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: ClusterFlow
    metadata:
      name: logging
      namespace: cattle-logging-system
    spec:
      globalOutputRefs:
      - cloudwatch
    EOF
    > Note, the aws_key_id and aws_sec_key should be removed if using an EC2 instance profile
  4. The logs will start sending to the CloudWatch once the ClusterOuput and ClusterFlow are created. You may visit the docs in the below to explore all the available configurations for the Rancher v2.5 logging.

Further reading

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.