Tail your logs with Tailing Sidecar Operator

When migrating to Kubernetes and re-architecting your applications into containers, logging is a critical piece to consider. The twelve-factor app methodology has a section dedicated to logging and outlines the importance of not worrying about routing and storage of your logs. As a best practice, applications running in containers should rely 100% on standard output (STDOUT). Unfortunately, getting logs from applications that do not write to STDOUT is non-trivial and has many things to consider. Logs which are not sent to STDOUT are not accessible through the conventional means we use regularly. You cannot use the kubectl logs command as they are not available in the /var/log/containers directory so they cannot be gathered in the same logging pipeline as other logs coming from containers. Not only have we at Sumo Logic hit this challenge in our migration to Kubernetes, but also have some of our customers. We all wanted to do something about it. That’s why I am excited to announce the release of the Tailing Sidecar Operator to make these challenges much easier.

Real example when Tailing Sidecar Operator is needed

Sometimes different kinds of logs are available in separate files. Let’s see an example with Slow Query Logs from MySQL database.

MySQL database was deployed in Kubernetes cluster with Slow Query Logs enabled. The commands below provide an example:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install my-release bitnami/mysql -f values.yaml

Slow Query Logs are by default available in

/bitnami/mysql/data/my-release-mysql-0-slow.log and these logs are accessible only from the container.

For a Kubernetes administrator it is not difficult to get them using kubectl exec command in the form like this:

$ kubectl exec -it my-release-mysql-0 --  tail -f /bitnami/mysql/data/my-release-mysql-0-slow.log 
# Time: 2021-03-25T14:13:46.150212Z
# User@Host: root[root] @  [10.1.37.76]  Id:  1779
# Query_time: 15.000808  Lock_time: 0.000000 Rows_sent: 1  Rows_examined: 1
SET timestamp=1616699611;
SELECT SLEEP(15);
# Time: 2021-03-25T14:14:20.697720Z
# User@Host: root[root] @  [10.1.37.76]  Id:  1779
# Query_time: 15.000605  Lock_time: 0.000000 Rows_sent: 1  Rows_examined: 1
SET timestamp=1616699645;
SELECT SLEEP(15);

However it is not the first thing in the mind of a Kubernetes administrator when something goes wrong in the cluster. Mostly kubectl logs command is the first idea, so it would be good to have logs available through this command.

The other issue is that Slow Query Logs are not available in /var/log/containers directory on the Kubernetes node the pod is running on. There is only one log file with logs from STDOUT for theMySQL Pod in /var/log/containers so the standard logging pipeline, like this, does not include Slow Query Logs logs.

How does the Tailing Sidecar Operator help in getting logs?

The Tailing Sidecar Operator adds additional sidecar containers to Pod and redirects logs from files to standard outputs of added containers. It makes logs from different files accessible through kubectl logs command and available in /var/log/container directory.

The concept for single tailing sidecar container is illustrated on schema below:

Tailing sidecar container uses Fluent Bit, with tail plugin and output plugin that sends records to the standard output, benefiting from its performance and features.

Tailing Sidecar Operator configuration

Configuration for tailing sidecar operator is provided through tailing-sidecar annotation added to Pod metadata:

metadata:
    annotations: 
        tailing-sidecar: <tailing-sidecar-configuration>

Tailing sidecar container needs to have provided volume with the log file and a path to the log file in the container, optionally a name of the sidecar container can be configured.

Configuration in annotation should be provided using following syntax:

metadata:
    annotations: 
        tailing-sidecar: <container0>:<volume0>:<path0>;<volume1>:<path1>;<volume2>:<path2>

Configuration for each tailing sidecar container is separated by “;” but elements in configuration of single tailing sidecar container are separated by “:”.

Tailing Sidecar Operator installation

Installation of Tailing Sidecar Operator is available through Helm Chart so before installation add Tailing Sidecar Operator Helm repository:

helm repo add tailing-sidecar https://sumologic.github.io/tailing-sidecar
helm repo update

And then install operator:

helm upgrade --install tailing-sidecar tailing-sidecar/tailing-sidecar-operator \
  -n tailing-sidecar-system \
  --create-namespace

See the power of Tailing Sidecar Operator

Coming back to the example with Slow Query Logs from MySQL database, there is only one thing which needs to be done to see the power of Tailing Sidecar Operator - the annotation must be added to Pod:

metadata:
    annotations: 
        tailing-sidecar: slow-query-logs:data:/bitnami/mysql/data/mysql-release-0-slow.log

For MySQL deployment used in example this section needs to be added to values.yaml:

primary:
    podAnnotations:
        tailing-sidecar: slow-query-logs:data:/bitnami/mysql/data/mysql-release-0-slow.log

Upgrade MySQL deployment, e.g.:

ROOT_PASSWORD=$(kubectl get secret --namespace default my-release-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode)
helm upgrade my-release bitnami/mysql --set auth.rootPassword=$ROOT_PASSWORD -f values.yaml

And see Slow Query Logs using kubectl logs:

$ kubectl logs my-release-mysql-0 slow-query-logs
# Time: 2021-03-25T16:38:16.709509Z
# User@Host: root[root] @  [10.1.37.82]  Id:    94
# Query_time: 15.000483  Lock_time: 0.000000 Rows_sent: 1  Rows_examined: 1
SET timestamp=1616704681;
SELECT SLEEP(15);
# Time: 2021-03-25T16:47:38.064183Z
# User@Host: root[root] @  [10.1.37.82]  Id:    19
# Query_time: 15.000392  Lock_time: 0.000000 Rows_sent: 1  Rows_examined: 1
use my_database;
SET timestamp=1616705243;
SELECT SLEEP(15);

Thanks to the Tailing Sidecar Operator there is a log file with Slow Query Logs in /var/log/containers so these logs will be gathered by a standard logging pipeline.

There is one important fact to mention - containers cannot be added to existing Pod. Tailing Sidecar Operator is able to add containers to Pod leveraging mutating admission webhook. In short, the Tailing Sidecar operator receives information about requests for creating a new Pod before it is created. Because of that, the operator can modify Pod specification. It is important to remember that adding or deleting elements in tailing-sidecar annotation corresponds to adding or deleting containers from Pod specification.

Benefits for Sumo Logic Customers

For Sumo Logic customers, the Tailing Sidecar Operator has been incorporated into the 2.1.0 release of our Kubernetes Collection Helm chart as an optional component. Now all Sumo Logic customers can easily leverage the Tailing Sidecar Operator to get easier logs collection from containers that do not write to STDOUT.

More about Tailing Sidecar Operator

We decided this project has great benefit to the Kubernetes community and can help anyone faced with the challenges of legacy applications that might not be able to write to STDOUT. We wanted to release this project as an open source project from which all could benefit. To get more information about Tailing Sidecar Operator please see Github repository: https://github.com/SumoLogic/tailing-sidecar

Happy Tailing!