Logs are often known as one of the three pillars of observability. They are great for debugging code running in production. However they can also create a lot of data and noise. You want to have deep insight into your application but at the same time you only care about the information that matters. Thus logging libraries have many logging levels, e.g.
- panic (5)
- fatal (4)
- error (3)
- warn (2)
- info (1)
- debug (0)
- trace (-1)
You specify your global logging level and all logs with this level and greater will show up. For example, if you set your global logging level to
warn (2) all logs with the levels
fatal (4) and
panic (5) will show up. That's great for production code because you can log only the most important things during normal operation. In case you've got a problem and have to dig deeper simply lower the logging level and all
debug logs appear as well.
So how do you change the logging level for your production application that is running in a Kubernetes cluster?
For this post we’re using Go but the principles can be applied to any programming language. We assume you have multiple replicas of your application running to ensure zero downtime deployments. We’re using a ConfigMap to store the current logging level. This ConfigMap is then translated into environment variables that the application reads during startup.
Save this very simple ConfigMap to a file called
Move it into your cluster by using
$ kubectl apply -f configmap.yaml
It should appear on your Kubernetes dashboard. Check your dashboard
At the moment our application doesn’t know anything about this ConfigMap. We have to connect our application to it by adding some information to the deployment. Here we have a basic deployment and we add all the data starting at
- name: app
- name: LOG_LEVEL
It’s really straightforward. We specify a new environment variable
LOG_LEVEL that reads the value from our ConfigMap named
configuration. Since our ConfigMap might have multiple key/value pairs we tell the deployment to use the
log_level key. Now we have the environment variable
LOG_LEVEL with the content
Here is a little diagram to illustrate the whole concept
Now everything is ready and you’re able to change the logging level on the fly. Simply change the content for the
log_level key in your ConfigMap and apply it to the cluster. Then initiate a rolling restart so your pods will pick up the new environment variables. We're using
make for automation.
On our local machines we keep the application outside the Kubernetes cluster. It makes debugging easier and restarts during development much faster since you don’t have to create new containers. We simply use
make to set the environment variable.
JAVA_OPTS -e "-Dlogging.config=<path for logback>\logback-spring.xml"Run your application and change path accordingly in configMap for Env Variable of JAVA_OPTS
Now, Every time if there’s changes in Logback.xml then configMap detects it and update accordingly log level in Application.
You can use following property in logback.xml for auto scan feature i.e.
<configuration debug=”true” scan=”true” scanPeriod=”10 seconds”>
You can mount configMap as Volume also. Check out my blog for that.