Log management in containers with Fluentd, Docker log drivers and Kontena

As we all know, application logs are crucial information in both debugging issues users are facing and also to see how an application is behaving. In multi-server container environments it is crucial to have one single place to check all the application logs. This blog post summarises how Kontena manages application logs and what possibilities that offers for further log processing.

Log capturing

The brilliant 12Factor principles define application logs as a stream of aggregated, time-ordered events. In the principles it is also stated that each application should not concern how and where the logs are stored, but instead just always dump logs into stdout unbuffered. This is fantastic advice as it really eases up application development from bundling different ways to store logs for each environment the application is used in.

Now with containers in use, this is even more important as container engines (Docker in this case in particular) capture the stdout and stderr streams. In the case of managing your containerized apps with the Kontena Platform, Kontena will also tap into the Docker engines log streams for each container. While streaming the application logs from each container, Kontena Agent then pushes the logs to be stored in Kontena Master's database. In Kontena Master, as the database is MongoDB, the logs are stored in a capped collection not to make them consume too much disk space. A capped collection behaves similarly to circular buffers: once the collection fills up, the older entries are overwritten to make room for new entries.

Kontena log capturing makes it possible to use one single CLI command to see application logs for all the containers in a given service, for example:

$ kontena service logs todo/web
2017-04-27T10:58:48.000Z [5]: `/` is not writable.  
2017-04-27T10:58:48.000Z [5]: Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.  
2017-04-27T10:58:49.000Z [5]: Puma starting in single mode...  
2017-04-27T10:58:49.000Z [5]: * Version 3.4.0 (ruby 2.3.1-p112), codename: Owl Bowl Brawl  
2017-04-27T10:58:49.000Z [5]: * Min threads: 0, max threads: 16  
2017-04-27T10:58:49.000Z [5]: * Environment: production  
2017-04-27T10:58:51.000Z [5]: * Listening on tcp://0.0.0.0:9292  
2017-04-27T10:58:51.000Z [5]: Use Ctrl-C to stop  
2017-04-27T10:59:31.000Z [9]: `/` is not writable.  
2017-04-27T10:59:31.000Z [9]: Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.  
2017-04-27T10:59:32.000Z [9]: Puma starting in single mode...  
2017-04-27T10:59:32.000Z [9]: * Version 3.4.0 (ruby 2.3.1-p112), codename: Owl Bowl Brawl  
2017-04-27T10:59:32.000Z [9]: * Min threads: 0, max threads: 16  
2017-04-27T10:59:32.000Z [9]: * Environment: production  
2017-04-27T10:59:34.000Z [9]: * Listening on tcp://0.0.0.0:9292  
2017-04-27T10:59:34.000Z [9]: Use Ctrl-C to stop  
2017-04-27T11:03:15.000Z [6]: `/` is not writable.  
2017-04-27T11:03:15.000Z [6]: Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.  
2017-04-27T11:03:16.000Z [6]: Puma starting in single mode...  
2017-04-27T11:03:16.000Z [6]: * Version 3.4.0 (ruby 2.3.1-p112), codename: Owl Bowl Brawl  
2017-04-27T11:03:16.000Z [6]: * Min threads: 0, max threads: 16  
2017-04-27T11:03:16.000Z [6]: * Environment: production  
2017-04-27T11:03:19.000Z [6]: * Listening on tcp://0.0.0.0:9292  
2017-04-27T11:03:19.000Z [6]: Use Ctrl-C to stop  

Log forwarding

While seeing the logs of an application (comprised of multiple containers) is fantastic, it does not offer any real search capabilities. When you have plenty of services and/or a multitude of container instances for a service, real query capabilities are necessary. The following chapters provide some alternatives on how to get your application logs streamed to a system that provides more sophisticated analytics and query capabilities. Setting up one of those systems warrants its own full post and there are plenty of good SaaS services for log management also available.

Fluentd forwarding

Kontena 1.2.0 came with an option to forward all container logs with fluentd protocol from the nodes. So while the Kontena Agent captures the container logs from Docker log streams it forwards the logs to a specified fluentd endpoint in addition to sending them to Kontena Master.

When Kontena Agent forwards the logs using fluentd, it tags all the log events with a pattern hostname.grid_name.stack.service.instance. That makes it possible to do aggregations and other analytics in the log processing system.

Docker log drivers

Another option to get logs shipped for further processing is to define log drivers to be used on Docker level.

web:  
  image: nginx
  logging:
    driver: syslog
    options:
      syslog-address: "tcp://192.168.0.42:123"

The downside of this approach is that the logs will not get stored on Kontena Master at all since Docker will send the logs directly from containers, thus Kontena Agent is not able to capture them.

Streaming logs from Kontena Master DB

Yet another option to get logs forwarded for further processing is to grab them from the Kontena Master database.

To gather logs from the Master database directly, you need to run the collector somewhere that has access to the Master database. Usually the database is not exposed to the outside world from the Master Node, so the most natural way is to run it alongside the Master.

For example, with fluentd you could use the following configuration to get the logs shipped to AWS S3:

<source>  
  type mongo_tail
  url "#{ENV['MONGODB_URL']}"
  collection container_logs
  tag_key name
  time_key created_at
  id_store_collection container_logs_tail
</source>

<match **>  
  @type s3

  aws_key_id "#{ENV['S3_ACCESS_KEY']}"
  aws_sec_key "#{ENV['S3_SECRET_KEY']}"
  s3_bucket "#{ENV['S3_BUCKET']}"
  s3_region "#{ENV['S3_REGION']}"
  buffer_type memory
  buffer_chunk_limit 256m
  buffer_queue_limit 128
  path logs/

  format json
  include_time_key true
  include_tag_key true

  s3_object_key_format %{path}/ts=%{time_slice}/%{index}_json.%{file_extension}
  time_slice_format %Y%m%d-%H
  time_slice_wait 30m
  utc
</match>  

Log processing

There's a plethora of different options available for log management and processing functionality. One of the most popular on-premise solutions is the open source ELK stack. For a commercial option, Splunk, offers fantastic capabilities but comes with a rather high price tag.

For hosted options, there's at least logz.io, Papertrail, Loggly, Logentries and AWS CloudWatch to name a few. As usual, the benefit of using a hosted solution is the non-existing maintenance burden which of course comes with a price tag.

I'm planning to do a write-up in the coming weeks on how to setup a private ELK stack with Kontena, so stay tuned.

About Kontena

Want to learn about real life use cases of Kontena, case studies, best practices, tips & tricks? Need some help with your project? Want to contribute to a project or help other people? Join the Kontena Forum to discuss more about our Kontena Platform, chat with other happy developers on our Slack discussion channel or meet people in person at one of our Meetup groups located all around the world.

Image Credits: Abstract Architecture Art Ceiling by Brandon Mowinkel.

Jussi Nummelin

Read more posts by this author.

Tampere, Finland
comments powered by Disqus