Best Practices for Creating Custom Logs - Part I

Overview

When logging information about your operating system, services, network, or anything else, usually there’s predefined log structures in place by the vendor. There are times when there aren’t predefined logs created by some software or you have custom application logs from your own software. Without properly planning your log syntax you’ll be using, things can get messy and your data may lose its integrity to properly tell the story. These best practices for creating custom logs can be applied to most logging solutions.

The 5 W's

There are 5 critical components of a good log structure*:

When did it happen (timestamp)
What happened (e.g., error codes, impact level, etc)
Where did it happen (e.g., hostnames, gateways, etc)
Who was involved (e.g., usernames)
Where he, she, or it came from (e.g., source IP)

Additionally, your custom logs should have a standard syntax that is easy to parse with distinct delimiters, a key-value pair, or a combination of both. An example of a good custom log is as follows:

2017-04-10 09:50:32 -0700 - dan12345 - 10.0.24.123 - GET - /checkout/flight/ - credit.payments.io - Success - 2 - 241.98

This log message shows when it was performed, what was performed, where it happened in your system, who performed it, and where that user came from. It’s also structured cleanly with a space-dash-space as the delimiters of each field. Optionally, you can also have key-value pairs to assist with parsing:

timestamp: 2017-04-10 09:50:32 -0700 - username: dan12345 - source_ip: 10.0.24.123 - method: GET - resource: /checkout/flight/ - gateway: credit.payments.io - audit: Success - flights_purchased: 2 - value: 241.98

Once you have your log syntax and what will be going into the logs, be sure to document this somewhere. You can document it by adding a comment at the top of each log file. Without documentation, you may forget or someone may not know what something like “2” or “241.98” represents (for this example, it means 2 flights in the checkout at a value of $241.98). You can document our log syntax as such:

Timestamp - username - user_ip - method - resource - gateway - audit - flights_purchased - value

In the second part of this three part series, we'll go into deeper details around timestamps and log content. In the final part, we'll go even deeper into log syntax and documentation.

*Source: Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management. Chuvakin, A., Phillips, C., & Schmidt, K. J. (2013).

Complete visibility for DevSecOps

Reduce downtime and move from reactive to proactive monitoring.

Start free trial

Dan Reichert

Dan Reichert is a Sales Engineer at Sumo Logic with over a decade of experience in technology in the US Army, IBM, and various startups. He is a graduate of the iSchool at Syracuse University with a masters degree in Information Management and University of Central Florida with a bachelors degree in Information Systems Technology. He is an AWS Certified Solutions Architect - Associate.

Best Practices for Creating Custom Logs - Part I

Overview

The 5 W's

Complete visibility for DevSecOps

Categories

Spotlight

Sumo Logic cloud-native SaaS analytics

Dan Reichert

Overview

The 5 W's

Complete visibility for DevSecOps

Categories

Spotlight

Share

Sumo Logic cloud-native SaaS analytics

Dan Reichert

Around the web

You're in good company