Integrated Container Security Monitoring with Twistlock

Twistlock provides dev-to-production security for the container environment. More specifically, The Twistlock container security suite offers 4 major areas of functionality:

• Vulnerability management that inspects the full stack of components in a container image and allows you to eradicate vulnerabilities before deployment.
• Compliance which enforces compliance with industry best practices and configuration policies, with 90+ built-in settings covering the entire CIS Docker benchmark.
• Access control that applies granular policies to managing user access to Docker, Swarm, and Kubernetes APIs. This capability builds on Twistlock’s authorization plugin framework that’s been shipping as a part of Docker itself since 1.10.
• Runtime defense, which combines static analysis, machine learning, Twistlock Labs research, and active threat feeds to protect container environments at scale, without human intervention.

Integration with Sumo Logic

Because Twistlock has a rich set of data about the operations of a containerized environment, integrating with powerful operational analytics tools like Sumo Logic is a natural fit. In addition to storing all event data in its own database, Twistlock also writes events out via standard syslog messages so it’s easy to harvest and analyze using tools like Sumo Logic.

Setting up integration is easy, simply follow the standard steps for collecting logs from a Linux host that Sumo Logic has already automated. After a collector is installed on a host Twistlock is protecting, configure Sumo Logic to harvest the log files from /var/lib/twistlock/log/*.log:

In this case, the log collection is named “twistlock_logs” to make it easy to differentiate between standard Linux logs.

Note that Twistlock produces 2 main types of logs, aligned with our distributed architecture as illustrated below.

• Console logs track centralized activities such as rule management, configuration changes, and overall system health.
• Defender logs are produced on each node that Twistlock protects and are local in scope. These logs track activities such as authentication to the local node and runtime events that occur on the node.

Once log files are collected, searching, slicing, and visualizing data is done using the standard Sumo Logic query language and tools. Here’s a simple example of just looking across all Twistlock logs using thesource=”twistlock_logs” query:

Of course, the real power of a tool like Sumo Logic is being able to easily sort, filter, and drill down into log data. So, let’s assume you want to drill down a little further and look for process violations that Twistlock detected on a specific host. This is a common incident response scenario and this illustrates the power of Twistlock and Sumo Logic working together to identify the anomaly and to understand it more completely. To do this, we simply add a little more logic to the query:

(_sourceCategory=twistlock_logs (Process violation)) AND _sourcehost = “cto-stable-ubuntu.c.cto-sandbox.internal”

Perhaps you’re looking for a specific action that an attacker took, like running netcat, something that should likely never happen in your production containers. Again, because of Twistlock’s runtime defense, this anomaly is automatically detected as soon as it occurs without any human having to create a rule to do so. Because Twistlock understands the entrypoint on the image, how the container was launched via Docker APIs, and builds a predictive runtime model via machine learning, it can immediately identify the unexpected process activity. Once this data is in Sumo Logic, it’s easy to drill down even further and look for it:

(_sourceCategory=twistlock_logs (Process violation)) AND _sourcehost = “cto-stable-ubuntu.c.cto-sandbox.internal” AND nc

Of course, with Sumo Logic, you could also build much more sophisticated queries, for example, looking for any process violation that occurs on hosts called prod-* and is not caused by a common shell launching. Even more powerfully, you can correlate and visualize trends across multiple hosts. To take our example further, imagine we wanted to not just look for a specific process violation, but instead to visualize trends over time. The Twistlock dashboard provides good basic visualizations for this, but if you want to have full control of slicing and customizing the views, that’s where a tool like Sumo Logic really shines.

Here’s an example of us looking for process violations over time, grouped in 5 minute timeslices, and tracked per host, then overlaid on a line chart:

_sourceCategory=twistlock_logs (Process violation)| timeslice 5m | count as count by _timeslice, _sourceHost| transpose row _timeslice column _sourceHost

Of course, this just touches on some of the capabilities once Twistlock’s container security data is in a powerful tool like Sumo Logic. You may also build dashboards to summarize and visualize important queries, configure specific views of audit data to be available to only specific teams, and integrate container security event alerting into your overall security alert management process. Once the data is in, the possibilities are limitless.

Create a dashboard

Here we go over the steps of which to create a dashboard in Sumologic to show and analyze some of this data

• Login to Sumo Logic
• Create a new search
• Use the following query: (Replace twistlock/example with the tags you used when creating the Twistlock collector)
  • _sourceCategory=twistlock/example (violation) | timeslice 24h | count by _timeslice | order by _timeslice desc
  • Run the query and select the Aggregates tab
  • You should be looking at a list of dates and their total count of violations

• Select the single value viewer from the Aggregate Tab’s toolbar

• Click the “Add to dashboard” button on the right hand side to start creating a new dashboard by adding this chart as a panel
• Create the new panel
  • Enter a title for example: Violations (last 24 hours)
  • Enter a new dashboard name for example: Overview Dashboard

• Click Add

• As an optional step you can set coloring ranges for these values. This will help you quickly identify areas that need attention.
  • When editing the value choose Colors by Value Range… from the cog in the Aggregate Tab’s toolbar

• • Enter 1 – 30 and choose green for the color
  • Click Save
  • Enter 31-70 and choose orange for the color
  • Enter 71 – (leave blank) and choose red for the color
  • Click Save

• Create single value viewers using the same process as above for each of the queries below: (Replace twistlock/example with the tags you used when creating the Twistlock collector)
  1. Network Violations
     • _sourceCategory=twistlock/example (Network violation) | timeslice 24h | count by _timeslice | order by _timeslice desc
  2. Process Violations
     • _sourceCategory=twistlock/example (Network violation) | timeslice 24h | count by _timeslice | order by _timeslice desc
• Your dashboard should look similar to this

• Create another chart using the same process as above but this time use the search query: (Replace twistlock/example with the tags you used when creating the Twistlock collector)

• _sourceCategory=”twistlock/kevin” (violation) | timeslice 1d | count by _timeslice | order by _timeslice asc
• Run the query and select the Aggregates tab
• You should be looking at a list of dates and their total number of violations

• Select the area chartfrom the Aggregate Tab’s toolbar
• Click the “Add to dashboard” button on the right hand side to start creating a new dashboard by adding this chart as a panel
• Create the new dashboard panel
  • Enter a title for example: Violations by day
  • Select Overview Dashboard as the dashboard

• Click Add
• Resize the line chart so it extends the full width of the dashboard by clicking and dragging on the bottom right corner of the area chart panel
• Your dashboard should now look similar to the one below

• Use the following query: (Replace twistlock/example with the tags you used when creating the Twistlock collector)
  • _sourceCategory=”twistlock/example” (Denied)|parse “The command * * for user * by rule *’” as command, action, user, rulename | count by user | order by user asc
  • Run the query and select the Aggregates tab
  • You should be looking at a list of users and their total count of violations

• Select the column chart iconfrom the Aggregate Tab’s toolbar

• Click the “Add to dashboard” button on the right hand side to start creating a new dashboard by adding this chart as a panel
• Create the new panel
  • Enter a title for example: Top Users with Violations
  • Enter a new dashboard name for example: Overview Dashboard

• Click Add

• Create another chart using the same process as above but this time use the search query: (Replace twistlock/example with the tags you used when creating the Twistlock collector)

• _sourceCategory=”twistlock/example” (violation) | parse “.go:* * violation ” as linenumber, violation_type | count by violation_type | order by _count desc

• Create the new panel
  • Enter a title for example: Top Violation by Types
  • Select Overview Dashboard as the dashboard

• Click Add
• Your completed dashboard should now look similar to the one below

In summary, integrating Twistlock and Sumo Logic gives users powerful and automated security protection for containers and provides advanced analytic capabilities to fully understand and visualize that data in actionable ways. Because both products are built around open standards, integration is easy and users can begin reaping the benefits of this combined approach in minutes.