Accelerate Data Analytics with Sumo Logic’s Logs-to-Metrics Solution
https://www.sumologic.com/blog...">
If you’re building a new application from scratch and are responsible for maintaining its availability and performance, you might wonder whether you should be monitoring logs or metrics. For us, it’s a no-brainer that you’ll want both: metrics are fast and efficient for proactively monitoring the health of your system, while logs are essential for helping to troubleshoot the details of the issue itself to find the root cause.
To use a real world analogy, let’s say you go in for an annual check up and the doctor sees you have elevated blood pressure (“the metric”). He then asks you enough questions to discover that you’ve been eating fast food five nights a week (“the logs”), and recommends a diet change to normalize your blood pressure levels (“the fix”).
But what if you’re working with an existing application where logs have always been used for monitoring? Or you’re leveraging third-party services that are only sending you logs? These logs may often contain key performance indicators (KPIs) like latency, bytes sent and request time, and Sumo Logic is great for structuring this in a way to create dashboards and alerts. However, to get the performance benefits of metrics, you might consider re-instrumenting your application to output those KPIs as native metrics instead of logs. But we all know how much free time you have to do that.
Extract Metrics from Logs for High Performance Analytics
Still, you may be wondering: why would I spend time converting all of my log data to metrics? The long and short of it is this: to deliver the best customer experience to your users. And machine data analytics is essential for that. However, according to data we recently released, one of the biggest barriers to adopting a data analytics tool is the lack of real-time analytics to inform operational, security and business decisions. Without it, you’ll suffer from slow analytics and will lose customers in minutes. No one wants that, especially when customers are relying on your tools to help them resolve critical issues.
Sumo Logic’s Logs-to-Metrics solution is the answer to that challenge because we make it easy for you to turn logs into metrics that can be then used as valuable KPIs. And since we do the heavy lifting and work with you to create metrics from existing logs, you don’t have to worry about creating them from scratch.
Whether your KPIs are embedded in the logs themselves (e.g., latency, request_time) or you’re looking to compute KPIs by counting the logs (e.g., error count, request count), we’ve got you covered. Turning some of your logs into metrics will give you several key benefits:
High Performance Analytics: Storing data in a time-series database allows for lightning fast query times, since the data is optimized for speed and efficiency.
Thirteen-Month Data Retention: For all metrics, Sumo Logic provides 13-month retention by default, enabling quick long-term trending of critical business and operational KPIs.
Flexible and Low Latency Alerting: With metrics, you can leverage Sumo Logic’s real-time metrics alerting engine, which includes intuitive UI configuration, multiple threshold settings, missing data alerts, muting and more.
Never Re-Instrument Code Again: Gain all of the benefits of metrics without digging into your code to configure a metrics output.
Easy Configuration with Real-Time Validation
In order to make this metrics extraction as seamless as possible, we’ve created a fast way for you to validate your rules in real-time. There are three simple steps to pick out your metrics:
Specify a Scope: This is the set of logs that contain the metrics you are interested in. Typically, this contains one or more pieces of metadata and some keywords to narrow down the stream of logs. For example, “_sourceCategory=prod/checkout ERROR”.
Define a Parse Expression: Use Sumo Logic’s parsing language to extract out the important fields you’ll want to turn into metrics. You can even use regular expressions for more complex log lines.
Select Metrics and Dimensions: After successfully parsing your logs, select which fields are metrics and which are dimensions. Metrics will be the actual value you are interested in tracking, while dimensions are the groups you would want to aggregate those values by. For example, if you want to track the number of errors by service and region, “errors” would be a metric while “service” and “region” would be dimensions.
In real-time, Sumo Logic will show you a preview of your parse expression to make sure you’ve correctly extracted the right fields. You can also extract multiple metrics and dimensions from a single rule.
KPIs as Metrics = 100x Performance over Logs
As much as we love the performance of our log analytics at Sumo Logic, we really love the performance of our metrics. Transforming thousands (or millions) of unstructured log messages into structured visualizations on the fly is always possible, but when the data can be stored as a metric in our native time-series database, the resulting query performance can be astounding. In the simple comparison below, it’s pretty easy to see which chart belongs to metrics:
Low Latency Monitoring and Highly Flexible Alerting
After extracting metrics out of your logs, you can also take advantage of Sumo Logic’s real-time alerting engine, which monitors your metrics in real-time and triggers notifications within seconds of a condition being met. In additional to the low latency, some other benefits include:
Multiple Thresholds: Create different alerts based on the severity of the metric. For example, create a warning alert if CPU is above 60 for five minutes, but generate a critical alert if it’s ever above 90.
Multiple Notification Destinations: Send your alerts to multiple destinations. For example, create a PagerDuty incident and send an email when the monitor is critical, but just send a Slack message if it’s hit the warning threshold.
Missing Data: Get notified when data hasn’t been seen by Sumo Logic, which can be a symptom of misconfiguration or a deeper operational issue.
The Bigger Picture
Unstructured machine data is not always optimized for the kind of real-time analytics customers need to inform business decisions.
With this new release, users can now take advantage of Sumo Logic’s metrics capabilities without re-instrumenting their code by leveraging existing logs for more efficient analytics and insights.
In addition to the deep forensics and continuous intelligence provided by logs, customers can take advantage of metrics by easily extracting key performance indicators from unstructured logs, while still retaining those logs for root cause analysis.
These metrics can then be used with the Sumo Logic time series engine, providing 10 to 100 times the analytics performance improvements over unstructured log data searches, as well as support long-term trending of metrics. This allows them to move fast and continue to deliver a seamless experience for their end users.
Additional Resources
Read the press release on our latest product enhancements unveiled at DockerCon
Download the report by 451 Research & Sumo Logic to learn how machine data analytics helps organizations gain an advantage in the analytics economy
