Integrate Azure Functions with Sumo Logic Schedule Search

Azure Functions

Azure functions are event driven pieces of code that can used to integrate systems, build APIs, process data and trigger any action/reaction to events without having to worry about infrastructure to run it.

Sumo Logic Scheduled Search

Scheduled searches are standard saved searches that are executed on a schedule you set. Once configured, scheduled searches run continuously, making them a great tool for continuously monitoring your stack, infrastructure, process, build environment, etc.

Why integrate Sumo Logic Scheduled Search with Azure Function?

Answer is very simple: Using Sumo Logic’s machine learning algorithm and search capabilities, you can Monitor and alert on key metrics and KPIs in real time to rapidly identify problems, detect outliers, abnormal behavior using dynamic thresholds or any other event which is important for you. Once you have detected the event for your use case, you can have the Azure function respond to your event and take an appropriate action.

More info on real time monitoring using Sumo Logic can be found here.

Three Steps guide to Integrate Azure Function with Sumo Logic Schedule Search.

Case in point: Web app scheduled search detects an outage -> Sumo Logic triggers Azure Function via Webhook Connection –> Azure function gets executed, and takes preventive/corrective action.

Step 1: Create Azure Function and write the preventive/corrective action you want to take.

Step 2: Set up Sumo Logic Webhook Connection which will trigger Azure Function created in #1. To set up connection, follow the steps under ‘Setting up Webhook Connections’

Step 3: Create a Schedule Search that will monitor your infrastructure for any outage, call the Webhook connection created in #2.

Example

Sumo Logic with it’s machine learning capabilities can detect an outlier in incoming traffic. Given a series of time-stamped numerical values, using the Sumo Logic’s Outlier operator in a query can identify values in a sequence that seem unexpected, and would identify an alert or violation, for example, for a scheduled search. To do this, the Outlier operator tracks the moving average and standard deviation of the value, and detects or alerts when the difference between the value exceeds mean by some multiple of standard deviation, for example, 3 standard deviation.

In this example, we want to trigger an Azure Function whenever there is an outlier in incoming traffic for Azure Web Apps.

Step 1: Create an Azure function – for this example I have following Node.js function

#r "Newtonsoft.Json"
using System;
using System.Net;
using Newtonsoft.Json;

public static async Task Run(HttpRequestMessage req, TraceWriter log)
{
 log.Info($"Webhook was triggered Version 2.0!");
 string jsonContent = await req.Content.ReadAsStringAsync();
 dynamic data = JsonConvert.DeserializeObject(jsonContent);
 log.Info($"Webhook was triggered - TEXT: {data.text}!");
 log.Info($"Webhook was triggered - RAW : {data.raw} !");
 log.Info($"Webhook was triggered - NUM : {data.num} !");
 log.Info($"Webhook was triggered - AGG : {data.agg}!");
 /* Add More Logic to handle an outage */
 return req.CreateResponse(HttpStatusCode.OK, new {
 greeting = $"Hello"
 });
}

Copy and paste Function Url in a separate notepad, you will need this in Step 2

Step 2: Create Sumo Logic Webhook Connection.

From your Sumo Logic account: Go to Manage -> Connections, Click Add and then click Webhook.

1. 1. 1. 1. Provide appropriate name and Description.
         2. Copy paste the Azure Function Url (from step #1) in URL field.
         3. For Payload, add following JSON.
         4. Test connection, and Save it.

{ 
 "text": "$SearchName ran over $TimeRange at $FireTime", 
 "raw": "$RawResultsJson",
 "num": "$NumRawResults",
 "agg": "$AggregateResultsJson" 
}

Step 3: Create a Schedule Search.

Scheduled searches are saved searches that run automatically at specified intervals. When a scheduled search is configured to send an alert, it can be sent to another tool via a Webhook Connection.

From your Sumo Logic account, copy paste following search and click Save As

_sourceCategory=Azure/webapp
| parse regex "\d+-\d+-\d+ \d+:\d+:\d+ (?<s_sitename>\S+) (?<cs_method>\S+) (?<cs_uri_stem>\S+) (?<cs_uri_query>\S+) (?<src_port>\S+) (?<src_user>\S+) (?<client_ip>\S+) (?<cs_user_agent>\S+) (?<cs_cookie>\S+) (?<cs_referrer>\S+) (?<cs_host>\S+) (?<sc_status>\S+) (?<sc_substatus>\S+) (?<sc_win32_status>\S+) (?<sc_bytes>\S+) (?<cs_bytes>\S+) (?<time_taken>\S+)"
| timeslice 5m
| count by _timeslice
| outlier _count
| where _count_violation=1 

Note: This assumes you have _sourceCategory set up with Azure/webapp. If you don’t have this source set up, then you can use your own search to schedule it.

• In the Save Search As dialog, enter a name for the search and an optional description.
• Click Schedule this search.
• Choose 60 Minutes for Run Frequency
• Last 60 Minutes for Time range for scheduled search
• For Alert condition, choose Send notification only if the condition below is satisfied
• Number of results Greater than > 0
• For Alert Type choose Webhook
• For Webhook, choose the Webhook connection you created in Step 2 from dropdown.
• Click Save

Depending upon Run Frequency of your scheduled search, you can check the logs of your Azure function from portal to confirm it got triggered.

2016-08-25T20:50:36.349 Webhook was triggered Version 2.0!
2016-08-25T20:50:36.349 Webhook was triggered - TEXT: Malicious Client ran over 2016-08-25 19:45:00 UTC - 2016-08-25 20:45:00 UTC at 2016-08-25 20:45:00 UTC!
2016-08-25T20:50:36.349 Webhook was triggered - RAW : !
2016-08-25T20:50:36.349 Webhook was triggered - NUM : 90 !
2016-08-25T20:50:36.351 Webhook was triggered - AGG : [{"Approxcount":13,"client_ip":"60.4.192.44"},{"Approxcount":9,"client_ip":"125.34.187"},{"Approxcount":6,"client_ip":"62.64.0.1"},{"Approxcount":6,"client_ip":"125.34.14"}]!
2016-08-25T20:50:36.351 Function completed (Success, Id=72f78e55-7d12-49a9-aa94-8bb347f72672)
2016-08-25T20:52:25 No new trace in the past 1 min(s).
2016-08-25T20:52:49.248 Function started (Id=d22f92cf-0cf7-4ab2-ad0e-fa2f23e25e09)
2016-08-25T20:52:49.248 Webhook was triggered Version 2.0!
2016-08-25T20:52:49.248 Webhook was triggered - TEXT: Errors Last Hour ran over 2016-08-25 19:45:00 UTC - 2016-08-25 20:45:00 UTC at 2016-08-25 20:45:00 UTC!
2016-08-25T20:52:49.248 Webhook was triggered - RAW : !
2016-08-25T20:52:49.248 Webhook was triggered - NUM : 90 !
2016-08-25T20:52:49.248 Webhook was triggered - AGG : [{"server_errors":39.0}]!
2016-08-25T20:52:49.248 Function completed (Success, Id=d22f92cf-0cf7-4ab2-ad0e-fa2f23e25e09)

Summary

We created a scheduled search which runs every 60 minutes, to find an outlier in last 60 minutes of incoming traffic data. If there is an outlier, webhook connection gets activated and triggers Azure function.

More Reading

Building Great Alerts

• Actionable – there should be a playbook for every alert received
• Directed – there should be an owner to follow the playbook
• Dynamic – Static thresholds can have “false positives”