Security response center

May 23, 2024

On May 20, 2024, Tenable Research discovered a critical memory corruption vulnerability dubbed Linguistic Lumberjack in Fluent Bit (CVE-2024-4323), a core component in the monitoring infrastructure of many cloud services. Sumo Logic has updated all applicable systems to ensure we are not vulnerable and is continuing to monitor our corporate security posture as well as our third-party vendors to ensure they are dealing with the situation as appropriate.

From an open source perspective, although by default none of our solutions are exposed to the internet, some do leverage the impacted versions of Fluent Bit. We have released updated versions in those instances. See below for details.

• Tailing sidecar operator version 0.13.0 contains the fix.

• Kubernetes Collection Helm Chart v4.7.1 contains the above fix.

• Kubernetes Collection Helm Chart v3.19.3 contains the above fix and a separate upgrade to Fluent Bit.

• No upgrades available for Kubernetes Collection Helm Chart version 2. Support for this version ended on 2023-07-20. Customers are encouraged to upgrade to v4.7.1.

• No upgrades available for Helm Operator. An upgrade to version 4 of the Helm Chart (where there’s no Fluent Bit) is in progress.

We will continue to monitor and update as appropriate.

November 20, 2023 - 3:30 PM PST

Update: Sumo Logic Security Incident

To our valued customers:

We want to provide you with an update on Sumo Logic’s recent security incident. We take the safety and reliability of our platform seriously. This is why we took immediate action to secure our platform as soon as we detected a potential security incident, including the recommendation to rotate all credentials.

We are grateful to share that the diligent investigation led by our security and engineering teams uncovered no proof of customer data impact and no threat of customer data impact present. These findings were verified by third-party forensic experts and the investigation of this incident is now complete and closed.

We remain committed to providing all of our customers with secure and reliable digital experience and are doing everything we can to emerge safer from this incident. To that end, we will be undertaking additional evaluation to learn from this incident and identify any measures or modifications to prevent future incidents.

As we have done throughout this process, we encourage you to keep an eye on the Security Response Center where we have posted pertinent updates. We also would like to provide you with additional tools and information.

FURTHER STEPS YOU CAN TAKE

While this investigation is complete and closed, and there is no proof of customer data impact and no threat of customer data impact present, Sumo Logic is providing the Indicators of Compromise (IOCs) and supporting documentation so customers can inspect their own environments, which we recommend.

Indicators of Compromise

34.201.113.45

66.225.222.68

159.223.118.253

54.183.5.235

34.207.95.146

44.203.0.45

45.154.98.33

18.189.57.10

185.220.101.56

34.238.239.207

185.220.101.58

45.76.10.28

How to Inspect Your Own Environments

As we formally close this investigation, we want to share some of the techniques we used internally, leveraging the Sumo Logic platform to further guide you in further inspecting your environments and ensuring your security posture.

We used our own solutions within Sumo Logic. In this particular instance, we used both our Cloud SIEM to search our environment for Indicators of Compromise (IOCs) and malicious IPs. In addition, we also used our Cloud Infrastructure Security (CIS) solution for enterprise audit activities such as access key creation, deletion and active vs. inactive. Below is a sample query you can run within your Sumo Logic instance. This will return any source categories where the IOCs are present and will show you where to do further investigation.

Example Search in Sumo Logic

_sourcecategory=* ("34.201.113.45" or "66.225.222.68" or "159.223.118.253" or "54.183.5.235" or "34.207.95.146" or "44.203.0.45" or "45.154.98.33" or "18.189.57.10" or "185.220.101.56" or "34.238.239.207" or "185.220.101.58" or "45.76.10.28")
| timeslice 1m
| parse regex "(?<ioc_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" multi
| where ioc_ip in ("34.201.113.45","66.225.222.68","159.223.118.253","54.183.5.235", "34.207.95.146", "44.203.0.45", "45.154.98.33", "18.189.57.10", "185.220.101.56", "34.238.239.207", "185.220.101.58", "45.76.10.28")
| count _timeslice, _sourcecategory, ioc_ip

If you have questions about this guidance, don't hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

Thank you for your patience and understanding throughout this process. We look forward to continuing to help our customers turn insights into action and deliver reliable and secure digital experiences.

November 14, 2023 - 6:00PM PST

We continue to make progress with the investigation. We are still strongly advising customers to rotate the Sumo Logic API access credentials as soon as possible, and no later than 48 hours. Specifically, we advise you to rotate immediately:

• Sumo Logic API access keys

What you could also rotate as an additional precautionary measure:

• Third-party credentials that have been stored with Sumo as part of webhook connection configuration

If you have questions about steps to take, don't hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

November 12, 2023 - 3:30PM PST

We continue to make progress with the investigation. However, the substantial guidance we have as of today remains the same as provided on November 10th.

———

If you have questions about steps to take, don't hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

November 11, 2023 - 3:30PM PST

We continue to make progress with the investigation. However, the substantial guidance we have as of today remains the same as provided on November 10th.

———

If you have questions about steps to take, don't hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

November 10, 2023 - 4:00PM PST

WHAT SHOULD YOU DO (if you have not done so yet):
We recommend that customers rotate credentials that are either used to access Sumo Logic or that you have provided to Sumo Logic to access other systems.

Specifically, we advise you to rotate immediately:

• Sumo Logic API access keys

What you could also rotate as an additional precautionary measure:

• Third-party credentials that have been stored with Sumo as part of webhook connection configuration

———

WE NO LONGER RECOMMEND THE FOLLOWING MEASURES FOR THIS INCIDENT:

• Sumo Logic installed collector credentials
• Third-party credentials that have been stored with Sumo for the purpose of data collection by the hosted collector (e.g., credentials for S3 access)
• User passwords to Sumo Logic accounts

———

If you have questions about steps to take, don't hesitate to get in touch with our customer support team at https://support.sumologic.com/support/s/

November 9, 2023 - 2:00PM PST

Per the recommendations mentioned in the November 7th & 8th Security Notices, we have created a playbook to guide our customers through the process of updating their API access keys.

Here is a direct link to the playbook: link

The information contained in this website is provided “as is,” without any warranty of any kind, either express or implied. Users are solely responsible for adequate protection and backup of the data and equipment used in connection herewith. If Users require assistance, please contact our Support team via opening a ticket using our Support Console.

November 8, 2023

As an outcome of our ongoing investigation, we are reducing the scope of the additional precautionary measures mentioned in our November 7th message. Here is the updated recommendation:

What you could also rotate as an additional precautionary measure:

• Third-party credentials that have been stored with Sumo as part of webhook connection configuration

November 7, 2023

To Our Valued Customers:

At Sumo Logic, ensuring the security and reliability of our customers’ digital experience is our top priority. We have always placed great emphasis on protecting our customers against threats, and we understand and deeply value the trust our customers place in us.

To that end, we are writing to notify you, as a precautionary measure, of a possible security incident within our platform.

WHAT HAPPENED:

On Friday, November 3rd, 2023, Sumo Logic discovered evidence of a potential security incident. The activity identified used a compromised credential to access a Sumo Logic AWS account. We have not at this time discovered any impacts to our networks or systems, and customer data has been and remains encrypted.

WHAT HAVE WE DONE:

Immediately upon detection we locked down the exposed infrastructure and rotated every potentially exposed credential for our infrastructure out of an abundance of caution. We are continuing to thoroughly investigate the origin and extent of this incident. We have identified the potentially exposed credentials and have added extra security measures to further protect our systems. This includes improved monitoring and fixing any possible gaps to prevent any similar events and we are continuing to monitor our logs to look for further signs of malicious activity. We have taken actions to stop the threat to our infrastructure and are advising customers to rotate their credentials.

WHAT SHOULD YOU DO:

We recommend that customers rotate credentials that are either used to access Sumo Logic or that you have provided to Sumo Logic to access other systems. Specifically:

What we advise you rotate immediately:

• Sumo Logic API access keys (If you need assistance with this, please contact Sumo Support at https://support.sumologic.com/support/s/)

What you could also rotate as an additional precautionary measure:

• Sumo Logic installed collector credentials

• Third-party credentials that have been stored with Sumo for the purpose of data collection by the hosted collector (e.g., credentials for S3 access)

• Third-party credentials that have been stored with Sumo as part of webhook connection configuration

• User passwords to Sumo Logic accounts

If you have questions about steps to take, please do not hesitate to contact our customer support team at https://support.sumologic.com/support/s/

WHAT HAPPENS NEXT:

While the investigation into this incident is ongoing, we remain committed to doing everything we can to promote a safe and secure digital experience.

We will directly notify customers if evidence of malicious access to their Sumo Logic accounts is found. Customers may find updates at our Security Response Center.

Your security remains our top priority and we want to reiterate how much we value you putting your trust in us. Thank you for your understanding through this process.

October 26, 2023
Last month Google published CVE-2023-5129 (marked as a duplicate) and CVE-2023-4863 indicating vulnerabilities existed within libwebp and as a result within a number of downstream applications leveraging it. Sumo Logic has updated all applicable systems to ensure we are not vulnerable and are continuing to monitor our corporate security posture as well as our third-party vendors to ensure they are dealing with the situation as appropriate.

October 26, 2023
Sumo Logic is aware of the vulnerability CVE-2023-44487, also known as "HTTP/2 Rapid Reset Attack.” Sumo Logic has mitigations in place that we inherit from AWS mechanisms. We do not believe we are susceptible based on our scanning, testing, and inherited mitigations from our AWS-based infrastructure. We have followed the current guidance published by AWS in this regard.

September 28, 2023
Sumo Logic is aware of the vulnerabilities (CVE-2023-5129 and CVE-2023-4863) that Google published indicating a critical vulnerability in libwebp. We are actively investigating to identify any and all areas where we may be leveraging the vulnerable versions of this library. As of now our investigations have revealed no indications of compromise. We will keep this page up to date as our investigations continue.

June 20, 2023
Sumo Logic is aware of the published vulnerabilities (CVE-2023-34362, CVE-2023-35036, and CVE-2023-35708) by Progress software indicating that the MOVEit Transfer tool was vulnerable to multiple SQL injection vulnerabilities. Our investigation suggests that Sumo Logic is not impacted as our products and services do not use MOVEit Transfer tool. Our current assessment is that no action is required from our customers, but we will continue to monitor and update as appropriate.

Nov 4, 2022
Sumo Logic is aware of the recently announced OpenSSL vulnerabilities (CVE-2022-3786 and CVE-2022-3602). Our initial investigation suggests that Sumo Logic is not impacted as our products and services do not rely on OpenSSL. Our current assessment is that no action is required from our customers, but we will continue to monitor and update as appropriate.

April 1, 2022
Sumo Logic has validated that we do not use any part of the vulnerable Spring Cloud framework found in CVE-2022-22963. We have also reviewed CVE-2022-22965 and have validated that Sumo Logic is not vulnerable to known exploitable methods. Out of an abundance of caution, we will be updating our Sumo Logic Service; however, no action is required on your part. The Sumo Logic collector is not vulnerable to known Spring Cloud framework exploitation methods.

March 31, 2022
Sumo Logic is aware of CVE-2022-22963, which is an exploitable Remote Code Execution (RCE) in Spring Cloud Functions. Our initial investigation suggests that Sumo Logic is not impacted. Our current assessment is that no action is required on your part at this time, but we will continue to monitor and keep you posted as our analysis progresses.

Our Global Operations Center investigated Okta’s evolving situation as Sumo Logic currently leverages Okta's federated identity solution. So far we have no evidence that Sumo Logic, our employees or services are impacted in any way.

Beginning early in the morning on Dec. 10th, Sumo Logic’s security team investigated and validated the nature and severity of the exploit against potential points of compromise and determined that at NO time was Sumo Logic exploited.

We use a custom SumoLog4Layout library that never invokes custom lookups (as compared to Apache Log4j) so the Sumo Logic Service was never impacted.