Control Your Data Flow with Ingest Budgets

We are pleased to announce the release of Ingest Budgets, a new feature which enables our users to track and control how much data is ingested into Sumo Logic and avoid overages in environments where data ingestion can spike unexpectedly. With Ingest Budgets, users can create budgets with ingestion thresholds that either cap data ingestion to a daily limit or simply alert whenever the threshold is exceeded.

Ingest Budgets delivers on one of the five most popular user-requested features. For many of our users, controlling ingest and collection across the entirety of an organization can be a challenge. Enterprise customers may be collecting logs and metrics from several internal development teams and environments, where both mission-critical production data as well as pre-production data is collected from dev and test instances. This pre-production data is valuable for baselining and benchmarking applications performance before release, but these pre-production instances can also be the biggest contributors to unexpected ingestion. Sometimes a team may run a performance test which causes a spike in ingestion. Other times, overages are caused by a bug that appears in a test environment, filling the logs with errors or stack traces for several days before the bug is noticed. Administrators want these teams to continue to collect data into Sumo Logic, but they also needed a mechanism to keep each team’s ingest within a set budget.

Ingest Budgets allows you to do just that. You can now create Ingest Budgets which limit collection to a daily threshold you specify. You can then assign a group of related collectors to a budget, where each collector consumes the budget as a shared pool ingest capacity. Once a budget is 85% consumed, events are logged to the Sumo Logic Audit Index where you may setup alerts triggered from a saved search. If you wish to enforce a cap on ingestion, you can configure the budget to run in drop_data mode. In this mode, collection will be stopped for each collector associated to the budget, once the budget capacity has been exceeded.

For some customers, stopping collection isn’t an option. They may be required to continue to ingest data no matter how large the overage is. For such cases, you may also configure the budget to use “keep collecting” mode when collecting mission-critical data that you wish to track overages but not stop the ingestion of data. That’s a great fit for users who’s compliance or security policies require that data always be collected, but still need to track and alert on budget overages

As an example, some of our pilot users of Ingest Budgets have assigned all collectors in their development environment to a single budget, separate from Test or Production environments. This allows them to track and control ingest for all development projects and instances. These same customers then also created a separate budget to track production ingestion, set to audit_only mode, so that they may record and alert on budget overages without stopping ingest of any of their mission critical data.

Other customers, typically larger enterprise users, have more complex organizational structures where collectors are grouped by team or squad. Each of these teams may analyse logs and metrics in Sumo Logic independently, so their Sumo administrators prefer to track and control ingestion at the team level. Administrators within such organizations can follow a budget per-team model, where all test, dev, and staging instances for a single are controlled by a single budget. This is an excellent way for administrators to both control and gain visibility into how each team is consuming the overall account quota.

Ingest Budgets works on a daily cycle, where each budget is reset every 24 hours at a time of your choice. When the scheduled restart occurs, all stopped collectors in the budget will be started and the total consumption of the budget is reset to zero. In addition to setting the time you wish a budget to restart, you may also specify the timezone for the start time. The allows you to align your budget controls to the same business cycles as the rest of your organization.

Users can also restart the budget on command, rather than waiting for the next scheduled restart. This is handy for situations where an admin has received an alert that a budget has been exceeded. The admin then investigates the budget overage and may discover that a particular app is the cause of the overage. The admin can then fix the problem in the application and reset the budget and resume collection on this budget.

We’re elated to launch Ingest Budgets and make this new level of control fully available to our Enterprise customers. We encourage you to set up a Ingest Budget today! More details and information about Ingest Budgets and instructions on how to set up a budget is available in our documentation.