I" class="redactor-autoparser-object">https://www.sumologic.com/blog... bet that November 9, 2016, will be one of those days in the future when kids will ask their parents, “Where were you when you found out that a former reality TV show host won the U.S. Presidency?” Of course, when I get that question, — it’s hard to tell at this point if it will be sooner than later – I’ll say, “I was standing in front of a thousand people in Lisbon, Portugal, telling them about the marvels of machine data.”
I hope any resulting confusion I encounter from my response will not mean that machine data continued it’s identity crisis as an underground phenomenon unknown to most. Machine data is my life’s mission, and my goal is to bring it out of the unknown and into the masses, so it receives the respect it deserves. Hence, I was proud to have the opportunity to speak at Web Summit 2016 this past week in front of more than a thousand data lovers, like me, to declare that, “Machine Data’s time has come — Machine Data is Ready for the Masses!”
So gather around, and let me explain why.
When I co-founded Sumo Logic in 2010, we had a strong sense coming out of ArcSight that machine data was undervalued. We saw it’s role and relevance transcending from security analytics — a very useful context for sure — to a myriad of IT use cases, especially with the emergence of new business models that depend on modern software applications. Modern software applications are today’s platform for new services and offerings, not even imagined 20 years ago (think Uber, Twitter, AWS, AirBnB, to name a few). They are the driving force behind the shift from disparate, product-based economies to a global, interconnected, services-based economy that I feel excited to be part of.
Granted, software has been around since the inception of the computer, but what makes modern applications so different is that the role of software itself in modern business has shifted from functional enabler (ERP anyone?) to the platform that runs the business. Sumo Logic itself, is a classic example: we offer a machine data analytics service that processes close to 100 TB of data per day, with millions of queries scanning multiple petabytes per day. Unlike ERP packaged software or single instance cloud offerings in which the customer sits outside the application managing it’s needs, all of our 1200+ customers are inside our application platform, consuming the analytics and insights — without the management headaches — to better run their modern applications, and hence their businesses. Pretty amazing stuff.
Furthermore, I believe the increasing reliance on software through the growth of modern applications is substantially speeding innovation in general. As DevOps practices become more prevalent, providing innovation improvements hourly and daily, our customers now feel the pressure to survive (and thrive) in a sub-microsecond world. These new competitive conditions are also creating a rate of change that is exponential. What is my takeaway from this? Optimization is no longer linear — now it’s a loop that companies can never jump off of for fear of being left behind or disappearing altogether (like dinosaurs). For me, this brings to mind two interesting paradoxes that mirror the classic “chicken-egg” conundrum:
- “Nach dev Spiel its simmer for dev Spiel.” That’s German for “after the game is before the game” as expressed by Sepp Herberger, a German titan, football team trainer, and maybe the greatest German ever. In his own way, he was describing an optimization philosophy that he employed to underpin a winning culture based on the idea that preparation for the next game never stops. Not even for a moment. I find this analogous to our business world today in which the need to make decisions never stops — you make one, and then the next one needs to be made, and then the next, and so on. So, in other words, we need to find a way to make decisions continuously, which also drives the need for a continuous Intelligence that can support continuous decision-making.
- “Processing data creates exponentially more data.” That’s from me, and it means that the more data we process in the pursuit of human decision-making, the more derivative data it generates from the original data processed. So, we have another nice, strange loop here. That derivative data is machine data, and I’d wager more machine data is created than is being processed in the first place. But, this insight-rich data source – the exact record of your modern application’s and system’s events, metrics and behaviors – is mostly written-off as waste. It is overwritten, rolled off, or disappeared in the giant trash compactor in the matrix. I suspect that’s because making sense of it in a meaningful way for business has been non-existent given machine data’s nature: it’s poly-structured — not “unstructured” — as is commonly said. Poly-structured data has structure, but there are many different structures. Sometimes they are even nested. And the data comes without instructions. It just is. There are no schemas, and thus no easy way to dig into its value in a comprehensive and expedient way – until now.
odern alchemy business. We try to turn that #!*$ storm coming out the other end of you infrastructure into something useful for our customers. Because, machine data is not just about keeping modern applications running so the business stays open. It’s about real-time insights – golden nuggets if you will — to enable business success.
For example: because machine data essentially represents an exact record of every event, action, log or time-series metric of a modern application, collectively, it can operate as an information/insights platform that spans every function of your business. Technology platforms, like Sumo Logic, serve up this information in the form of real-time analytics viewable via graphical dashboards that can easily be made available to stakeholders across functions. Imagine that — every function, sharing the same data platform, monitoring events and KPIs relevant to their function in real-time. Sounds hard to believe. But we’re closer than you think.
The companies today that understand the power of machine data to manage their modern application environments are ahead of the game. That’s what we see now with our customers. But, eventually, all businesses will get there and become software-based sooner or later. Once there, they need to figure out continuously how that software is doing, in order to know how the business is doing. In other words, software for the masses will create machine data for the masses – and I suspect that machine data will finally get the respect it deserves. In the meantime, I happily carry the mantle.
Agree or disagree, let me know. Reply to this blog, or hit me up on @raychaser on Twitter
