Why is streaming interesting for my business
Streaming data has definitely been the talk of the town for the last few years. Like 99% of technology, streaming will simply add new abilities to your overall data story, it is not your data story in complete - quite the contrary. We can say this because CluedIn itself is a streaming pipeline but it is important to remember that streaming solves certain problems in the data space - let’s say that a it solves 5 to 10%.
These white papers are not for sharing definitions of what streaming is, a quick “google” will give you that answer. These white papers are for looking past all the marketing and getting to the core of the value of streaming and where it makes sense in your business.
There are many types of streaming platforms and most are solving similar problems but with variations. The core of data streaming can be simplified into one sentence. Instead of processing data at query time, let’s process it at write time. So it is obvious from this that we lose the ability to change our mind on what we want to aggregate when we do it at write time, but what we make up for is preparing an answer up front, so that when people query for this, it is already calculated. The downside for query time would be that running aggregations at query time would take a long time to run every time you wanted to run the query. Of course, with some smart caching etc. we can do a really good job at reusing calculations, however this doesn’t scale.