Time Series DB's: Clickhouse at Sentry (Ted Kaemming, James Cunningham)

<p>Listen to the Data Eng podcast: <a href="https://www.dataengineeringpodcast.com/snuba-event-data-warehouse-episode-108/">https://www.dataengineeringpodcast.com/snuba-event-data-warehouse-episode-108/</a> (11mins in)</p><ul><li><a href="https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/">https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/</a></li></ul><p><br><strong>Transcript</strong></p><p>James Cunningham</p><p><br></p><p>Yeah, so I'd say as far as all the decisions that we made in order to go into this new platform, one of the biggest leaders was that we had a big push for having environments be kind of like a first class filtration, we had to build a new dimensionality of data across all this denormalized data, essentially doubled the storage that we had. And then we said to ourselves, like all this is great, this looks cool. environments are dope. But what happens we want to add another dimension and have dimension or we're just going to continue to, I guess, like, extrapolate across this data set and eventually end up with 100 terabytes of, you know, five different dimensions of data. So we said ourselves That we kind of needed a flat event model that we'd be able to kind of search across and to ourselves, you know, there are a few other pieces that we want. And on top of that, we want to be able to search across these arbitrary fields that we really, really looked into whether those are custom tags or something that we kind of promote, whether that is like releases or traces or searching across messages. We didn't want that to take as long as it did. And some of the other parts is that we have all this data stored in, you know, this tag store and all these searches that we have to go through. But we have in a completely different side for time series data that again, had to have that dimensionality in it. If we search across these arbitrary fields, the next thing that a customer would ask for is, Hey, can I please see a pr