Frequently Asked Questions

We didn’t make the decision to implement a new query language lightly; in fact we’ve tried repeatedly to find ways to build on existing languages rather than implement our own. What we’ve found is that fundamental abstractions matter.

SQL really wants to reason about static tables. Using SQL as the query language forces a lossy conversion from the natural representation to tables. There’s a reason that Gremlin is popular for graphs and PromQL for timeseries. These query languages use fundamental abstractions that are aligned with the data being queried.

We believe that most appropriate abstraction for reasoning about event data is the timeline, and we built our query language around this idea.

We’ve tried very hard to implement a simple, familiar query syntax with intuitive, predicable semantics, and to provide extensive documentation and example code. We’re always looking for ways to improve - if you see things that are confusing or unexpected we’d love to hear about it. Start a GitHub discussion and let’s improve Kaskada together!

Event processors must choose between optimizing for throughput or latency, but the tradeoff is asymmetric - improving p90 end-to-end latency by a minute can require design choices that reduce throughput by multiple orders of magnitude.

Modern CPU’s and GPU’s are much more efficient when applying the same operation repeatedly, so compute efficiency increases as the batches you compute over get larger. In a real-time system, however, working with larger batches means spending more time buffering, which means increased end-to-end latency.

One of Kaskada’s design goals is to provide interactive results over large historical event sets. This ability is critical to understanding the historical context of a computation or "bootstrapping" new real-time computations.

To support these type of bulk/OLAP workloads, Kaskada is built using the Apache Arrow library for columnar computation. Columnar compute is difficult to "tack on" to an event processor after-the-fact; it’s a foundational design decision that has implications throughout the engine’s implementation.

By building on Arrow, Kaskada optimizes for throughput over latency. We believe that end-to-end latencies measured in 100’s of milliseconds are sufficient for most real-time applications, and that sacrificing single-digit latencies is an acceptable tradeoff for orders-of-magnitude improvements in bulk efficiency.

Another important tradeoff Kaskada makes is to focus on single-process performance. In the early days of "Big Data", commodity hardware came with a single core and 2Gb of RAM. The only way to implement a computation over terabyte-sized datasets was to distribute it across multiple machines.

Recent compute engines are increasingly de-emphasizing distributed execution in favor of high-performance on a single physical instance. To quote from an excellent article from DuckDB's Jordan Tigani titled Big Data is Dead:

Currently, Kaskada takes a hybrid approach to distributed execution. Distributed workers can be used when initially loading data. Data loading is expensive becuse it involves sorting events chronologically. Kaskada currently executes each query in a single process. Fully-distributed execution is on our roadmap, however we find that "vertical scaling" is sufficient for the vast majority of use cases.

Timeseries databases are a popular way to work with temporal data. A timeseries captures a series of values, each associated with a different time. In most cases, the series corresponds to a standard interval, for example seconds or minutes. Having a pre-defined series is useful for some operations, for example it is easier to reason about time intervals in which nothing happened than it would be with standard SQL.

The downside to starting with a standard interval is that in some cases your source data doesn’t conform to the timeseries format - timeseries are often generated by counting event occurrences in each time interval. Information is often lost in the transformation from instantaneous events into windowed aggregations.

Timelines are similar to timeseries - both capture values associated with different times. The difference is that a timeline describes an arbitrary number of values and doesn’t depend on a standard interval. In this sense, a timeseries is a special-case of a timeline.

Kaskada provides operations for transforming a timeline into a time series. For example, to transform an event timeline Purchase into a daily event-count timeseries:

An "event" is any fact about the world associated with a specific time. Events record simple observations, for example:

Events are powerful because they’re facts; they don’t change over time. Alice may cancel her purchase, or return the tennis ball for a refund, but this doesn’t change the fact of her original purchase.

Events are produced in many different ways:

By collecting events as they occur, applications can react to changes in the world as they happen.

Events are often treated as "raw" data in need of pre-processing before they can be used. You might be accustomed to working with data that’s been through a number of ETL operations to noramlize, filter, and aggreagte the raw events into an easier to use set of tables. This type of data processing can help ensure consistent business logic and minimize the time required to start working with "correct" data.

Unfortunately, this practice can have some downsides:

Kaskada is designed to allow practitioners to describe the full set of cleaning operations needed to transform raw events into actionable data. Collaboration through code-sharing makes it easier to understand how outputs are defined, and makes it easier to iterate on those definitions.