✳flâneur — a map of the web's best reading

Incremental Database Computations

feldera.com · 1,609 words · saved by 1 readers

In this blog post we explain how an incremental query engine like Feldera actually operates as seen by a user. We start by motivating the need for incremental computation in databases. Future blog posts will describe the internal operations in detail. Many data analysis systems need to periodically execute the same computation on changing datasets. Several examples are: We have ordered these examples in order of increasing frequencies. In all these cases we have a database that stores the data, which keeps changing, and a set of queries that are evaluated periodically to produce results reflecting the latest changes. The following diagram illustrates this process: The horizontal axis shows the evolution of time. The silver cylinder is a database that keeps changing. Let's assume that the database is very large (1 billion rows), while each of the changes only modifies a small part of the database (100 rows, across one or multiple tables); we use the Greek letter "delta" (Δ) to indicate

In previous blog posts we have: informally introduced incremental computation , introduced Z-sets as a representation for database tables and changes to database tables In this blog post we explain how an incremental query engine like Feldera actually operates as seen by a user. We start by motivating the need for incremental computation in databases. Future blog posts will describe the internal operations in detail. Periodic queries and views Many data analysis systems need to periodically execute the same computation on changing datasets. Several examples are: Generating company financial re

Explore this link on the map →

saved by

related reading