The data platform

Ronja is a lakehouse: columnar files in object storage, queried in place by a fast analytical engine, with transforms materialized back into the same store. This page walks the layers from disk up, in the terms you’d use to evaluate any lakehouse.

Storage — columnar Parquet in S3

Every table is stored as one or more Parquet files in S3, under a deterministic per-table key prefix. Parquet is the format across the board: connectors upload it, the manual push API takes it, and derived and dynamic builds write it. Column statistics — type, min/max, missing count, cardinality, examples — are computed at build time and stored separately in Postgres, so schema and stats are queryable without opening a data file.

Parquet is the right substrate for analytics precisely because it’s columnar: a query that reads 3 of 50 columns from a 10 GB file pulls roughly 600 MB, not 10 GB — the engine prunes columns and skips row groups over HTTP range requests. Real columnar read behavior, not a row store pretending.

Partition files

A table’s data is held as one or more partition files:

• A Derived table written in SQL is built as one file of at most 2.5 GB. DuckDB writes it in row groups, each with its own column statistics, so a query that filters still skips the row groups it does not need.
• A Derived table written in Python is split after its build: Ronja reads the file’s row-group metadata, groups row groups up to a maximum partition size, and writes each group out as its own partition file.
• A Dynamic table gains a data file with each write, and Ronja merges small ones in the background.

A build creates its partition records as pending, flips each to building as it starts, then to ready; a write to a Dynamic table creates its record as building, since it starts at once. So the build progress you see in the product is the real state of each partition, not a spinner.

The query engine — DuckDB, reading Parquet in place

The query engine is DuckDB: embedded, columnar, and vectorized. There’s no Trino, no Snowflake, no separate warehouse in the query path. Ronja tables are referenced in SQL with dbt-style {{ ref('tableID') }}, and Ronja rewrites each ref into a read_parquet(...) over that table’s resolved partition files in S3. DuckDB reads the Parquet directly off S3 — there is no load-into-warehouse step, and no copy of your data sitting in a proprietary format.

DuckDB runs the SQL in one of two places, and Ronja picks per query. Small queries and small table builds go to a warm query tier: DuckDB processes that stay running and keep the Parquet files they recently read cached on local disk next to the engine, so a chart, a metric, a table’s health check, an app’s query or a question in a conversation is typically answered in tens of milliseconds. A query or build too large for it, or one it cannot take at that moment, runs on AWS Lambda instead, with the same answer. There DuckDB gets 9 GB of memory, 2.5 GB of disk to spill to, and 13 minutes; the tables a query or build names may hold at most 4 GB of data in all (each counted at its whole size, however the SQL filters it), a query’s answer is at most 64 MB, and a table build writes a table of at most 2.5 GB. A table written in Python builds on Lambda with 6 GB of memory, 7 GB of spill disk and 14 minutes. A build or query that needs more than that fails with an error that says what it ran out of, and Ronja does not quietly retry it on a bigger machine. The fix is in the SQL: read fewer columns, filter earlier, aggregate before joining, or split a large table into smaller ones. The 4 GB input limit is the exception, which no change to the SQL moves: read fewer or smaller tables. Long-running Python work belongs in a Workflow on dedicated compute.

The honest framing for the Snowflake comparison: the win is that you pay for the queries that run, not for a cluster that sits idle, and not sub-second latency at every scale. The warm query tier is a small pool sized for small queries, not a warehouse holding your tables in RAM: it caches the files it has recently read and lets them go, and a larger query is a serverless invocation, where Lambda cold starts are short. There’s no published benchmark against Snowflake here; the performance story is columnar pushdown and the warm tier’s local cache, a qualitative claim rather than a measured latency one.

Note

Interactive queries default to a cap of 100k rows and 50 table references per query. A workflow may read at most 100 distinct tables across its whole file set, so splitting one wide query into several relieves the per-query cap of 50 but does not raise the workflow-wide budget of 100. A per-session cache pre-stages the Parquet files a session touches onto the container’s local disk, so repeated reads within a session don’t re-fetch from S3.

Backslashes in SQL reach the engine exactly as written

SQL is handed to the query engine byte for byte, so a backslash in a string literal is a backslash. Write '\s' for a whitespace class in a regular expression, '\d' for a digit class, and E'\n' when you want a real newline rather than the two characters. This holds almost everywhere SQL is written — in a conversation with Ronja, in a chart, in a Table, in an App, in a Workflow, in the .sql files a pipeline folder deploys, and in queries sent through the API. The one exception is the optionsQuery of a Workflow’s select parameter, which always reads backslashes the older way described below, so one with a backslash escape the older reading decoded (such as \1, \b, \' or \\) is refused with an error naming the parameter and the escape: write it without that escape (chr(92) is a backslash). \d, \s and other escapes the older reading left alone run as written.

Tables, Apps and Workflows created before this changed still read backslashes the older way: one round of escape decoding happened before the engine saw the SQL, so the same regular expression had to be written '\\s'. They keep working as they always have — nothing was rewritten underneath you — with one exception: a query whose SQL has a backslash escape the older reading decoded (such as \1, \b, \', \\ or \n), sent by one of these Apps or Workflows, by a Script or by a health check still on the older reading, is refused with an error naming the escape and saying what to do: an App or a Workflow must be edited and published, a Script edited. So is the build of a Derived table written in SQL that the older reading read differently — a backslash escape it decoded (such as \1, \b, \' or \\), or a line break inside a quoted value, among others — with an error naming the table, the escape and its line: its recipe must be edited and published. In a query or a table’s SQL, \d, \s and other escapes the older reading left alone run as written, and most escapes inside a comment don’t count; the error names any that does. Editing its code, as below, is the fix. A Legacy SQL escapes chip says so on an App’s and a Workflow’s page, and on the page of a Derived table written in SQL; the other kinds of table store no SQL of their own, so there is nothing for the chip to describe and it does not appear. A health check written before the change also keeps the older reading until its rule is edited, and shows no chip; one whose rule has a backslash escape the older reading decoded reads Couldn’t be evaluated, with the refusal as its detail, until the rule is rewritten.

Editing the code of one of these moves it onto the current behaviour automatically — whether you make the edit or Ronja does, in the app or from a CLI folder. Renaming it, changing its description or changing any other setting leaves it as it is; only a change to the SQL or code itself moves it. When Ronja makes the edit she rewrites the backslashes the older reading depended on in that same edit; before you publish, compare the draft with the published version to check nothing was missed. For a Table, an App or a Workflow the edit lands on a draft, so the chip clears when you publish it — until then the published version keeps reading backslashes the older way, and your test runs use the new one. A Workflow that uses a shared module is the one exception: it reads the older way for as long as the module version it uses does, test runs included, so its chip stays until that module has been edited and published and the Workflow is published again.

Restore from version history brings back the rules the version was written under, along with its SQL. Restoring a version from before the change puts the older reading back and the chip returns; restoring a later one keeps the current behaviour. A draft works the same way: it keeps the rules its SQL was copied under when it was opened, so publishing a draft that someone opened before the change — say, to edit a description — brings the older reading back with that draft’s SQL, and the chip returns.

Transforms — SQL materialized to Parquet (the dbt model)

A derived table is exactly the dbt “model” concept. It stores DuckDB SQL — with {{ ref('tableID') }} references to its inputs — plus the list of upstream table IDs it reads. A ref may be written as a name instead, {{ ref('Sales.orders') }}, which is resolved to the id when the SQL is saved and stored as the id — so the DAG edge is always an id and a rename never touches it. On build, the refs are rewritten to read the upstream partitions directly, the SQL runs on DuckDB on Lambda, and the result is materialized to Parquet and split into partitions. The difference from dbt is that the physical materialization and partitioning are automatic — you describe the transform, Ronja handles the layout.

The rebuild DAG — dependents re-materialize automatically

This is dbt’s “run the whole DAG when upstream changes,” as an event cascade. When any table’s data changes — it finishes materializing, or rows are written to, cleared from or restored into a Dynamic table — it emits an event; Ronja looks up the live tables that read it as an input, marks each one behind, and re-materializes it against the now-updated upstream. (After a clear that re-materialization fails, since a clear removes the table’s data and column types and there is nothing to build from, and the dependent keeps serving its earlier data. An input that was built or written and simply holds zero rows is different: its dependents rebuild to zero rows and keep their columns.) Each rebuild emits its own event, so the cascade propagates transitively down the DAG. Integration tables refresh on connector sync, on a pushed-file Build, or on a manual admin rebuild; Dynamic tables change on each row write, whether it comes from your own app over the API, from an agent, or from a Workflow writing its output table. The derived DAG below them re-materializes through the same mechanism regardless of which of those started it.

Three guardrails worth knowing: builds are serialized behind locks with a stale escape hatch, so two builds don’t race; a join point — a table reading several inputs that the same upstream change reaches — re-materializes once, after the last of its inputs has caught up, rather than once per path (a diamond of depth n would otherwise cost 2ⁿ builds and publish a wrong intermediate join after each), with a bounded wait so a stuck input can never hold it back for good; and a failed draft build is a no-op on the live table — the live table stays queryable, and the failed draft persists for the agent to fix rather than corrupting what’s already there.

Snapshots and version history — Postgres ledger plus S3 versioning

The single source of truth for a table’s current data files is a Postgres partition ledger (the ready partition rows and their file keys), not an open-table-format catalog. If you want the honest version of the “open table format” question: Ronja implements safe snapshot-swap and version history in Postgres plus S3 versioning, not via Iceberg or Delta.

The mechanics that give you the time-travel property people expect from a table format:

• A rebuild writes new files, never over the current ones, and only flips the partition ledger to the new file set once the new build is complete — so a reader querying mid-rebuild always sees a valid, consistent file set, never a half-built table.
• Committed versions are immutable snapshot rows, so you can restore an earlier committed version of a table.
• Old files aren’t deleted inline; a background reaper removes orphans, and S3 versioning gives a 30-day recovery window.
• A dynamic table that is appended to in many small writes is compacted in the background: its small data files are merged into larger ones and swapped into the ledger in one step. The files it replaced are kept for the 7-day version-history window, so a query already reading them, or a restore, still finds them.

Lineage — computed from real edges, not guessed

Lineage in Ronja is derived from real reference edges — a derived table’s declared inputs, a workflow’s input and output tables, an app’s referenced tables — not inferred by a model. You reach it two ways:

• The dependency graph. A table’s page shows its inputs, dependents, and the workflows that produce it, and a graph view shows how that table connects across features to the workflows, notes, and apps around it.
• “Explain this.” Any resource can be explained in grounded plain language: Ronja walks its immediate sources and dependents and writes prose that may only translate the facts in the graph — it never invents an edge, and when the walk is truncated the explanation is forced to hedge. Most of the sources and dependents it names are links that open in a new tab, so you can step straight from the prose to that table, workflow, automation or secret.

Because lineage is real, every answer is traceable: you can always walk from a number back to the tables and transforms that produced it.

For the anti-hallucination side of that story — why the numbers themselves come from an engine, and how the figures written about them are checked — see Deterministic by design. For the hands-on table workflow, see Create tables with Ronja and the Tables concept.