The data platform

Ronja is a lakehouse: columnar files in object storage, queried in place by a fast analytical engine, with transforms materialized back into the same store. This page walks the layers from disk up, in the terms you’d use to evaluate any lakehouse.

Storage — columnar Parquet in S3

Every table is stored as one or more Parquet files in S3, under a deterministic per-table key prefix. Parquet is the format across the board: connectors upload it, the manual push API takes it, and derived and dynamic builds write it. Column statistics — type, min/max, missing count, cardinality, examples — are computed at build time and stored separately in Postgres, so schema and stats are queryable without opening a data file.

Parquet is the right substrate for analytics precisely because it’s columnar: a query that reads 3 of 50 columns from a 10 GB file pulls roughly 600 MB, not 10 GB — the engine prunes columns and skips row groups over HTTP range requests. Real columnar read behavior, not a row store pretending.

Partitioning — row-group split and optional Hive partitioning

A single Parquet file is a coarse unit, so Ronja splits tables into partition files two ways:

• Row-group partitioning (the default for derived tables). After a table is materialized to one Parquet file, Ronja reads that file’s row-group metadata, groups row groups up to a configured maximum partition size, and writes each group out as its own partition file. Each partition build can run locally or as its own AWS Lambda, several at a time.
• Hive-style partitioning (opt-in). A partition-key column drives Hive-style partitioning, one Lambda per partition — useful when you want a specific column to bound the physical layout.

Partition records are created as pending, flipped to building, then ready, so the build progress you see in the product is the real state of each partition, not a spinner.

The query engine — DuckDB, reading Parquet in place

The query engine is DuckDB: embedded, columnar, and vectorized. There’s no Trino, no Snowflake, no separate warehouse in the query path. Ronja tables are referenced in SQL with dbt-style {{ ref('tableID') }}, and Ronja rewrites each ref into a read_parquet(...) over that table’s resolved partition files in S3. DuckDB reads the Parquet directly off S3 — there is no load-into-warehouse step, and no copy of your data sitting in a proprietary format.

Execution runs inside a container that opens a DuckDB connection and runs the SQL. Compute is AWS Lambda. Every table build and every query runs on one, where DuckDB gets 6 GB of memory, 7 GB of disk to spill to, and 14 minutes. A big table whose SQL works row by row can be built in several pieces at once, each on its own Lambda (a Parallel build). A build or query that needs more than one Lambda can give fails with an error that says it ran out of resources, and Ronja does not quietly retry it on a bigger machine. The fix is in the SQL: read fewer columns, filter earlier, or aggregate before joining. Long-running Python work belongs in a Workflow on dedicated compute.

The honest framing for the Snowflake comparison: the win is zero idle cost, not sub-second latency at every scale. There’s no always-on cluster keeping data hot in RAM, so a query is a serverless invocation, and Lambda cold starts are short. You trade a persistent warm cluster for paying nothing when nothing runs. There’s no published benchmark against Snowflake here; the performance story is columnar pushdown plus parallel builds, a qualitative claim rather than a measured latency one.

Note

Interactive queries default to a cap of 100k rows and 50 table references per query. A workflow may bind at most 100 distinct tables across its whole file set, so splitting one wide query into several relieves the per-query cap of 50 but does not raise the workflow-wide budget of 100. A per-session cache pre-stages the Parquet files a session touches onto the container’s local disk, so repeated reads within a session don’t re-fetch from S3.

Backslashes in SQL reach the engine exactly as written

SQL is handed to the query engine byte for byte, so a backslash in a string literal is a backslash. Write '\s' for a whitespace class in a regular expression, '\d' for a digit class, and E'\n' when you want a real newline rather than the two characters. This holds almost everywhere SQL is written — in a conversation with Ronja, in a chart, in a Table, in an App, in a Workflow, in the .sql files a pipeline folder deploys, and in queries sent through the API. The one exception is the optionsQuery of a Workflow’s select parameter, which always reads backslashes the older way described below; leave its backslashes as they are.

Tables, Apps and Workflows created before this changed still read backslashes the older way: one round of escape decoding happened before the engine saw the SQL, so the same regular expression had to be written '\\s'. They keep working exactly as they always have — nothing was rewritten underneath you. A Legacy SQL escapes chip says so on an App’s and a Workflow’s page, and on the page of a Derived table written in SQL; the other kinds of table store no SQL of their own, so there is nothing for the chip to describe and it does not appear. A health check written before the change also keeps the older reading until its rule is edited, and shows no chip.

Editing the code of one of these moves it onto the current behaviour automatically — whether you make the edit or Ronja does, in the app or from a CLI folder. Renaming it, changing its description or changing any other setting leaves it as it is; only a change to the SQL or code itself moves it. When Ronja makes the edit she rewrites the backslashes the older reading depended on in that same edit; before you publish, compare the draft with the published version to check nothing was missed. For a Table, an App or a Workflow the edit lands on a draft, so the chip clears when you publish it — until then the published version keeps reading backslashes the older way, and your test runs use the new one. A Workflow that uses a shared module is the one exception: it reads the older way for as long as the module version it uses does, test runs included, so its chip stays until that module has been edited and published and the Workflow is published again.

Restore from version history brings back the rules the version was written under, along with its SQL. Restoring a version from before the change puts the older reading back and the chip returns; restoring a later one keeps the current behaviour. A draft works the same way: it keeps the rules its SQL was copied under when it was opened, so publishing a draft that someone opened before the change — say, to edit a description — brings the older reading back with that draft’s SQL, and the chip returns.

Transforms — SQL materialized to Parquet (the dbt model)

A derived table is exactly the dbt “model” concept. It stores DuckDB SQL — with {{ ref('tableID') }} references to its inputs — plus the list of upstream table IDs it reads. A ref may be written as a name instead, {{ ref('Sales.orders') }}, which is resolved to the id when the SQL is saved and stored as the id — so the DAG edge is always an id and a rename never touches it. On build, the refs are rewritten to read the upstream partitions directly, the SQL runs on DuckDB on Lambda, and the result is materialized to Parquet and split into partitions. The difference from dbt is that the physical materialization and partitioning are automatic — you describe the transform, Ronja handles the layout.

The rebuild DAG — dependents re-materialize automatically

This is dbt’s “run the whole DAG when upstream changes,” as an event cascade. When any table’s data changes — it finishes materializing, or rows are written to, cleared from or restored into a Dynamic table — it emits an event; Ronja looks up the live tables that read it as an input, marks each one behind, and re-materializes it against the now-updated upstream. (After a clear that re-materialization fails, since a clear removes the table’s data and column types and there is nothing to build from, and the dependent keeps serving its earlier data. An input that was built or written and simply holds zero rows is different: its dependents rebuild to zero rows and keep their columns.) Each rebuild emits its own event, so the cascade propagates transitively down the DAG. Integration tables refresh on connector sync, on a pushed-file Build, or on a manual admin rebuild; Dynamic tables change on each row write, whether it comes from your own app over the API, from an agent, or from a Workflow writing its output table. The derived DAG below them re-materializes through the same mechanism regardless of which of those started it.

Three guardrails worth knowing: builds are serialized behind locks with a stale escape hatch, so two builds don’t race; a join point — a table reading several inputs that the same upstream change reaches — re-materializes once, after the last of its inputs has caught up, rather than once per path (a diamond of depth n would otherwise cost 2ⁿ builds and publish a wrong intermediate join after each), with a bounded wait so a stuck input can never hold it back for good; and a failed draft build is a no-op on the live table — the live table stays queryable, and the failed draft persists for the agent to fix rather than corrupting what’s already there.

Snapshots and version history — Postgres ledger plus S3 versioning

The single source of truth for a table’s current data files is a Postgres partition ledger (the ready partition rows and their file keys), not an open-table-format catalog. If you want the honest version of the “open table format” question: Ronja implements safe snapshot-swap and version history in Postgres plus S3 versioning, not via Iceberg or Delta.

The mechanics that give you the time-travel property people expect from a table format:

• A rebuild writes to a fresh sub-prefix and only flips the partition ledger to the new file set once the new build is complete — so a reader querying mid-rebuild always sees a valid, consistent file set, never a half-built table.
• Committed versions are immutable snapshot rows, so you can restore an earlier committed version of a table.
• Old files aren’t deleted inline; a background reaper removes orphans, and S3 versioning gives a 30-day recovery window.
• A dynamic table that is appended to in many small writes is compacted in the background: its small data files are merged into larger ones and swapped into the ledger in one step. The files it replaced are kept for the 7-day version-history window, so a query already reading them, or a restore, still finds them.

Lineage — computed from real edges, not guessed

Lineage in Ronja is derived from real reference edges — a derived table’s declared inputs, a workflow’s input and output tables, an app’s referenced tables — not inferred by a model. You reach it two ways:

• The dependency graph. A table’s page shows its inputs, dependents, and the workflows that produce it, and a graph view shows how that table connects across features to the workflows, notes, and apps around it.
• “Explain this.” Any resource can be explained in grounded plain language: Ronja walks its immediate sources and dependents and writes prose that may only translate the facts in the graph — it never invents an edge, and when the walk is truncated the explanation is forced to hedge. Most of the sources and dependents it names are links that open in a new tab, so you can step straight from the prose to that table, workflow, automation or secret.

Because lineage is real, every answer is traceable: you can always walk from a number back to the tables and transforms that produced it.

For the anti-hallucination side of that story — why the numbers themselves come from an engine, and how the figures written about them are checked — see Deterministic by design. For the hands-on table workflow, see Create tables with Ronja and the Tables concept.