The data platform

Ronja is a lakehouse: columnar files in object storage, queried in place by a fast analytical engine, with transforms materialized back into the same store. This page walks the layers from disk up, in the terms you’d use to evaluate any lakehouse.

Storage — columnar Parquet in S3

Every table is stored as one or more Parquet files in S3, under a deterministic per-table key prefix. Parquet is the format across the board: connectors upload it, the manual push API takes it, and derived and dynamic builds write it. Column statistics — type, min/max, missing count, cardinality, examples — are computed at build time and stored separately in Postgres, so schema and stats are queryable without opening a data file.

Parquet is the right substrate for analytics precisely because it’s columnar: a query that reads 3 of 50 columns from a 10 GB file pulls roughly 600 MB, not 10 GB — the engine prunes columns and skips row groups over HTTP range requests. Real columnar read behavior, not a row store pretending.

Partitioning — row-group split and optional Hive partitioning

A single Parquet file is a coarse unit, so Ronja splits tables into partition files two ways:

• Row-group partitioning (the default for derived tables). After a table is materialized to one Parquet file, Ronja reads that file’s row-group metadata, groups row groups up to a configured maximum partition size, and writes each group out as its own partition file. Each partition build can run locally or as its own AWS Lambda, several at a time.
• Hive-style partitioning (opt-in). A partition-key column drives Hive-style partitioning, one Lambda per partition — useful when you want a specific column to bound the physical layout.

Partition records are created as pending, flipped to building, then ready, so the build progress you see in the product is the real state of each partition, not a spinner.

The query engine — DuckDB, reading Parquet in place

The query engine is DuckDB: embedded, columnar, and vectorized. There’s no Trino, no Snowflake, no separate warehouse in the query path. Ronja tables are referenced in SQL with dbt-style {{ ref('tableID') }}, and Ronja rewrites each ref into a read_parquet(...) over that table’s resolved partition files in S3. DuckDB reads the Parquet directly off S3 — there is no load-into-warehouse step, and no copy of your data sitting in a proprietary format.

Execution runs inside a container that opens a DuckDB connection and runs the SQL. Compute is AWS Lambda or AWS Batch, sized to the data: a light tier on Lambda (6 GB RAM) for small working sets, and Batch tiers scaling up to 16 vCPU and 120 GB of RAM for heavy joins and unnests. The route is chosen automatically by inspecting the actual generated code, and a Lambda that runs out of memory or hits its timeout auto-escalates to Batch with no code rewrite.

The honest framing for the Snowflake comparison: the win is zero idle cost, not sub-second latency at every scale. There’s no always-on cluster keeping data hot in RAM, so a query is a serverless invocation — Lambda cold starts are short, Batch cold start is around 30 seconds. You trade a persistent warm cluster for paying nothing when nothing runs, plus the headroom to scale to 120 GB when a single query needs it. There’s no published benchmark against Snowflake here; the performance story is columnar pushdown plus right-sized compute — qualitative, not a measured latency claim.

Note

Interactive queries default to a cap of 100k rows and 10 table references (workflows allow up to 50). A per-session cache pre-stages the Parquet files a session touches onto the container’s local disk, so repeated reads within a session don’t re-fetch from S3.

Transforms — SQL materialized to Parquet (the dbt model)

A derived table is exactly the dbt “model” concept. It stores DuckDB SQL — with {{ ref('tableID') }} references to its inputs — plus the list of upstream table IDs it reads. On build, the refs are rewritten to read the upstream partitions directly, the SQL runs on DuckDB (Lambda or Batch), and the result is materialized to Parquet and split into partitions. The difference from dbt is that the physical materialization and partitioning are automatic — you describe the transform, Ronja handles the layout.

The rebuild DAG — dependents re-materialize automatically

This is dbt’s “run the whole DAG when upstream changes,” as an event cascade. When any table finishes materializing, it emits an event; Ronja looks up the live tables that read it as an input, marks each invalidated, and re-materializes it against the now-updated upstream. Each rebuild emits its own event, so the cascade propagates transitively down the DAG. Integration tables refresh on connector sync, and the derived DAG below them re-materializes through the same mechanism.

Two guardrails worth knowing: builds are serialized behind locks with a stale escape hatch, so two builds don’t race; and a failed draft build is a no-op on the live table — the live table stays queryable, and the failed draft persists for the agent to fix rather than corrupting what’s already there.

Snapshots and version history — Postgres ledger plus S3 versioning

The single source of truth for a table’s current data files is a Postgres partition ledger (the ready partition rows and their file keys), not an open-table-format catalog. If you want the honest version of the “open table format” question: Ronja implements safe snapshot-swap and version history in Postgres plus S3 versioning, not via Iceberg or Delta.

The mechanics that give you the time-travel property people expect from a table format:

• A rebuild writes to a fresh sub-prefix and only flips the partition ledger to the new file set once the new build is complete — so a reader querying mid-rebuild always sees a valid, consistent file set, never a half-built table.
• Committed versions are immutable snapshot rows, so you can restore an earlier committed version of a table.
• Old files aren’t deleted inline; a background reaper removes orphans, and S3 versioning gives a 30-day recovery window.

Lineage — computed from real edges, not guessed

Lineage in Ronja is derived from real reference edges — a derived table’s declared inputs, a workflow’s input and output tables, a data app’s referenced tables — not inferred by a model. You reach it two ways:

• The dependency graph. A table’s page shows its inputs, dependents, and the workflows that produce it, and a graph view shows how that table connects across features to the workflows, notes, and apps around it.
• “Explain this.” Any resource can be explained in grounded plain language: Ronja walks its immediate sources and dependents and writes prose that may only translate the facts in the graph — it never invents an edge, and when the walk is truncated the explanation is forced to hedge.

Because lineage is real, every answer is traceable: you can always walk from a number back to the tables and transforms that produced it.

For the anti-hallucination side of that story — why the numbers themselves are computed, not generated — see Deterministic by design. For the hands-on table workflow, see Create tables with Ronja and the Tables concept.