Curation Labs · The post-AI data stack

Analysis is cheap.
Agreement isn’t.

Fifteen years of data infrastructure on one canvas, after Ian Macomber’s post, and where the platform we’re building sits on it.

SpeakerRemy

AudienceCuration Labs

Runtime~25 min, then questions

AfterIan Macomber, 2026

Source — I. Macomber,
The Shape and Feel of the Post-AI Data Stack (30 Aug 2026)

The source

Ian Macomber’s post tells fifteen years of data infrastructure
in four diagrams.

Macomber's ~2013 pre-modern data stack: two OLTP databases inside the reachable boundary, everything else outside it. — **Fig. 1 · 2013**Questions that fit on one server.

Macomber's ~2016 cloud data warehouse stack: the boundary now covers clickstream and a warehouse feeding BI tools. — **Fig. 2 · 2016**Questions across the whole business.

Macomber's ~2020 Modern Data Stack: managed ETL in, dbt in the warehouse, Reverse ETL back out to operational systems. — **Fig. 3 · 2020**From reporter to operator.

Macomber's ~2026 post-AI data stack: an agent harness between data stores and interfaces, fed by company context and a feedback loop. — **Fig. 4 · 2026**Encode data judgment as infrastructure for agents.

The dashed purple line in each one is everything you can actually query.

The post is by Ian Macomber. He started his career in data at Wayfair in 2013 and now works on data at Ramp. It walks through fifteen years of data infrastructure, one diagram per era, and ends with an argument about what changes once agents are the ones asking the questions. I'm going to use his four diagrams to teach the infrastructure, and I'm going to do it the same way each time: what the previous setup couldn't do, the component that fixed it, and what still couldn't be done afterwards. Every box on the diagram exists because something before it failed. Most of you haven't worked on a data platform. You don't need to know what a warehouse is. By the end you'll know why one exists, and then why we didn't build one. [Delivery] Point at the dashed line in fig. 1, then in fig. 4. Don't walk the boxes: the canvas does that for the next twenty minutes. His diagrams come back full-screen, one per era.

Source — I. Macomber, agent-readable artifacts (2026)

The picture of 2027 the post paints

In 2027 your CPO reads the dashboard
without ever opening it.

“By 2027, your CPO will learn about your team’s product launch through a subagent, deployed by an agent, that scans every product launch dashboard once a week.”Ian Macomber · 2026

Does our work show up in that paragraph, and is the paragraph right?

How to read the next twenty minutes

One question, five sources, and one dashed line.

Running question

Did Tuesday’s deploy slow checkout, and which customers noticed?

Every era, four beats

walls left standing
one component per wall
run the question
name the new walls

I'll carry one question through the whole talk: did Tuesday's deploy slow down checkout, and which customers noticed? Engineering, product, support and finance all care about it, and it needs all five sources on the left: App DB, Billing, Telemetry, Tickets, Docs. This is the canvas for every era. Each column answers one plain question: how does data get in, where does it live, what does a number mean, who does the asking, what comes out, who wants to know. Each era answers them differently. Watch which of the five sources end up inside the dashed line. Pushing it outward is the whole story. Every era gets the same four beats: the walls left standing from the last one, one component per wall, run the question, and name the new walls. [Delivery] The walls-left-standing beat is what makes this a chain instead of a list. The band on screen is the promise; keep it every era. The small panel on the right is his four diagrams; the one we're redrawing lights up.

After — I. Macomber, fig. 1 (2026)

Act one · ~2013 · Pre-modern

In 2013 you could only ask what fit on one server.

Blocked

Did Tuesday’s deploy slow checkout, and which customers noticed?

Can

Count orders per hour from the App DB, if the DBA runs it for you.

Can’t

Join them to Billing, see latency at all, or get the answer before Monday.

Three walls: islands, one machine, and events not stored.

This is his first diagram. There's nothing to compare it to yet, so let me describe the walls. In 2013 the data lives in the databases that run the application. The App DB holds the orders. Billing holds the invoices. Those are two different servers, and there is no such thing as a query across two servers. That's the first wall: the data is in islands. These databases are built for writes. A row is one order, kept together so the app can fetch it in a millisecond. That's the right shape for serving a page and the wrong shape for average checkout time by hour, which wants one column across millions of rows. The terms are OLTP, transaction processing, and OLAP, analytical processing. Same data, opposite access patterns. And the machine doing the analytical read is the same machine serving customers. An analyst writes a reasonable-looking join, runs it against the production replica, the replica falls behind, and the on-call gets paged. One bad question hurts every customer. The DBA's job becomes saying no, and everything goes through the DBA. That's the second wall: analysis and serving share one machine, so analysis is rationed. Telemetry, the stream of latency events, isn't stored at all. It's JSON, it arrives at thousands of events a second, and a relational database has nowhere sensible to put it. Third wall: nowhere to keep events. Run the question. You can count orders per hour from the App DB, if the DBA runs it for you. You can't join those to Billing. You can't see latency, because Telemetry isn't captured. And the answer, if you get one, is a spreadsheet emailed on Monday. Three walls: islands, one machine, events not stored. Keep count. The next slide knocks down all three. [Delivery] Click 1 brings up his fig. 1 full-screen; click 2 returns to the canvas. The coral tags are the walls, one per click: ISLANDS, then ONE MACHINE as storage and compute fuse, then NOT STORED. Tell the production-replica story over ONE MACHINE.

After — I. Macomber, fig. 2 (2026)

Act one · ~2016 · Cloud warehouse

Storage splits from compute, and all the data lands in one place.

Answerable

Did Tuesday’s deploy slow checkout, and which customers noticed?

Can

Join orders, invoices and latency events in one place. A data scientist finds the slowdown.

Can’t

Tie it to support tickets, which live in the helpdesk. Nobody else can rerun the notebook.

Three new walls: bespoke pipelines, personal definitions, and every answer ends in a slide.

Three walls left standing from 2013: islands, one machine, events not stored. The cloud data warehouse is a direct answer to those three walls, one feature each. Islands. The warehouse is one place that everything lands in. App DB and Billing are copied into the same store, and now they can be joined. One machine. Watch the fused box split in two. Storage becomes cheap object storage, effectively unlimited. Compute becomes clusters you spin up when you need them and shut down when you don't. Analysis runs on its own compute, so it can't page the on-call, and it can be as big as the question needs. This split is the idea the rest of the talk depends on. Keep the two boxes in your head; they come back in Act 3 with our names on them. Events not stored. The warehouse reads semi-structured data: JSON goes in a column and you query inside it. Ugly, but Telemetry is now inside the dashed line. Run the question. App DB, Billing and Telemetry are in one place, so orders, invoices and latency events can be joined. A data scientist can find the slowdown. Now the new walls. It took that data scientist about a week, because getting Tickets into the warehouse means asking engineering to write and maintain a custom pipeline, and the queue for that is long. The definition of checkout latency and the whole analysis live in a notebook on one laptop. Nobody else can rerun it, and next month someone will define it differently. And the output is a chart. Nothing in the product changes because of it. Three new walls: pipelines are bespoke, definitions are personal, and every answer ends in a slide. [Delivery] Click 1 is his fig. 2. The split on click 3 is the one piece of motion in Act 1: let it finish before you speak. If running long, drop the JSON sentence.

After — I. Macomber, fig. 3 (2026)

Act one · ~2020 · Modern data stack

Data comes in with a click, and the answer goes back into the product.

Actionable

Did Tuesday’s deploy slow checkout, and which customers noticed?

Can

Tickets land by managed ETL, latency is defined once in dbt, affected accounts get tagged in the CRM.

Can’t

Read the incident doc or the PR, or try a new angle without someone writing SQL.

Three walls left: text goes unread, every question needs SQL, and definitions run on the honor system.

Three walls left standing from 2016: bespoke pipelines, personal definitions, and answers that end in a slide. The modern data stack is three tools, one per wall. Bespoke pipelines. Managed ETL: services that connect to your ticket system, your CRM, your ad platforms, and land the data in the warehouse. The post's contrast is vivid: Hadoop at Wayfair took an outside implementation team, three data engineers and four months, and by 2020 a new head of data with a credit card could have reporting up by lunch. You click a button and pay monthly instead of asking engineering for a pipeline. That's why Tickets comes inside the dashed line. Personal definitions. dbt: metrics defined in version-controlled SQL, run on a schedule, with a lineage graph showing what each one depends on. Checkout latency is defined once, in one file, reviewed like code. Answers that end in a slide. Reverse ETL: the loop along the bottom. Results flow back out into the operational systems. The post calls this the shift from reporter to operator. The data team stops producing decks and starts changing what happens in the product. Run the question. Tickets are in. Checkout latency means one thing. Affected accounts get tagged in the CRM, and support sees the tag on the next ticket. Actionable. New walls. Docs is still grey: the incident doc and the pull request description are unstructured text, and nothing in this stack reads text. Second, every new angle on the question still needs a person who writes SQL, so throughput is capped by the size of the data team, and the queue comes back in a new form. Third, the definitions exist in dbt, but they're enforced by humans reading documentation. A new person, or a new tool, can ignore them without anything noticing. It's an honor system. [Delivery] Click 1 is his fig. 3. The act loop on click 4 is the first loop on the canvas; trace it with your hand from the right edge back to the sources.

After — I. Macomber, figs. 1–3 (2026)

End of act one

Each era removed a constraint on
who could ask what of which data.

EraKnocked downLeft standing

~2013

—

islands · one machine · not stored

~2016

islands · one machine · not stored

bespoke · personal · ends in a slide

~2020

bespoke · personal · ends in a slide

unread · needs SQL · honor system

The next era removes two of these three, and runs straight into the third.

After — I. Macomber, fig. 4 (2026)

Act two · 2026 · Post-AI

An agent harness sits between all the data and every interface.

Agent-answerable?

Did Tuesday’s deploy slow checkout, and which customers noticed?

Can

An agent reads the incident doc, the PR and the metric definition, and answers in Slack in minutes.

Can’t

Promise the coding agent and the Slack bot use the same definition of latency. Nothing measures it.

AI made producing analysis nearly free. It did not make agreeing on reality free.

Three walls left standing from 2020: text goes unread, every question needs SQL, and definitions run on the honor system. Two of them fall quickly. Text goes unread. Now models read text. The incident doc becomes rows you can join. Every source is inside the dashed line. Every question needs SQL. Now an agent writes the SQL. That's the new box in the middle: the harness, which is what an agent runs inside: its tools, its model, its skills, its evals. It sits between all the data and every interface. The person asking no longer needs to know SQL, or the schema, or which table to use. The queue is gone. The post adds two things around it. Context feeds the harness: the semantic layer, which is where definitions live, and the domain docs. And a feedback loop, along the top, routes every answer, correction and eval back into that context. That leaves the third wall, and this is where the post's argument starts. Definitions run on the honor system. When one data scientist wrote every query, that was fine: one person, one reading of the definition, one answer. Now any agent through any interface writes the query, and each of them reads the definition, or doesn't, on its own. Run the question. Any interface produces a number in minutes. The question is whether they all produce the same one. Here's the claim. AI made producing analysis nearly free. It did not make agreeing on reality free. The scarce resource becomes consensus. [Delivery] Click 1 is his fig. 4, the one zoom that earns a sentence: point at his harness, his company context and his orange feedback loop, then come back to ours. On click 5 the honor-system tag becomes SAME ANSWER?, and that tag stays on the canvas until the close.

Source — I. Macomber, agent-testable consensus (2026)

The fan-out

Ask the same question through five interfaces
and five numbers come back.

Same data · same question · five definitions

The question

Did Tuesday’s deploy slow checkout, and which customers noticed?

Slack botp95 server latency · noticed = opened a ticket

Coding agentp50 client latency · noticed = abandoned a cart

Notebook assistantp95 client latency · noticed = ticket or abandoned cart

BI tool’s chatp99, timeouts count · noticed = any slow session

A coworker’s agentmean latency · noticed = churned within a week

1 2 3 4 5 412customers 1,980customers 2,241customers 6,870customers 38customers

The exec

Five numbers. Which one is right?

1–5each interface chose its own definition of checkout latency and of noticed · the numbers are illustrative

“Assume the models are never wrong, your context is just underspecified.”

Take our question and ask it five ways: a Slack bot, a coding agent in the terminal, a notebook assistant, a BI tool's chat, a coworker's general-purpose agent. Each one decides what checkout latency means. p50 or p95? Server time or client time? Does a timeout count? Then each decides what noticed means. Opened a ticket? Abandoned a cart? Each choice is reasonable, and you get five numbers. The exec who receives all five doesn't know which is right, and stops trusting all of them. The post's advice, roughly: assume the model is never wrong, and your context was underspecified. That turns "the AI is flaky" into "our definitions have gaps", which we know how to fix. [Delivery] Never cut this slide. Let the five numbers land in silence for a beat before you speak. The numbers are illustrative; say so if asked.

Source — I. Macomber, agent-testable consensus (2026)

The post’s move

Consensus sounds vague.
It can be engineered, asserted and measured.

results = ask_everywhere(
    interfaces=["coworker", "coding_agent", "bi", "slack"],
    question="What was net revenue retention last quarter?",
)

assert same_metric_definition(results)
assert values_within_tolerance(results, relative=0.00001)
assert same_authorization_outcome(results)
assert required_evidence_used(results)

Ian Macomber · 2026 · pseudocode from the postSame definition, same value, same permissions, same evidence.

01Take every metric in last quarter’s board deck.
02Ask about each one through every interface and model.
03Count the distinct answers.
04One: celebrate. More than one: read the traces, fix, repeat.

A test tells you the agents disagreed. It can’t stop them disagreeing.

Source — I. Macomber, post-AI data stack (2026)

Four requirements

Four things the stack must be.
We’ll check each one against ours.

RequirementWhat it meansHow we’ll check

Agent-readable artifacts

Outputs are plain files an agent can open, not screenshots in a BI tool.

Can an agent read the output as a file?

Agent-operable tools

Everything is reachable by API or command. Nothing is UI-only.

Is there anything only a human can click?

Agent-agnostic context

Definitions live where any agent can read them, not inside one vendor.

Who owns the definitions?

Agent-testable consensus

The fan-out runs as a test and returns a number.

Does it?

The post asks for consensus you can test. We want consensus the data layer guarantees, and the test to prove it.

Act three · Today · Our platform

Same canvas, our names on the boxes, and a status on each.

built partial still to build requirement met

Answerable by one agent

Did Tuesday’s deploy slow checkout, and which customers noticed?

Can

An agent spins up a sandbox, queries the Iceberg tables with DuckDB, and hands back a Marimo notebook.

Can’t

Guarantee a second agent reads the same version, and the same definition of affected customer.

Storage and compute are the solved part. The dashed boxes are the scarce resource, and they’re ours to build.

The 2026 diagram tells you what the boxes are. It doesn't tell you how to build them without buying the walls back. A warehouse vendor gives you the 2016 boxes with its own opinion attached to both, and its chat interface becomes one more voice in the fan-out. So the constraint we designed against was this: nothing in the stack may have an opinion about which agent uses it. Same canvas, our names on the boxes, a status on each. Solid teal is built, dashed teal is partial, dashed coral is still to build. Left to right. Pipelines land data as Iceberg tables. Partially: some sources are there, some aren't. Storage is Iceberg on R2. This is the 2016 storage box. The R2 Data Catalog is the index of which tables exist. Compute is DuckDB, an in-process engine: it runs inside whatever process asks for it, no server. Each agent gets its own. Ten agents, ten engines, all reading the same tables, none able to slow the others down. The 2013 shared-machine wall, solved structurally rather than by policy. Container sandboxes are the harness. An agent's work happens in an isolated container with scoped credentials. When the task is done, the container is gone. Marimo notebooks are the artifacts. A Marimo notebook is a plain Python file, so an agent can read it, git can diff it, and it runs as a script. Requirement one, agent-readable: yes. Nothing here has a UI, so any agent can drive it. Requirement two, agent-operable: yes. The format, catalog and engine are all open, so no vendor owns the definitions. Requirement three, agent-agnostic: yes. Then the dashed boxes. No semantic layer, no domain docs, no evals. Docs aren't in yet, so the dashed line pulls back. And the feedback loop is still to build: no path for one agent's correction to improve the next agent's answer. Run the question. One agent can answer it end to end. Nothing guarantees that a second agent reads the same version, or the same definition. [Delivery] Confirm every status on this slide before presenting; the README lists them as unconfirmed. The gold tags are the three requirements met, and the marks under the kicker fill with them. The next slide explains the storage box; skip it if running long.

Format · catalog · engine

Iceberg is the binding, the catalog is the index,
and DuckDB is the reader.

In a libraryIn our platformWhat it knows

The shelvesR2

Holds files. Doesn’t know what’s in them.

The bindingIceberg

Which Parquet files make this table, its schema, every version it has had.

The indexR2 Data Catalog

Which tables exist, and where their metadata lives.

The readerDuckDB

Checks the index, opens the book, reads.

That’s the 2016 split taken to its end: the storage layer has no opinion about what reads it.

Think of R2 as shelves. It holds files and doesn't know what's in them. Iceberg is the binding on each book: a small set of metadata files that say this table is these Parquet files, here's the schema, here's every version it has ever had. The R2 Data Catalog is the library's index: which tables exist, and where their metadata lives. DuckDB is the reader, who checks the index, opens the book, and reads. Format, catalog, engine: three separate things, any of which can be swapped. That's the 2016 split taken to its end. The storage layer has no opinion about what reads it. [Delivery] Cuttable. If running long, skip this slide and say its heading over the storage step of the platform slide.

Our question, today

One agent answers it end to end.
Two agents disagree, and nothing notices.

Agent A

spins up a sandbox
opens DuckDB, reads the catalog
joins orders, latency events and tickets
affected customer = opened a ticket
returns a Marimo notebook: 412 customers

Agent B, same question

spins up a sandbox
opens DuckDB, reads the catalog
joins orders, latency events and tickets
affected customer = abandoned a cart
returns a Marimo notebook: 1,980 customers

✓Readable

plain .py notebooks

✓Operable

no UI anywhere

✓Agnostic

open format, catalog, engine

OPENConsensus

no shared definitions, no versioned reads

Consensus: not guaranteed, and not yet tested.

Here's our question today. One agent spins up a sandbox, joins orders, latency events and tickets through DuckDB, and hands back a Marimo notebook with a number. That works now. Now ask a second agent the same thing. It also works, and it defines affected customer differently, because nothing tells it how we define it. Two notebooks, two numbers. Readable, yes. Operable, yes. Agnostic, yes. Consensus: no shared definitions, nothing that pins which version an agent read, and no path for one agent's correction to reach the next. Nothing guarantees it, and we can't yet even test for it. [Delivery] The two numbers match the first two interfaces on the fan-out slide on purpose. If someone notices, that's the point.

Close · Next

Fill the dashed box, turn on the loop, and many agents work on one truth.

Consensus by design

Did Tuesday’s deploy slow checkout, and which customers noticed?

Can

Analysts run in parallel on the same committed data and definitions. A correction commits once, and every agent sees it.

Can’t

Replace the human call on what noticed means. That call is made once, then written into the semantic layer.

Many agents in parallel, one committed truth, and every correction written back once.

So the wall is: one agent answers well, two agents disagree, and nothing learns from the disagreement. Filling the dashed box and turning on the loop gives every agent one truth to read. But it also changes what the bottom of the diagram can be. The harness becomes a shared workspace: one sandbox, shared files, several agents working in it at once. The artifacts become project notebooks, a findings log of claims and their evidence, and definitions that get written back into the semantic layer. And the consumer stops being any agent and becomes a set of roles: a lead, analysts, a reviewer. Run the question one more time, and the answer is consensus by design. Here's how. [Delivery] The SAME ANSWER? tag that has sat on the canvas since 2026 comes down on click 2, when the loop turns on. Let the room see it go.

The task force on Tuesday’s deploy

Three analysts in parallel, one reviewer, one human call,
written back once and reused.

Agreement is a step in the loop, not a hope at the end

Lead agent

Frames the question, splits it three ways.

1

Analyst

Latency by endpoint around the deploy.

Analyst

Cart abandonment and ticket volume, same window.

Analyst

Affected accounts and their plan tier.

2

Reviewer agent

Same checkout latency? Same affected?

3 sent back: a different definition of affected 4

Human

Makes the one judgment call: what noticed means.

5

Semantic layer

The call, written once.

6

1the lead splits the question

2three notebooks go to review

3one is sent back

4the judgment call goes to a human

5written into the semantic layer

6every next question starts from it

Without the dashed box, this is three agents producing three numbers faster.

Here's the task force on Tuesday's deploy. A lead agent takes the question and splits it: one analyst measures latency by endpoint around the deploy time, one measures cart abandonment and ticket volume for the same window, one pulls the affected accounts and their plan tier. A reviewer agent checks that all three used the same definition of checkout latency and the same definition of affected, and sends one back when it didn't. A human signs off on the one judgment call a human should make: what noticed means for this incident. That call is written into the semantic layer once. The next time anyone asks anything involving customer impact, every agent starts from it. That's the loop: agree on definitions, execute in parallel, review against those definitions, write the agreements back. Consensus stops being something we hope for at the end and becomes a step in the middle. This only works because of Act 2. Three agents can be reconciled precisely because there's a shared place where definitions live. Without the dashed box, the task force is three agents producing three numbers faster. [Delivery] Never cut this slide. Trace path 3 with your hand: the send-back is the moment consensus becomes a step.

Next · Consensus by default

Every agent reads the same committed version,
so parallel work adds up to one answer.

Every read resolves to a committed version · every result names the version it used

Analyst 1latency by endpoint

Analyst 2abandonment, tickets

Analyst 3affected accounts

Data layerone committed history

v41 Tuesday’s data reads v41 1 v42 late tickets land reads v42 reads v42 2 commits the fix def v8 internal accounts excluded 3 × on def v7: rejected re-reads def v8 v43 affected-accounts table reads v43 reads v43

notebook · v43 · def v8

4

One answerv43 · def v8

Without the guarantee

1readers see half the batch

2the fix stays in one notebook

3the last writer wins

4three numbers for the reviewer to catch

A test tells you agents diverged. The data layer keeps them from diverging.

On the last slide the reviewer caught a divergence after the work was done. That's worth having, but it isn't the objective. The objective is a data layer every agent can trust, so there's little left to catch. Same three analysts, and the data changes under them. A late batch of Tuesday's tickets lands mid-run. It commits as one new version, so every agent sees all of it or none of it. Analyst two finds internal test accounts inflating affected customers. The fix commits once, as a new definition, and every agent's next read uses it. Analyst three tries to write a table built on the old definition. The layer rejects it, and it re-reads and commits on the new one. Every notebook names the version it used, so the reviewer checks one line. Iceberg already gives tables this: atomic commits in one linear history. Making definitions and corrections commits too is what we build. A test tells you agents diverged. The data layer keeps them from diverging. [Delivery] This is the design target, not what's built today. One click per numbered event; the coral row underneath is what each one looks like without the guarantee.

Source — I. Macomber, the last line of the post (2026)

Back to 2027

“…your CPO will learn about your team’s product launch through a subagent, deployed by an agent…”

The job of the post-AI data scientist is to make sure the meaning survives the edit.Ian Macomber · 2026

Three asks

Put agreement in the data layer,
and keep testing that it holds.

01 · every team

Write down what your data means

A page per team: what the tables mean, what the metrics mean, the known traps. It becomes the first version of our shared definitions.

Agents can only agree on what’s written down.

02 · the platform

Make every change a commit

Data, definitions and corrections land as one new version that every agent reads next. Every result records the version it used.

Parallel work adds up instead of drifting apart.

03 · every release

Keep the fan-out as the proof

The five metrics leadership reads weekly, asked through every interface we have. More than one number means a guarantee has a crack.

The proof, not the guarantee.

Agents shouldn’t have to negotiate the truth. They should be able to read it.

Appendix · Questions

Likely questions, short answers.

Why not a warehouse?

We’d get the 2016 boxes with a vendor’s opinion attached to both. Iceberg plus DuckDB gives the same separation with no lock-in: requirement three.

Why DuckDB, not Spark?

One engine per agent, no cluster to schedule, no shared failure. At our data sizes in-process is faster and simpler. If one table outgrows one container, that’s the day to revisit.

Why Marimo, not Jupyter?

Jupyter notebooks are JSON with hidden execution order. Marimo files are plain Python that run the same way every time, so agents can read them and people can review them.

Isn’t a semantic layer just docs?

Partly, and that’s the point: written-down definitions are the infrastructure. The other part is the evals that check agents against them.

Why not a reviewer agent?

A reviewer catches divergence after the work is done, and the work gets redone. The data layer stops it at the read and the write. Keep the reviewer for judgment calls.

When are we done?

When every read and write goes through committed versions and shared definitions, and the fan-out keeps returning one answer after we change something.

Appendix · How each era kept agreement

How each era kept everyone on the same numbers.

EraHow agreement was keptWhere it cracks

~2013

By bottleneck. The DBA ran every query, so there was one reading of the data.

Nobody else could ask. Questions waited in a queue.

~2016

By one person. A data scientist’s notebook held the definition and the analysis.

Nobody else could rerun it, and the next person defined it differently.

~2020

By scarcity. Metrics in dbt, dashboards in folders that matched the org chart.

Anyone who skipped the docs, person or tool, could diverge unnoticed.

2026

By testing. Ask every interface, count the answers, fix the context.

A test finds divergence after it happens, one question at a time.

Today

By working alone. One agent is consistent with itself.

A second agent reads its own version and its own definition.

Next

By construction. Every read resolves to a committed version and a shared definition.

Only the judgment calls, and those stay with a human.

Appendix · Where divergence gets in

Six cracks that parallel agents open, and what closes each.

CrackWithout a guaranteeWith the data layer

Stale read

An agent works from a copy it pulled an hour ago.

Every read resolves to the latest committed version.

Half-written table

A reader sees part of another agent’s write.

Commits are all or nothing. Readers never see a partial write.

Concurrent writes

The last writer silently wins, and one agent’s work vanishes.

A write built on an old version is rejected, then re-read and retried.

Forked definition

Each agent keeps its own “affected customer”.

One versioned definition. Changing it is a commit.

A fix that stays put

The correction lives in one notebook.

The correction commits once, and every agent’s next read uses it.

An orphan number

Nobody can say which data produced it.

Every result records the version and definition it used.

Analysis is cheap.Agreement isn’t.

Ian Macomber’s post tells fifteen years of data infrastructurein four diagrams.

In 2027 your CPO reads the dashboardwithout ever opening it.

One question, five sources, and one dashed line.

In 2013 you could only ask what fit on one server.

Storage splits from compute, and all the data lands in one place.

Data comes in with a click, and the answer goes back into the product.

Each era removed a constraint onwho could ask what of which data.

An agent harness sits between all the data and every interface.

Ask the same question through five interfacesand five numbers come back.

Consensus sounds vague.It can be engineered, asserted and measured.

Four things the stack must be.We’ll check each one against ours.

Same canvas, our names on the boxes, and a status on each.

Iceberg is the binding, the catalog is the index,and DuckDB is the reader.

One agent answers it end to end.Two agents disagree, and nothing notices.

Fill the dashed box, turn on the loop, and many agents work on one truth.

Three analysts in parallel, one reviewer, one human call,written back once and reused.

Every agent reads the same committed version,so parallel work adds up to one answer.

Put agreement in the data layer,and keep testing that it holds.

Write down what your data means

Make every change a commit

Keep the fan-out as the proof

Likely questions, short answers.

How each era kept everyone on the same numbers.

Six cracks that parallel agents open, and what closes each.

Analysis is cheap.
Agreement isn’t.

Ian Macomber’s post tells fifteen years of data infrastructure
in four diagrams.

In 2027 your CPO reads the dashboard
without ever opening it.

Each era removed a constraint on
who could ask what of which data.

Ask the same question through five interfaces
and five numbers come back.

Consensus sounds vague.
It can be engineered, asserted and measured.

Four things the stack must be.
We’ll check each one against ours.

Iceberg is the binding, the catalog is the index,
and DuckDB is the reader.

One agent answers it end to end.
Two agents disagree, and nothing notices.

Three analysts in parallel, one reviewer, one human call,
written back once and reused.

Every agent reads the same committed version,
so parallel work adds up to one answer.

Put agreement in the data layer,
and keep testing that it holds.