How to Build a Data Pipeline With an Agent Swarm

They talk and negotiate.

Agents think like humans. The longer they think, the better their responses.

Imagine you’re taking the GRE or the GMAT and you only had 30 minutes to finish all the sections. You wouldn’t fare well. With three hours, you would ace the exam. But we can’t let our agent mull things over for three hours. The latency is not practical.

So we divide and conquer. Give each section of the exam to a different agent and let each one use the full 30 minutes (not saying we should let AI Agents take our SATs and GREs now). That’s the exact concept of distributed processing we use in Apache Spark, now applied to AI agents. That’s an agent swarm, and within Claude Code, it’s called an agent team.

In the previous episode, we built an end-to-end data pipeline using a single agent. This time, the same task, from data ingestion all the way to building data marts, but using a team of agents.

The setup

A typical dbt project, with context in three files: CONVENTIONS.md for business rules, STANDARDS for data engineering standards, and REQUIREMENTS for the data mart.

You all heard of the OpenAI hacking Hugging Face incident. There, the swarm of agents unintentionally built a message board where they exchanged messages. Claude Code lets you have a message board on purpose.

Agent teams are still experimental, so they’re off by default, and Claude uses tmux to start each agent in a different terminal. The prompt asks the same thing as last time, but this time it tells it to use an agent team, create a plan with tasks, and exchange messages on a message board.

The team and the plan

The parent agent builds the whole plan first, then spawns four teammates, one each for a data source: ERP, e-commerce, CRM, and POS. A fifth agent, for the data marts, stays on standby until all four source pipelines are built to staging.

Six agents in total, each with defined task ownership. The lead owns the project YAML, profiles, macros, and reports. Each teammate gets a source. The task list is divided into 10 tasks, each assigned to a teammate.

We typically use fancy project management tools and Gantt charts, etc. But for agents that work in a terminal and that read and think in text, project planning via markdown files is the best possible way.

The message board

I’m glad the agents decided to communicate in English, and not some newly invented language, or in binary, or in tokens.

The ECOM teammate proposes a few things to the CRM teammate, as points one to six. The CRM agent responds: it agrees on points one, three, four, five, and six, but it has its own opinion on point two and pushes back. Finally, the ECOM agent concedes, saying, “Agreed on point number two; we’ll do it as you’re proposing it.”

It's brilliant to watch agents communicate as people do. They propose things; they push back; they collaborate and negotiate. It’s the first time I’ve seen a swarm of agents communicate, and it’s beautiful! I am impressed.

I’m just glad I’m alive today watching agents communicate.

The results

All tasks done. Eight sources, five intermediate models, six marts, and 149 tests, in 11 minutes and 7 seconds. A single agent took about 20 minutes end-to-end. Is this of any consequence? That’s yet to be seen.

The sanity check: 2.19 million in revenue from the source files, and the data mart matches to the cent.

To what use?

One agent is already bad, as in bad for humans. I started off this series asking, am I still a data engineer if Claude does all the work? Now it’s six teammates doing the job. So am I still relevant?

One agent or a swarm: they still don’t understand business context unless I provide it. That business experience, those years of building data pipelines, is still relevant. Even then, someone always has to spot-check things to make sure they’re not completely haywire.

Agent swarms can do things faster and more efficiently, debatable. Maybe it’ll cost you more in tokens. However, here are a few use cases from my career where I can see myself making use of a team of agents:

1. Migrations. More often than not, they are time-bound. We can’t have a migration going on for three whole years. Business depends on the new system going live. The agent can get us ~ 60 to 80% code completion and accuracy, so our human teammates preserve their cognitive bandwidth for mission-critical work and don’t burn out.

2. Niche skills. I once had to migrate 10,000 PL/SQL files, and I couldn’t find anyone who graduated in the last 10 years who even understood PL/SQL. An agent swarm can help you fill that talent gap.

3. Time to value. Any mission-critical project where the enterprise is losing money because we can’t deliver the data, the report, or the feature soon enough.

I’m not saying agent teams can replace humans. Not yet. And I’m not saying they’re more economical or more efficient than a team of data engineers. All I’m saying is this feature is available; I could find a few use cases for it, and I was excited to see the agents in action.

Let me know in the comments what you think of agent teammates, and whether you are already using agent swarms in your modern data stack.