Modern robotics is no longer about mathematical models or rule-based control, sensing, planning, and acting. It increasingly relies on learning patterns directly from data rather than requiring explicit mathematical models.
So it’s not incorrect to say that modern robotics is as much about manipulation, locomotion, and whole-body control as it is about collecting, storing, and manipulating a large corpus of multimodal, time-series data. Thus, I can confidently say that modern robotics is a data engineering problem.
When I first joined Databricks back in the fall of 2017, our motto was “Make Big Data Simple.” It was even our WiFi password. Fast forward almost a decade, and Databricks and the ensuing cloud-based distributed data-processing technologies have indeed solved almost all enterprise data problems. So where is the next frontier of the world’s toughest data problems?
I think it’s robotics. And hear me out why.
Classical robotics works beautifully in a factory, where the same part shows up at the same place tens of thousands of times. But it falls apart at home, where neither you nor your kids put the same item in the same place twice. So modern robotics has learned a new approach. Instead of writing down the rules, you show the robot what to do repeatedly, and it learns the pattern.
Over the past decade, as data-driven robotics techniques have been adopted more widely, robotics data has flooded in. Hugging Face hosts about 80,000 LeRobot datasets, up from just 1,100 at the end of 2024. The largest academic robotics datasets, like AgiBot World, have crossed a million trajectories from over 100 robots. And that’s before you count all the teleoperation logs, fleet telemetry, and other metadata that streams continuously from a fleet of robots. So robotics data is indeed big data, maybe even bigger.
However, the challenges robotics companies and fleet managers face are entirely different from those of enterprise data, the web text large language models are trained on, or even industrial IoT data. Physical telemetry is highly multi-modal and high-dimensional. A single moment in a robot’s life contains camera frames, sensorimotor data from all the joints, the commands it was issued, and a timestamp for each, all captured by different sensors.
The primary challenge shifts from simple storage and processing to high-throughput streaming and downsampling. And if the streams drift out of sync, even by a single frame, nothing in the robot crashes. The robot just learns the wrong thing, quietly. That’s what makes robotics data the next frontier of data engineering.
The series
That’s exactly what I want to focus on in my next series on AI-assisted modern data engineering.
Over the next 13 weeks, I’ll build a robot data platform in public. Not a robot, but the engine behind it. Along the way, I’ll cover data engineering and the modern data stack, plus applied AI, using a cool robot arm as our prop and data source.
Zero robotics background is assumed or required here, because honestly, I’m just a humble data engineer, and I have no robotics background either.
Meet Yantra
To get started, I invested in a pair of robotic arms, a leader and a follower: the LeRobot SO-ARM101 Pro. I’ve named it Yantra, which means machine in Sanskrit.
The black one is the leader and the white one is the follower. The follower follows every move the leader makes, in real time.
This is how you teach a robot using data. Every time I move the leader to accomplish a task, the system records two things: what the robot saw and what it was told to do. That pair, repeated thousands of times, is your robot training data.
The follower has two cameras. One is mounted on the wrist, which gives it a point of view of what it’s doing. The other stands on the sidelines, observing the big picture. Two views, like our eyes and our hands: one for the big picture and one for the finer details. And the pair together is how the robot learns
But the arm isn’t today’s star. The data is.
Hello World, robotics edition
If you’re getting started with programming, or a new programming language, you would run a Hello World program, right? A dataset, or simulation, called PushT is the equivalent of Hello World in robotics.
As usual, everything is in the GitHub repo for this series, in the week 1 module. I cloned it with Claude Code, my favorite AI assistant, into ~/Projects/paida-wk01.
Then I downloaded the PushT LeRobot dataset from Hugging Face and opened its first episode in a visualization tool called Rerun:
uv run lerobot-dataset-viz --repo-id lerobot/pusht --episode-index 0The blue dot is a simulated robot, and its task is to push the T onto the T outline. In this episode, it does it perfectly. But this is just one episode. A human did this 206 times, and a robot learns from all those attempts. Each attempt is called an episode.
An episode has many frames, and a frame is just a moment in time. At any given moment, a few things are recorded. The state is the robot’s state: its x and y coordinates. In Rerun, the 0 graph is x and the 1 graph is y, so at any moment I have a timestamp, an x, and a y. Then there’s the action: what the robot was told to do next to accomplish the task, also recorded as an x and a y.
If you look at the state and action graphs, they’re almost identical. But at certain timestamps, they deviate slightly because of simulated real-world physics. When a robot pushes on an object, the object also pushes back. So we issue a command, but the robot may not accomplish it exactly. That’s how the physical world works.
In Rerun, everything comes together seamlessly. When I press play or click a point, everything moves together: the state, the action, and the actual camera. But in reality, different sensors capture all of these. In data engineering terms, they’re disparate data sources, and stitching them together without missing a frame is where the skill and the challenge go.
Inside a LeRobot dataset
So let’s open up the dataset and see what it actually looks like. It’s stored in a format called the LeRobot dataset, developed by Hugging Face, and it’s structured into three distinct parts:
Metadata:
meta/info.jsonandmeta/stats.json. What the structure of the data is, what it’s about, what format it’s in, and where things are stored.Tabular data: all the machine states, the joint states, and the sensorimotor data, stored in Parquet under
data/.Visual data: the actual camera frames, stored as MP4 under
videos/.
The metadata says the robot type is unknown because it ran in simulation. There are 206 episodes in total, 206 different experiments, if I may call them that, and 25,650 frames recorded at 10 frames per second, all accomplishing a single task: pushing the T object onto the T outline.
Then the features, or what columns: the observation image, the observation state, the action, the episode index, the frame index, the timestamp, the reward, and the success flag. These are the things we saw stitched together in Rerun to get a single unified view of the robot’s state at each moment.
You could use Apache or some other framework to query the parquet data, but I’m using DuckDB because it’s lightweight.
DESCRIBE SELECT * FROM 'data/*/*.parquet';It has all those features: state, action, episode index, frame index, timestamp, reward, success. But something is conspicuously missing: the observation image. Which makes sense, right? We typically don’t store MP4 data in Parquet files. That’s why it’s stored separately.
This query shows each episode’s first and last timestamp:
SELECT episode_index,
MIN(timestamp) AS first_ts,
MAX(timestamp) AS last_ts,
COUNT(*) AS frames
FROM 'data/*/*.parquet'
GROUP BY episode_index
ORDER BY episode_index
LIMIT 5;The first timestamp is always 0.0. It’s time since the episode started, not time in the video.
The videos, meanwhile, are one big uber MP4 file with many episodes packed into it. The episode metadata tells me, for each episode, where its clip starts and ends inside that file:
SELECT episode_index,
"videos/observation.image/from_timestamp" AS v_from,
"videos/observation.image/to_timestamp" AS v_to
FROM 'meta/episodes/*/*.parquet'
ORDER BY episode_index
LIMIT 5;The metadata is relational, helping to join the visual and tabular data together and giving the location of the clip for each frame:
SELECT d.episode_index,
d.frame_index,
d.timestamp,
e."videos/observation.image/from_timestamp" + d.timestamp AS video_position
FROM 'data/*/*.parquet' d
JOIN 'meta/episodes/*/*.parquet' e USING (episode_index)
WHERE d.episode_index = 1
LIMIT 5;PushT is just a small, simulated, and perfect dataset. A real robot training dataset has many more episodes and tasks, and it grows very fast.
Why it matters
I started off by saying that robotics is the next frontier of data engineering, and this is a glimpse of why. The data is complex, multimodal, and high-dimensional, and it comes from different sources that must be stitched together without missing a single frame.
If I have to use an analogy, it’s like recording and editing videos. You have multiple cameras and multiple clips, and everything has to be stitched together at exactly the same time, or it won’t be in sync. If your video editing is out of sync, it’s obvious, right? Your lips move, but the voice comes from elsewhere. But with a robot, nothing fails. Nothing is obvious. The robot just learns the wrong thing, and in real-world applications, it won’t be able to do its job. We won’t know until it’s too late.
So data engineering becomes very important in data-driven robot training.
Three takeaways
Modern robots learn from data, not from handwritten rules. That makes robotics a data problem.
A robot dataset has three parts: metadata, tabular data with numerical sensorimotor joint states, and camera frames in MP4.
The hardest part of robotics data isn’t storing it; it’s stitching it together so all the timestamps match, and the training isn’t misguided. And it all depends on a bit of arithmetic on a set of timestamps.
Where I am, and what’s next
So that’s where I’m starting. Two weeks in, I can take a robot dataset apart down to the row and stitch it back together so the frames match. But I’m not going to pretend that I’m a roboticist, which I’m not. I’m just a data engineer delving into a new domain, exploring new data-engineering challenges through robotics, with you as my witness.
Next week, I’ll go from a simulated dataset to a real robot dataset that the community recorded on Hugging Face, and take it apart. That’s where things get complex. In simulated data, things are perfect: once you join, everything matches seamlessly. But in real-world robotics, things aren’t perfect, because the physical world isn’t. So let’s see what challenges I face as I take apart and stitch together a real-world robot dataset. See you then.
Sources: Hugging Face LeRobot datasets · TechTimes on LeRobot Hub growth · AgiBot World (The Robot Report)