How to Run a Robot Training Data Pilot: Brief, Mission, QA, Delivery
A robot training data pilot is a small first collection from a data supplier, run to check that the data fits your model and pipeline before you commit to a larger order. It runs in five steps: the brief, a custom mission, collection, QA and delivery. This guide covers what to decide at each step and how to judge the result.
It's written for ML and robotics engineers, research leads and data or procurement leads. For the bigger picture of where data comes from, read our guide to where robot training data comes from in 2026.
Why start with a pilot?
A pilot tests the whole path from task to training run on a small scale, so problems with the task design, the demonstrators, the QA rules or the format show up while they're cheap to fix.
Small datasets can be informative. In the ALOHA work, a robot learned six fine-grained manipulation tasks at 80 to 90% success from about 10 minutes of demonstrations. Imitation learning is also sensitive to data quality: the robomimic study lists "the dependence on the quality of the demonstrations" among its main lessons. A pilot is the cheapest way to find out whether a supplier's demonstrations help your model or add noise.
It also tests the parts that have nothing to do with learning: whether the files load, whether timestamps line up, whether every session has a consent record, and how the supplier handles rejected work.
- 01BriefRobot, task, data type, volume and timeline
- 02Custom missionMission or training game, plus written QA rules
- 03CollectionMatched to trainers by level, skill scores and industry
- 04QAEvery episode checked against the agreed rules
- 05DeliveryAgreed format, with each session's consent record
Step 1: Write the brief
The brief defines the robot, the task, the data type, the volume and the timeline, plus what a successful episode looks like. Everything later is checked against it.
A good brief answers these questions:
- Robot and embodiment. Which robot, or which class of robot (arm, mobile manipulator, humanoid, drone, vehicle), and its action space.
- Task. What the demonstrator should do, from which starting conditions, and what counts as success and as failure.
- Data type. Human demonstrations, raw input data, evaluation labels on your robot's attempts, or a mix.
- Observations and actions. Which signals you need in each episode, at what rate, and how they should be synchronized.
- Volume and variety. How many episodes, and how much variation in scenes, objects and starting states.
- Demonstrators. Any skill or background they need, such as fast reactions, flight experience or forklift experience.
- Format. The delivery format your training pipeline already reads.
- Evaluation plan. How you'll decide whether the pilot worked.
The last point is the one most often skipped. Decide the metric before collection starts, or the result will be argued about afterwards.
On our robotics training data page, the request form asks for robot type, data type, volume and timeline, and we follow up on the rest by email or on a call.
Step 2: Design the mission
The mission turns your task into something a person can perform repeatably, in a game, a simulator or a training game built for the task, with written acceptance rules agreed before anyone starts.
Mission design covers the environment, the starting states and how they vary, the controls, the episode length, and what gets recorded. It also covers the acceptance rules: the checks every episode must pass to be delivered. Write them down. "Reach the target without a collision, within the time limit, with no idle period over two seconds" can be checked. "Do it well" can't.
Two design decisions matter more than they look:
- Consistency versus variety. Research on data quality in imitation learning shows that more state diversity isn't always better; it helps when the actions stay consistent. Vary the scenes on purpose, and keep the way the task is done consistent.
- Failures and recoveries. Decide whether you want only clean successes, or also episodes where the demonstrator recovers from a mistake. Recoveries can help a policy that drifts, but only if they're labeled.
At Metagaming, we design the mission or training game around your task and agree the QA rules with you before collection starts. If your task doesn't map well to a game or simulator, the mission design step is where that should come out.
Step 3: Collect the data
Collection matches the mission to demonstrators with the right skills and records each session against the brief.
Who demonstrates matters as much as how many. Ask how demonstrators are vetted and whether their skill is measured or self-reported. We cover the reasoning in why skilled gamers make good robot demonstrators. At Metagaming, missions go to accepted trainers matched by level, per-skill scores and industry, and every trainer is an adult who has agreed to how their data is used.
For a pilot, it's worth reviewing an early batch before the rest is collected. If the first episodes show a problem with the task design or the rules, fixing it then is far cheaper than after the whole pilot is in.
Step 4: QA every episode
QA checks every episode against the acceptance rules from step 2, and episodes that fail are not delivered.
A practical QA pass has two layers. Automated checks catch technical faults: missing or out-of-order timestamps, dropped frames, episodes that are too short or too long, values out of range, and duplicates. Human review catches what scripts miss: a demonstrator who technically reached the goal but in a way you don't want a robot to copy. Figure describes a similar approach for its human video dataset, combining automated quality filtering with human fraud review, deduplication and rebalancing.
At Metagaming, every task is checked against the agreed rules, and work that fails isn't delivered or paid. Ask any supplier for the rejection reasons along with the data. A pilot's rejection pattern tells you whether the rules, the mission or the demonstrators need to change.
Step 5: Delivery
Delivery means the data arrives in the format agreed in the brief, documented well enough that someone outside the project can use it, with a consent record for each session.
Formats are a decision for your pipeline. Two widely used options in robot learning are Hugging Face's LeRobotDataset, which stores states, actions and timestamps in Parquet, camera video in MP4 and episode boundaries in metadata, and RLDS, an ecosystem for recording and sharing sequential decision-making data in a standard, lossless format. Whatever the format, test-load the pilot delivery into your own training code on day one.
Ask for documentation with the data. Datasheets for Datasets (Gebru et al.) proposes that every dataset come with a document covering "its motivation, composition, collection process," and more. For a robot dataset, that means the brief, the mission, the acceptance rules, the QA results, the demonstrator metadata and the consent terms.
At Metagaming, you receive the data in the agreed format, and every delivered session carries its consent record: the exact wording and version the trainer agreed to. We capture inputs, timing and task state, never faces, voices or anything outside the task. See how we collect data.
How do you judge the pilot?
Judge the pilot against the metric you set in the brief, and on how the process went, then decide whether to scale, change or stop.
- Model impact. Train with and without the pilot data, or fine-tune on it, and compare on held-out scenarios you defined up front.
- Data quality. Spot-check episodes yourself. Look at consistency between demonstrators and at the rejection reasons.
- Fit with your pipeline. Did the data load without custom fixes? Were the signals and rates what you asked for?
- Process. Was the supplier clear about the timeline, quick to fix issues and open about what failed QA?
If the pilot works, the scale-up brief is mostly written: keep the mission and rules, change the volume and variety, and agree terms such as exclusivity in the contract.
Common pilot mistakes
- Starting collection before the success metric is agreed.
- Acceptance rules that can't be checked, such as "natural" or "smooth" with no definition.
- Asking for maximum variety and getting inconsistent demonstrations.
- Not test-loading the first delivery into the training pipeline.
- No record of who demonstrated, at what skill level, and with what consent.
FAQ
How big should a robot training data pilot be?
Big enough to train or fine-tune once and see an effect on your metric, and small enough to redo if the mission needs changing. At Metagaming, pilot size is agreed in the brief.
How long does a pilot take?
It depends on the volume and on how specialized the demonstrators need to be. We give you a timeline after the brief, before collection starts.
Who owns the pilot data?
Ownership and exclusivity should be in the contract before collection. At Metagaming, exclusivity is agreed per project.
Can a pilot include teleoperation?
At Metagaming, not yet. Teleoperation is coming soon, starting with trainers at Expert level and above. Pilots are open now for human demonstrations, gameplay input data, evaluation labels and custom training games.
Sources
- Zhao, Kumar, Levine and Finn, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023)
- Mandlekar et al., What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (CoRL 2021)
- Belkhale, Cui and Sadigh, Data Quality in Imitation Learning (2023)
- Figure, Introducing Index (2026)
- Hugging Face, LeRobotDataset v3.0 documentation
- Ramos et al., RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021)
- Gebru et al., Datasheets for Datasets (2018; CACM 2021)
Ready to test the fit? Request a dataset or pilot with your robot, task, data type, volume and timeline.