For AI labs · Data formats and specs
Robot training data formats and specs
Each Metagaming session records a trainer's inputs with timestamps, the game or simulation state your brief lists, the trainer's level and skill scores, and the consent version they agreed to. We deliver it in the format your pipeline reads, agreed in the brief: LeRobot, RLDS, HDF5, MCAP and JSON or CSV are available on request.
What does each session capture?
A session holds what the trainer did, when they did it and the task state they acted on, plus who did it (by trainer name) and the consent behind it. Nothing outside the game or task is recorded.
Captured per session
- Inputs: keys, clicks, taps, controller and wheel movements
- A timestamp for every input
- The game or simulation state that your task brief lists
- For evaluation labels: the judgment itself, such as what went right, what went wrong, or which of two runs was better
- The trainer's trainer name, level and per-skill scores
- The consent version the trainer agreed to
Never captured
- Webcam or faces
- Microphone or voices
- Real names or legal identity
- Game logins or passwords
- Anything outside the game or task
Skill scores cover six skills: speed, accuracy, packing, navigation, timing and steady hands. They come from our skill tests and from quality review on missions. Skills a trainer reports about themselves are kept, but labelled as claimed. The full rules are on how we collect data.
Which delivery formats can you get?
You get the data in the format your training pipeline already reads. The five below are available on request; the exact format, field mapping and file layout are agreed in the brief, before collection starts.
| Format | What it is | Often used for |
|---|---|---|
| LeRobot | Hugging Face's LeRobotDataset format for robot learning data. Version 3 stores states, actions and timestamps in Apache Parquet, camera video in MP4, and schema, frame rate and episode boundaries as JSON and Parquet metadata. LeRobot docs | Teams training with the LeRobot library or sharing on the Hugging Face Hub |
| RLDS | Reinforcement Learning Datasets: each episode holds a sequence of steps, each step with fields such as observation, action and reward plus is_first and is_last flags, loaded through TensorFlow Datasets. RLDS on GitHub | TensorFlow and JAX pipelines; the Open X-Embodiment datasets use the RLDS episode format |
| HDF5 | A self-describing file format from the HDF Group for large n-dimensional arrays, with metadata kept in the same file. HDF Group | Teams that want arrays they can slice directly, one file per episode or batch |
| MCAP | An open-source container format for multimodal log data: multiple channels of timestamped, pre-serialized messages. It is the default log format in ROS 2. mcap.dev | ROS 2 stacks and log replay tools |
| JSON / CSV | Plain-text formats any language can read: JSON (RFC 8259) for nested records, CSV (RFC 4180) for flat tables. | Evaluation labels, small pilots and quick inspection |
Need a custom schema or another container? Say so in the brief.
How are timing and sampling handled?
Every input carries the time it happened. The rate at which game or simulation state is sampled is set per task in the brief, because it depends on the game or sim.
A driving sim, a packing game and a custom training game expose state differently, so we confirm the rate, the timestamp clock, units and any resampling your loader needs before collection starts, and write them into the dataset card.
How is the data checked?
Every task is checked against written acceptance rules that you agree before collection starts. Work that fails review isn't delivered.
- Rules per task. Each task has written acceptance rules, agreed with you in the brief.
- Every task reviewed. Work that fails isn't delivered or paid, and the trainer is told why.
- Levels earned on quality. Trainers reach Skilled after 50 tasks at 90%+ quality, and Expert after 250 tasks at 95%+ quality plus a top 10% skill test score. Genius is invite only: the top 1%. Level goals are starting targets and may change.
- Matched trainers. Missions go to accepted trainers with the right level, skill scores and industry.
- Only the named game. Our desktop recorder records only the games a mission names, and we list a game for a mission only after checking that recording it is safe with that game's anti-cheat.
- Consent on every session. Every delivered session is stamped with the consent version the trainer agreed to. All trainers are 18+.
Example dataset card schema
This is an example schema, showing the fields a Metagaming dataset card can carry. It doesn't describe an existing dataset, and your final card is agreed in the brief.
| Field | What it holds |
|---|---|
| dataset_name | The name agreed for the dataset |
| version | The dataset version, updated on each delivery |
| data_type | Human demonstrations, gameplay input data, evaluation labels or custom training game |
| task_description | The task in plain words, as written in the brief |
| environment | The game, simulation or custom training game used |
| delivery_format | LeRobot, RLDS, HDF5, MCAP, JSON or CSV, as agreed |
| action_fields | The inputs recorded, with names, types and units |
| state_fields | The game or simulation state recorded, with names, types and units |
| time_base | The clock and units used for timestamps, and the state sampling rate |
| label_schema | For evaluation labels: the questions asked and the allowed answers |
| episode_count | How many episodes are in the delivery |
| total_duration | The total recorded time in the delivery |
| trainer_fields | Trainer name, level and per-skill scores; never a real name |
| qa_rules | The acceptance rules each task was checked against |
| consent_version | The consent version stamped on each session |
| not_captured | What the data never contains: webcam, microphone, identity, anything outside the task |
| known_limitations | Gaps to keep in mind, such as differences between game or sim physics and your robot |
| license_terms | Usage rights and any exclusivity, as set in the contract |
Data formats and specs: questions from labs
Which delivery formats do you support?
LeRobot, RLDS, HDF5, MCAP and JSON or CSV are available on request. The exact format, field names and file layout are agreed in the brief, to fit your training pipeline.
What sampling rate do you record at?
Every input is recorded with the time it happened. The rate at which game or simulation state is sampled depends on the game or sim, so it is set per task in the brief and confirmed before collection starts.
Do you record video of the trainer?
No. We never record the webcam, the microphone or anything outside the game or task. If your model needs rendered frames from the game or simulation itself, say so in the brief and we'll confirm whether that task can include them.
Can the data follow our own schema?
Yes. Field names, units and file layout are agreed in the brief, so the delivery fits the loader you already use.
What do we see about each trainer?
A trainer name, their level and their per-skill scores, never a real name or legal identity. Every delivered session also carries the consent version the trainer agreed to. Read how we collect data.
Request a dataset or pilot
Tell us your robot, task and the format your pipeline reads. Most projects start with a small pilot, so you can check the data fits your model before a larger order.
By industry: autonomous driving, drones, warehouse robotics, manipulation and humanoids.