Skip to content

For AI labs · Data formats and specs

Robot training data formats and specs

Each Metagaming session records a trainer's inputs with timestamps, the game or simulation state your brief lists, the trainer's level and skill scores, and the consent version they agreed to. We deliver it in the format your pipeline reads, agreed in the brief: LeRobot, RLDS, HDF5, MCAP and JSON or CSV are available on request.

Request a dataset or pilot How we collect data

What does each session capture?

A session holds what the trainer did, when they did it and the task state they acted on, plus who did it (by trainer name) and the consent behind it. Nothing outside the game or task is recorded.

Captured per session

  • Inputs: keys, clicks, taps, controller and wheel movements
  • A timestamp for every input
  • The game or simulation state that your task brief lists
  • For evaluation labels: the judgment itself, such as what went right, what went wrong, or which of two runs was better
  • The trainer's trainer name, level and per-skill scores
  • The consent version the trainer agreed to

Never captured

  • Webcam or faces
  • Microphone or voices
  • Real names or legal identity
  • Game logins or passwords
  • Anything outside the game or task

Skill scores cover six skills: speed, accuracy, packing, navigation, timing and steady hands. They come from our skill tests and from quality review on missions. Skills a trainer reports about themselves are kept, but labelled as claimed. The full rules are on how we collect data.

Which delivery formats can you get?

You get the data in the format your training pipeline already reads. The five below are available on request; the exact format, field mapping and file layout are agreed in the brief, before collection starts.

FormatWhat it isOften used for
LeRobotHugging Face's LeRobotDataset format for robot learning data. Version 3 stores states, actions and timestamps in Apache Parquet, camera video in MP4, and schema, frame rate and episode boundaries as JSON and Parquet metadata. LeRobot docsTeams training with the LeRobot library or sharing on the Hugging Face Hub
RLDSReinforcement Learning Datasets: each episode holds a sequence of steps, each step with fields such as observation, action and reward plus is_first and is_last flags, loaded through TensorFlow Datasets. RLDS on GitHubTensorFlow and JAX pipelines; the Open X-Embodiment datasets use the RLDS episode format
HDF5A self-describing file format from the HDF Group for large n-dimensional arrays, with metadata kept in the same file. HDF GroupTeams that want arrays they can slice directly, one file per episode or batch
MCAPAn open-source container format for multimodal log data: multiple channels of timestamped, pre-serialized messages. It is the default log format in ROS 2. mcap.devROS 2 stacks and log replay tools
JSON / CSVPlain-text formats any language can read: JSON (RFC 8259) for nested records, CSV (RFC 4180) for flat tables.Evaluation labels, small pilots and quick inspection

Need a custom schema or another container? Say so in the brief.

How are timing and sampling handled?

Every input carries the time it happened. The rate at which game or simulation state is sampled is set per task in the brief, because it depends on the game or sim.

A driving sim, a packing game and a custom training game expose state differently, so we confirm the rate, the timestamp clock, units and any resampling your loader needs before collection starts, and write them into the dataset card.

How is the data checked?

Every task is checked against written acceptance rules that you agree before collection starts. Work that fails review isn't delivered.

  • Rules per task. Each task has written acceptance rules, agreed with you in the brief.
  • Every task reviewed. Work that fails isn't delivered or paid, and the trainer is told why.
  • Levels earned on quality. Trainers reach Skilled after 50 tasks at 90%+ quality, and Expert after 250 tasks at 95%+ quality plus a top 10% skill test score. Genius is invite only: the top 1%. Level goals are starting targets and may change.
  • Matched trainers. Missions go to accepted trainers with the right level, skill scores and industry.
  • Only the named game. Our desktop recorder records only the games a mission names, and we list a game for a mission only after checking that recording it is safe with that game's anti-cheat.
  • Consent on every session. Every delivered session is stamped with the consent version the trainer agreed to. All trainers are 18+.

Example dataset card schema

This is an example schema, showing the fields a Metagaming dataset card can carry. It doesn't describe an existing dataset, and your final card is agreed in the brief.

FieldWhat it holds
dataset_nameThe name agreed for the dataset
versionThe dataset version, updated on each delivery
data_typeHuman demonstrations, gameplay input data, evaluation labels or custom training game
task_descriptionThe task in plain words, as written in the brief
environmentThe game, simulation or custom training game used
delivery_formatLeRobot, RLDS, HDF5, MCAP, JSON or CSV, as agreed
action_fieldsThe inputs recorded, with names, types and units
state_fieldsThe game or simulation state recorded, with names, types and units
time_baseThe clock and units used for timestamps, and the state sampling rate
label_schemaFor evaluation labels: the questions asked and the allowed answers
episode_countHow many episodes are in the delivery
total_durationThe total recorded time in the delivery
trainer_fieldsTrainer name, level and per-skill scores; never a real name
qa_rulesThe acceptance rules each task was checked against
consent_versionThe consent version stamped on each session
not_capturedWhat the data never contains: webcam, microphone, identity, anything outside the task
known_limitationsGaps to keep in mind, such as differences between game or sim physics and your robot
license_termsUsage rights and any exclusivity, as set in the contract

Data formats and specs: questions from labs

Which delivery formats do you support?

LeRobot, RLDS, HDF5, MCAP and JSON or CSV are available on request. The exact format, field names and file layout are agreed in the brief, to fit your training pipeline.

What sampling rate do you record at?

Every input is recorded with the time it happened. The rate at which game or simulation state is sampled depends on the game or sim, so it is set per task in the brief and confirmed before collection starts.

Do you record video of the trainer?

No. We never record the webcam, the microphone or anything outside the game or task. If your model needs rendered frames from the game or simulation itself, say so in the brief and we'll confirm whether that task can include them.

Can the data follow our own schema?

Yes. Field names, units and file layout are agreed in the brief, so the delivery fits the loader you already use.

What do we see about each trainer?

A trainer name, their level and their per-skill scores, never a real name or legal identity. Every delivered session also carries the consent version the trainer agreed to. Read how we collect data.

We're looking for the most skilled people to train the future.

Think you're one of them? Prove it.

Apply free

Free to apply · 18+ · Every application is reviewed