Skip to content
  1. Home
  2. Blog

Where Robot Training Data Comes From in 2026: Teleop, Sim, Video and Games

By the Metagaming team · · 8 min read

Robot training data is the set of recorded examples a robot policy learns from: what was observed, what action was taken and when, usually stored as time-stamped episodes. In 2026 it comes from five places: teleoperation, simulation, human video, games and logs from deployed robots. This guide compares them for ML and robotics teams deciding where their next dataset should come from.

Unlike text or images, this data can't be scraped. As Figure put it when it launched its human video dataset, "The data needed to scale a truly general purpose robot doesn't exist on the internet - it has to come from the real world." Every episode has to be generated by a robot, a simulator or a person, and each source trades cost, realism and scale differently.

What counts as robot training data?

An episode of robot training data pairs observations with actions over time. Observations are camera frames, joint positions, forces or game and simulator state. Actions are the commands the demonstrator or robot sent. Both carry timestamps, so a model can learn which action followed which state.

Most teams also store a task description, whether the episode succeeded, and metadata about the robot, scene and demonstrator. That structure is what lets data from different sources be pooled. The Open X-Embodiment project combined data from 22 robot types across 21 institutions, covering 527 skills, and reported that a model trained on the pool showed positive transfer: it improved on several robots by learning from experience collected on others.

Fig. 01Five sources of robot training data
  • Teleoperation
    Gives
    Actions in the robot's own space
    Watch for
    Slow, needs the robot and operators
  • Simulation
    Gives
    Scale and perfect labels
    Watch for
    The gap to real physics
  • Human video
    Gives
    Huge variety of real tasks
    Watch for
    No action labels, human bodies
  • Games
    Gives
    Skilled control, exact inputs and state
    Watch for
    Game physics aren't robot physics
  • Deployment logs
    Gives
    The real operating conditions
    Watch for
    Mostly routine and unlabeled

Most robot models in 2026 train on a mix: broad sources for general behavior, data on the target robot for its body.

Teleoperation: the most direct data, and the slowest

Teleoperation gives the most directly usable robot training data, because a person drives the actual robot and every action is already in the robot's own action space. The cost is time, hardware and people.

A small amount of good teleop data can go a long way. In the ALOHA work by Zhao, Kumar, Levine and Finn, a low-cost bimanual robot learned six fine-grained tasks, such as opening a translucent condiment cup and slotting a battery, at 80 to 90% success from only about 10 minutes of demonstrations. At scale, it gets expensive. The DROID dataset holds 76,000 demonstration trajectories, or 350 hours of interaction, across 564 scenes and 84 tasks. It took 50 data collectors across North America, Asia and Europe 12 months to gather.

Teleop also needs the robot to be where the operator is, or a reliable remote link. Most teleoperation jobs today are on-site, in a lab or a collection facility. Related methods lower the hardware barrier: kinesthetic teaching, where a person moves the arm by hand, and hand-held grippers like the Universal Manipulation Interface, which records people doing tasks in the wild and transfers those skills to deployable robot policies.

Simulation and synthetic data: cheap scale, with a reality gap

Simulation produces robot training data at a scale no physical fleet can match, with perfect labels, but a policy trained only in sim has to cross the gap to real physics and real cameras.

The standard tool for that gap is domain randomization: vary textures, lighting and physics so much that the real world looks like one more variation. In the original 2017 paper, an object detector trained only on simulated images with random textures located real objects to within about 1.5 cm, which the authors describe as the first transfer of a network trained only on simulated RGB images to real robotic control. Contact-rich manipulation is harder. Friction, deformable objects and tight tolerances are difficult to simulate well, which is why vendor guides such as Label Studio's 2026 overview treat sim and physical data as complements rather than substitutes.

Simulation is only as good as its demonstrations, too. Scripted motions are easy to generate but rarely look like skilled human behavior, so many teams still need people inside the simulator.

Human video: abundant, but without action labels

Video of people doing tasks is the largest pool of robot training data available, but it shows what happened, not the commands a robot would need to repeat it.

Two approaches are closing that gap. The first is learning action labels from a small labeled set. OpenAI's Video PreTraining (VPT) trained an inverse dynamics model on a small amount of labeled Minecraft play, used it to label a large pool of unlabeled online gameplay video, and, after fine-tuning, produced the first computer agents to craft diamond tools. The second is collecting purpose-made video at scale. Figure's Project Go-Big reported in September 2025 that its robots learned to navigate cluttered real-world spaces "after training exclusively on egocentric human video," with no robot demonstrations. In August 2026 Figure introduced Index, a dataset recorded by paid contributors through a phone app; Figure says it has passed 16 million uploaded videos.

The open questions with video are the embodiment gap (a human hand is not a gripper) and quality control at volume. Figure describes a pipeline of automated filtering, human fraud review, deduplication, rebalancing and annotation before any video is used.

Games: skilled human control with exact state and inputs

Games and game-like simulators produce robot training data where every input, timestamp and state change is recorded exactly, and where skilled people already spend hours performing hard control tasks.

Researchers have used games this way for a decade. In "Playing for Data" (ECCV 2016), Richter and colleagues extracted pixel-level labels for 25,000 images from a commercial video game. Models trained on those images plus one third of a real street-scene dataset (CamVid) beat models trained on all of CamVid. VPT learned from people playing Minecraft with a keyboard and mouse at 20 Hz. DeepMind's SIMA trains agents to follow language instructions across research environments and commercial video games, acting through keyboard and mouse like a human player.

For robotics, game data fits best where the hard part is perception and decision-making under time pressure: driving, flying, navigating, avoiding obstacles, packing and placing. Games also make good evaluation environments, because the same scenario can be replayed for every demonstrator. The limit is physics. Game physics are not robot physics, so game data works best for pretraining, behavior priors and evaluation, alongside some data on the target robot. We cover the human side in why skilled gamers make good robot demonstrators.

This is the source Metagaming works in. We collect human demonstrations in games and simulators, gameplay input data, evaluation labels and custom training games built around a lab's task, from vetted adult trainers. Teleoperation is coming soon. See robotics training data from skilled humans for what's open for pilots now.

Deployment logs: the real distribution, mostly unlabeled

Once robots are working, their own logs become robot training data that matches real operating conditions better than anything else, but most of it is routine, unlabeled and noisy.

The valuable parts are the rare ones: failures, near misses and moments where a human operator stepped in. Finding and labeling those is an annotation problem more than a collection problem, which is why fleet operators invest in triage, review queues and evaluation labels on robot attempts.

How labs combine sources in 2026

Most serious robot models now train on a mix of sources rather than one. Broad data teaches general behavior; data on the target robot teaches its body.

NVIDIA's GR00T N1 humanoid foundation model, for example, was trained on "a heterogeneous mixture of real-robot trajectories, human videos, and synthetically generated datasets." A common pattern is to pretrain on large, cheaper sources (video, simulation, games), then fine-tune on a smaller set of teleoperated episodes for the specific robot and task, and evaluate on held-out scenarios.

Whichever sources you use, ask the same questions of any supplier:

  • What is in one episode? Which observations, which actions, at what rate, and how they're synchronized.
  • Who made it? How demonstrators are vetted and whether skill is measured or self-reported.
  • How was it checked? Written acceptance rules per task, and what happens to episodes that fail them.
  • Can you use it? Consent records for every person in the data, and clear ownership and exclusivity terms.
  • Will it load? A delivery format that fits your training pipeline, agreed before collection starts.

The cheapest way to answer those questions is a small pilot. Our step-by-step guide to running a robot training data pilot covers the brief, mission design, QA and delivery.

FAQ

What is the best source of robot training data?

There isn't one. Teleoperation on the target robot is the most directly usable, simulation and video give scale, games give skilled human control with exact inputs and state, and deployment logs show real conditions. Most teams mix them.

Can a robot learn from video alone?

For some skills, yes. Figure reported that its humanoid learned navigation from egocentric human video with no robot demonstrations. Fine manipulation usually still needs some data on the robot itself.

Is simulation data enough to train a real robot?

Sometimes for perception and navigation, especially with domain randomization. For contact-rich manipulation, most teams still add real-world data.

How is game data different from simulation data?

Often it isn't, technically: many games are simulators. The difference is usually who's at the controls. Game data comes from skilled people playing, so it captures human decisions and control under pressure, not scripted motion.

Sources

Building a robot that needs skilled human data? Request a dataset or pilot and tell us the robot, the task and the data type.