Data Is the Training Recipe

Jinzhou Li 10 min read

Robot learning data used to sound like a collection problem: collect demonstrations, train a policy, and evaluate success. I now see it more as a training-recipe problem. Each source shifts the states a policy sees, the failures it exposes, and the corrections we can make. The harder decision is what role each source should play in the loop.

The old framing was data collection

In 2024, I talked with a friend who was building a data startup company. At that time, my mental model was simple: what kind of data does a robot need? I mostly thought about teleoperation: which tasks to collect, which initial states to cover, how many recoveries to include, and how to keep action labels clean.

I remember this was also shaped by the 2023 discussions around Diffusion Policy. Strong behavior cloning depended as much on the data recipe as on the policy class, including haptic teleoperation, good robot control, visuotactile sensing, and recoveries around likely failure states [Chi et al., 2023]. The lesson was that data is not a pile of successful trajectories. It is an interface between the human, the robot, and the failures we expect at deployment.

Data sources are changing

That framing still matters, but the data landscape already looks different. Robot data is no longer a single stream of teleoperated demonstrations on one platform. Cross-embodiment datasets pool experience across robots, tasks, scenes, and institutions [Open X-Embodiment Collaboration et al., 2023] [Khazatsky et al., 2024]. Synthetic pipelines expand small demonstration sets into varied trajectories [Mandlekar et al., 2023]. Simulation may first be an evaluation and iteration engine before it is a generator of trajectories [Genesis AI Team, 2026]. Human egocentric video offers a different signal: not robot actions directly, but priors about hands, objects, intent, and everyday procedures [Grauman et al., 2022]. Wearable physical-interaction data adds another branch. GEN-1 argues for pretraining from human physical activity data, then adapting to robot tasks with a small amount of robot data [Generalist Team, 2026].

These sources are complementary, but they do not automatically compose. Teleoperation can teach action grounding. Synthetic data can add coverage. Human video can teach task structure and affordances. Contact data can expose physical state that vision misses. The hard part is making them meet at the policy without simply adding another distribution gap.

The new bottleneck is the training recipe

In 2026, when I talked with him again in the Bay Area, the discussion had shifted from source discovery to recipe design: how much of each source, in what order, and for which capability? The difficult part is deciding the mixture, schedule, filtering, and loss weighting so that one source does not wash out another.

A plausible recipe looks more like curriculum design than "train on all available data": pretrain on broad but weakly aligned data, adapt on robot-specific demonstrations, add synthetic variation where coverage is missing, and finish with recovery data from the policy's own failures. The right schedule depends on the base policy, embodiment, task horizon, and modality.

Simulation changes the data loop

Genesis World 1.0 makes a useful distinction: simulation should first be an evaluation and iteration engine. Real robot experiments limit how fast we can compare policies, ablations, checkpoints, and data mixtures. A trustworthy closed-loop simulator turns part of that bottleneck from robot time into compute time [Genesis AI Team, 2026].

Evaluation can then drive collection. If simulation reveals brittleness under object variation, contact conditions, or camera shifts, the next dataset should target that weakness. In this role, simulation is less about replacing real data and more about deciding which real data is worth collecting.

This is also why evaluation should probably come before simulation-based data generation. If both training and evaluation depend on the same simulated distribution too early, improvement may only mean that the policy learned the simulator. A cleaner route is to build sim-real trust first, then use simulation more aggressively for post-training, RL, task generation, or synthetic data.

What kind of base model do we need?

A good base model for robotics cannot be judged only by offline imitation loss. It should be broad enough to understand common objects and task structure, stable enough to run for long horizons, and easy to steer when it is wrong. If a small amount of correction data, recovery data, force feedback, or tactile feedback cannot move the policy predictably, the base model is hard to use even if it looks strong on a benchmark.

I care about four properties: broad priors, correctable actions, adaptation without forgetting, and interpretable failures. GEN-1 is interesting in this light because its claim is a large physical-interaction pretraining base followed by a small amount of robot task data [Generalist Team, 2026]. In this view, behavior cloning may only be a warm-up. It gets the robot into the task distribution; post-training and recovery data make the policy reliable.

Modality priority should follow action ambiguity

Sensor choice should be driven by the ambiguity that remains for the action. For contact-rich manipulation, my rough priority is:

  1. Robot state and action history, because the controller needs embodiment state, timing, and recent motion.
  2. Vision, because it gives global object, scene, and geometry information.
  3. Task context or language, when the robot needs to bind a goal, preference, or long-horizon instruction.
  4. Force-torque, when the same visual state can correspond to different contact loads, constraints, or compliance.
  5. Tactile sensing, when local contact state determines the next action.

This order is a diagnostic, not a rule. Add a modality only when the data can support the variation it exposes. If tactile data is added before the dataset covers different force backgrounds, the policy may inherit a harder out-of-distribution problem. If force or tactile data enters during targeted post-training, after the base policy already handles the visual task distribution, it may be much more useful because it explains the remaining failures.

Step back: what is the problem itself?

Leaving aside industry roadmaps, the core object is the data loop itself. Each added source changes what states the policy understands, which failures become visible, what corrections are possible, and which modality becomes useful rather than harmful. The evaluation target should not be who has more data; it should be whether a source applies the right pressure to the policy.

Recent systems are useful as hints, not final answers. GEN-1 suggests that broad physical-interaction pretraining can make a small amount of robot task data more valuable [Generalist Team, 2026]. Sunday's ACT-2 preview suggests that targeted post-training and recovery collection can turn a base model into a more reliable deployed system [Sunday Robotics, 2026]. Genesis argues that simulation should first make evaluation and iteration faster, before we trust it as a training data generator [Genesis AI Team, 2026]. Together they point to the same research object: the loop that collects data, trains a policy, exposes failure, and decides what signal should enter next.

The structural questions matter more than any single dataset. What should be learned before the robot ever touches the task? What must be learned on the robot because embodiment matters? When should a new modality enter, and when does it only create a new distribution shift? Which failures call for more data, and which failures mean the representation or action space is wrong?

Episodes are not the only unit anymore

Another shift is the unit of accounting. Post-training data is no longer always counted by episodes. People increasingly talk in hours: robot-hours, human-hours, and effective data-hours. But one hour of robot operation is not one hour of useful supervision. A mature data collector can produce better corrections because they know how to expose failures, reset quickly, avoid ambiguous labels, and demonstrate recoveries the policy can learn from.

This is why the rise of data companies makes sense. If the bottleneck is effective data production, the product is an operation that turns human time, robot time, task design, sensor calibration, and failure mining into usable training signal.

Dexterous-hand data is still an open opportunity

Dexterous hands sharpen the opportunity. Broad manipulation datasets are valuable, but they still leave a large open space around multi-finger contact: regrasping, in-hand adjustment, recovery, hand-object occlusion, and morphology-specific failure. This gap is not just a missing dataset; it asks for a new data unit, not simply more successful episodes.

For dexterous hands, useful data may be organized around transitions: approach, first contact, force buildup, contact change, regrasp, recovery, and release. The key split is between general hand priors and behaviors tied to a specific sensor, hand design, or controller. This is where tactile and force signals feel important, but only when they explain failures that vision and proprioception cannot.

When should contact-rich data be used?

If we could scale tactile data with real contact information, what would change? My guess is that scale alone may not show a clear short-term effect. If a task is already solvable from vision and proprioception, adding large amounts of tactile data may only increase observation complexity. The value appears when touch changes the action choice by resolving what other observations cannot: local pressure, slip, contact geometry, deformation, or hidden force background.

The choice is not simply whether to use force explicitly or implicitly. Explicit force or torque feedback is useful for contact constraints, compliance, safety, and residual correction. An implicit tactile or contact representation may be enough when the signal mainly disambiguates the physical state behind the image. The design choice is what role contact should play in the policy.

The object is therefore synchronized interaction data: vision, proprioception, action, force, and touch recorded while the robot makes, maintains, loses, and recovers contact. This is harder than ordinary teleoperation. Sensors drift, contacts can be unsafe, materials change the signal, and the same visual state can hide different force backgrounds. Scaling contact data sharpens the system problem: is the added signal explaining a real failure mode, or just creating another out-of-distribution axis?

The upside is that the right contact-rich experience may teach priors vision cannot: when to push, when to yield, and when a small correction is better than a new plan. People are already expanding different contact regimes, not just adding one tactile sensor to a vision policy. FTP-1 studies cross-sensor tactile data [Yuan et al., 2026]. T-Rex focuses on high-frequency tactile feedback for reactive dexterous manipulation [Niu et al., 2026]. Together, these works suggest that contact data may become a new scaling axis, but only if we understand when it should enter the system.

Questions I am left with

The field is moving from isolated robot datasets toward data engines. That makes the research questions feel more operational, but not less scientific.

  1. How should we choose the mixture between human video, robot teleoperation, synthetic trajectories, recovery data, and contact-rich data?
  2. What makes a base model easy to steer, and can we measure that before deploying it on hardware?
  3. Which parts of a policy can be learned from human physical data, and which parts still require robot embodiment data?
  4. What is the right unit for dexterous-hand data: successful episodes, contact transitions, regrasp attempts, or recoveries?
  5. When should simulation be used for evaluation, failure mining, data generation, or post-training RL?
  6. When does a new modality remove action ambiguity, and when does it only introduce a new distribution shift?
  7. Can offline signals predict real robot success well enough to guide the next data collection round?

My current guess is that future progress will come from treating data as an adjustable part of the robot learning system. The goal is to choose the intervention at each stage, not to maximize every source at once.

These thoughts are still not fully mature. I am writing them down mostly as a working note, both to organize my own thinking and to share the questions I currently care about. If you disagree, see a gap, or think I am missing something important, I would be happy to discuss it and learn where this framing is wrong.

References

  1. Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS. Paper
  2. Open X-Embodiment Collaboration, Padalkar, A., Pooley, A., Jain, A., et al. (2023). Open X-Embodiment: Robotic Learning Datasets and RT-X Models. arXiv preprint arXiv:2310.08864. Paper
  3. Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., et al. (2024). DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset. RSS. Paper
  4. Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., and Fox, D. (2023). MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations. CoRL. Paper
  5. Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., et al. (2022). Ego4D: Around the World in 3,000 Hours of Egocentric Video. CVPR. Paper
  6. Genesis AI Team. (2026). The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward. Genesis AI Blog. Blog
  7. Sunday Robotics. (2026). ACT-2 Preview: Generalizing Reliability. Sunday Robotics Blog. Blog
  8. Generalist Team. (2026). GEN-1: Scaling Embodied Foundation Models to Mastery. Generalist AI Blog. Blog
  9. Yuan, C., Zhang, Z., Zhou, M., Chen, W., Wang, Y., Liu, Z., Niu, D., Wang, S., Zhang, H., Zhang, W., Hu, Y., Gong, Y., Xing, W., Wen, C., Lu, C., Zhang, K., and Gao, Y. (2026). FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation. arXiv preprint arXiv:2606.13102. Paper
  10. Niu, D., Liu, Z., Wang, Z., Shao, B., Yin, Z.-H., Pai, A., Sharma, Y., Saravalle, S., Zheng, R., Wang, J., Punamiya, R., Xu, M., Xie, Y., Jiang, Y., Fu, L., et al. (2026). T-Rex: Tactile-Reactive Dexterous Manipulation. arXiv preprint arXiv:2606.17055. Paper