'The simulation score is high, so it will work on hardware': where that breaks
- The claim
- The implication most common in paper abstracts and demo reels: a policy scoring X% on a simulation benchmark will perform close to X% when deployed on real hardware.
- Reality
- What real-hardware reproduction usually shows: the sim score is X%, the hardware score is clearly lower, and the more complex the task the further it falls; contact-rich tasks (insertion, deformables) barely transfer at all without real data or domain randomisation. The drop is not one thing called 'inaccurate simulation' — it is contact modelling, visual differences, mechanism backlash and sensor noise stacking, and any single one of them can take a task from 'works' to 'does not'.
- Where the gap is
- The gap is that 'the simulation validated the policy's logic' and 'the simulation guaranteed the policy's physical feasibility' are two different claims. A sim score proves the first — the policy learned the task's structure. It cannot prove the second — real contacts and noise are less forgiving than the model. The demo cases that 'perform close to sim' have almost always compensated real parameters in the same pipeline; that step just never makes the abstract.
- When it closes
- The gap narrows substantially when three things are in place: ① system identification with real-hardware data (friction, inertia, actuator response); ② domain randomisation (training against jittered parameters in sim); ③ safety nets and retry logic on hardware (reporting 'success within retries' rather than single-attempt rates). The number will always sit below the sim score — but that margin moves from 'unusable' to 'acceptable' through these three steps, not through a bigger model.
This site's call
Treat simulation as a filter, not a promise. A policy that scores well in sim is worth trying on hardware; do not copy the number into your expectations. The practical rule for builders: get the motion 'visibly stable' in sim before hardware — every real attempt costs a hundred times a sim rollout, so let simulation do what it is for: eliminating bad candidates cheaply.
Four sources of the drop
| Source | In simulation | On hardware |
|---|---|---|
| Contact model | Idealised friction and restitution | Wear, grime and deformation keep changing friction |
| Vision | Clean renders, controllable light | Reflections, shadows, white-balance drift |
| Backlash | Rigid joints, zero play | Printed parts and gearboxes all have play |
| Sensor noise | Often omitted or idealised | IMU vibration, encoder quantisation, dropped frames |
Sources
- Sim-to-real transfer literature and reproduction discussions (domain randomisation, system identification)
Objects in this entry
Last checked 2026-09-28