Most physics simulations maintain a separation that is rarely questioned. One system handles what objects look like. Another handles how they move. For offline rendering pipelines, this is fine. For anything requiring real-time physically plausible dynamics in robotics, embodied AI, or interactive world models, this separation is architecturally limiting.
What if material identity lived at the particle level — not the object level?
The problem with reconstruction-first approaches
PhysGaussian (CVPR 2024) showed that reconstructed 3D Gaussian scenes could be coupled to physics simulation, with Gaussian covariance updated by deformation. This was a meaningful advance. But it remained reconstruction-first: physics assigned to a captured scene after the fact. The geometry and simulation layers remained conceptually distinct.
GaussianFlesh asks a different question: what state must each Gaussian carry so that a single particle cloud can express geometry, material law, thermodynamics, and rendering all at once? Material identity becomes the primary key, instantiated before reconstruction, not inferred from it.
A unified particle state
Each particle stores position, velocity, deformation gradient, plastic state, rest and deformed covariance, mass, constitutive-law ID, stiffness parameters, yield and damage variables, temperature, latent-heat buffer, and solid/fluid phase. A shared Updated Lagrangian MLS-MPM solver advances all particles. During the transfer from particle to grid, each particle reads its own material model ID and dispatches to the appropriate stress law. All particles couple through the same grid.
This design makes multi-material heterogeneity a property of particle state rather than scene graph construction. A single connected Gaussian object can have a metal region, a rubber region, and a molten region while sharing one grid.
The dispatch is per particle, not per object.
Four material laws, one substrate
The system supports four constitutive laws that can coexist within a single scene:
- ✦Elastic rubber and jelly: corotated elasticity, with shape restoration implicit in the stress term
- ✦Metal: StVK stress with log-strain J2 return mapping, work hardening, permanent deformation after loading
- ✦Clay and sand: Drucker-Prager return mapping in Hencky strain space
- ✦Molten fluid: compressible pressure law with bulk modulus, activated on phase transition
The rendered Gaussian shape is updated once per frame by the physical push-forward: covariance follows deformation directly. Compression, stretch, and shear all change the rendered splat shape without a separate mesh proxy.
Thermomechanics: objects that actually melt
Heat diffuses on the same grid once per display frame. Before the melt point, shear modulus weakens continuously as temperature rises. At the melt point, excess heat fills a latent-heat buffer while temperature holds steady. Once the buffer saturates, the particle switches to fluid phase, its constitutive ID becomes fluid, and it begins responding to pressure rather than elastic stress.
This supports bottom-up melt fronts and solid/fluid coexistence within the same particle cloud and the same grid. An object can be partially solid and partially liquid simultaneously, with the boundary determined by local temperature history.
Trained 3DGS assets
GaussianFlesh can load trained 3D Gaussian Splatting PLY point clouds directly. The learned covariance, opacity, and spherical-harmonic appearance are preserved. Material identity is then painted per particle. A photoreal ficus or pillow can be simulated as rubber, metal, jelly, or a per-particle mixture without changing its visual texture, because appearance and physics are decoupled at the identity level.
The same photoreal object can be simulated as rubber, metal, or a per-particle mixture without changing its visual texture.
What this is actually about
A bouncing ball is not the finding. It is a demonstration that the primitive works.
World models, AI systems that simulate physical behavior, currently lack a grounded physics primitive. Video-based approaches learn correlations from pixels. They do not represent that a rubber ball and a metal ball dropped from the same height behave differently for reasons that can be computed, not just observed.
A substrate where any object can be instantiated, assigned a material identity, and simulated forward in the same Gaussian space that 4D vision systems use for reconstruction and tracking closes that loop. Future work will introduce vision-driven automatic material inference, closing the loop between scene understanding and physical instantiation.
We think the particle is the right place to start.