From JEPA world models to Semigroup-JEPA: latent dynamics consistency across gravity regimes

What JEPA world models had already established

Joint-Embedding Predictive Architecture learns latent representations by jointly training an encoder and predictor instead of reconstructing future pixels, and recent instantiations such as DINO-WM and LeWorldModel showed that these latent predictions support downstream planning and control under a fixed frame-to-frame transition law [1]. The broader JEPA line is now large enough to have its own efficiency debate: a frozen-teacher video SSL study reports that student models outperform V-JEPA 2 under frozen evaluation and dominate its accuracy-FLOPs Pareto frontier, suggesting elaborate online teacher-student and EMA collapse-prevention machinery may be unnecessary for high-quality representation [6]. A separate world-action model line pushes JEPA features into manipulation, using a frozen V-JEPA2.1 encoder plus a semantic autoencoder and diffusion branches to reach 92.64% average success on 50 RoboTwin tasks in clean environments, with only a 2.84% gap under randomization [2]. Together these establish that JEPA latents are useful for control and that their training recipe is still being actively simplified, but they leave open whether the learned update composes over long horizons or generalizes to out-of-distribution dynamics [1].

Gravity conditioning plus recursive rollout: the SG-JEPA change

SG-JEPA extends LeWorldModel by supplying the parameter governing the physics, gravity, to the temporal model via action-conditioning and jointly training encoder and predictor through a discounted K-step autoregressive rollout loss with SIGReg regularization [1]. Training samples gravity from a narrow Gaussian, N(4, 0.5^2) for 2D planar shapes and Arm Catcher Ball and N(9.8, 2.0^2) for the remaining 3D datasets, while held-out test sets use a wider grid including out-of-distribution values [1]. Against DINO-WM, which keeps a pretrained DINOv2 encoder frozen and trains an action-conditioned Transformer predictor, SG-JEPA reduces open-loop prediction error by up to 2x on 2D datasets and increases control success up to 2.5x on 3D robotic datasets using independently trained diffusion policies [1]. The comparison matters because DINO-WM's frozen encoder cannot adapt its features to the rollout objective, whereas SG-JEPA's encoder is trained through the same recursive loss that will be used at inference [1].

Where the gain actually comes from: encoder, not predictor

The paper's mechanistic claim is that the advantage lives in the representation. Starting from a trained checkpoint, replacing either the encoder or the predictor with a freshly initialized one and retraining jointly leaves performance unchanged when the encoder is retrained, but degrades it when the predictor is retrained [1]. A frozen-representation crossover tells the same story: a fresh Transformer predictor modestly reduces error on both latent sources, by -0.107 on the GRU-trained representation and -0.103 on the Transformer-trained representation, yet the GRU-trained representation remains better overall [1]. The GRU-trained representation also depends less on long context: shortening context from 20 to 4 frames produces a Transformer-source minus GRU-source history penalty of only 0.015 over the first five forecasts but 0.089 over the full 44-step rollout [1]. This is consistent with the paper's linear feature model, which separates local law-conditioned error from its recursive amplification and shows that back-propagating the multi-step rollout loss into the representation trains the encoder to keep features the predictor can carry forward [1].

Competing explanations: novelty detection, geometry conditioning, and physics reasoning

A competing approach argues that imperfect world models should be moderated at planning time rather than fixed in representation: WM-VAE adds a variational autoencoder novelty detector to DINO-WM and uses per-action reconstruction loss as a CEM cost, improving planning on granular, rope, and cloth manipulation in NVIDIA FleX [4]. This is a different lever, not a refutation, and the two could in principle combine, but the supplied evidence does not test that combination [1][4]. A geometry-conditioned diffusion precursor shows the same tension in a different domain: a DDPM conditioned on voxelized airway anatomy and guided by a signed distance field recovers velocity-magnitude statistics with Wasserstein distances on the order of 10^-3 to 10^-2, yet generalization to unseen anatomies remains fundamentally limited by the diversity of simulated geometries, with only nine simulated anatomies and degraded behavior on out-of-distribution cases [3]. That is a useful boundary for SG-JEPA's own claim: parameter generalization within a known law is not the same as generalization across geometry or morphology [1][3]. A validation-style result from a different modality, PhysMent, finds that LLM physics reasoning improves with more iterations only for some models, with Gemini 3.1 Pro rising from 29.5% to 66.7% while GPT-5.5 slightly declined from 51.4% to 48.6%, and that calculation tasks lag comparison tasks [5]. The parallel is that more rollout or more compute is not automatically better; what matters is whether the model has the right structure to use it [1][5].

Boundaries: one scalar, linear theory, and uneven shape transfer

The paper's own limitations are explicit. Experiments vary a single scalar, gravity, and the authors flag vector-valued physical variables or inferring dynamical parameters directly from observation as future work [1]. Transfer across object shapes is uneven: training on 2D triangle and square datasets transfers much of the house's translational dynamics but not its rotation, and the pentagon remains challenging, possibly requiring a generalist model trained on diverse data [1]. The theory uses a linear feature model while the neural predictor is nonlinear and history-dependent, and contacts can perturb transition branches, so the semigroup and law-coverage bounds are explanatory rather than exact for the deployed network [1]. The evidence boundary is therefore clear: conclusions rest on 2D and 3D specific gravity-field tasks and do not directly extend to real robots or all physical systems [1]. Adjacent precursors reinforce the caution. A spatiotemporal early-warning study of the compact-to-dendritic transition reports limited but systematic transferability across reaction-rate conditions, with performance degrading as the inference condition departs from training and latent-state dynamics changing accordingly [7]. A physics-inspired latent dynamics model for olfactory mixture similarity transfers zero-shot to one dataset at rho = 0.408 but yields near-zero correlation on single-molecule odor character, illustrating that a latent dynamics inductive bias can be useful in one regime and uninformative in another [9]. Broader latent-dynamics frameworks in biology and chemistry make similar moves, coupling learned latent spaces to physical or biological structure, but they do not test the specific semigroup composition claim [8][10][11][12].

About These Sources

This research page is built on 12 studies (11 peer-reviewed, 1 preprint) — published from 2025 to 2026, 12 from 2024 or later — selected as the most relevant from 12 studies that passed quality screening, drawn from 66 papers retrieved from a database of over 500 million.

Sources used in this answer