Vision & Multimodal Sep 2025 · #87 most cited · 65 citations

3D and 4D World Modeling: A Survey

Lingdong Kong et al.

arXiv:2509.07996

In short

A survey of world models that use native 3D and 4D data such as depth images, occupancy grids and LiDAR point clouds, not just 2D video. It proposes definitions and a taxonomy, and summarises datasets, metrics and applications like autonomous driving.

Why it matters

“World model” means many things; this pins down the 3D sense of it.

Read first

The 3 Field Guide ideas this paper leans on.

Starting from scratch? The full route 11 ideas · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Reinforcement Learning ✓ understood

    Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies.

  3. Agent ✓ understood

    In reinforcement learning, the learner and decision-maker that observes an environment, takes actions and adjusts its behaviour to maximize cumulative reward.

  4. Environment ✓ understood

    In RL, the world the agent interacts with, providing states, accepting actions, and returning rewards.

  5. World Model · read first ✓ understood

    A learned model of environment dynamics that can predict future states, used in model-based RL.

  6. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  7. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  8. Computer Vision ✓ understood

    The field of AI that gets computers to extract meaning from images and video: what is in them, where it is, and how it moves.

  9. Depth Estimation ✓ understood

    Predicting distance of objects from the camera using monocular or stereo images.

  10. 3D Reconstruction · read first ✓ understood

    Creating 3D models from 2D images using geometry and deep learning.

  11. Image Generation · read first ✓ understood

    Creating new images, often from a text prompt, with generative models such as GANs, VAEs and diffusion models that learn to turn random noise into realistic pictures.

In the frontier

Rank
#87 of 100
Citations
65
as of Aug 9, 2026
Published
Sep 2025

Topics: Video generation and world models , 3D and spatial intelligence , Robot policies

Selection: 1kpapers.com by Together AI, most-cited as of Aug 9, 2026

Nearby papers

Summary in our own words; read the paper for the details. ← All papers