flâneur

Aditya Ajay

0 followers · 218 views

on the atlas — 2

highlights — 5

  • RTFM’s spatial memory enables unbounded persistence. When generating a new frame, we retrieve nearby frames from the spatial memory of posed frames to form a custom context for the model. We refer to this technique as context juggling: the model uses different context frames when generating in different regions of space. This allows RTFM to persist large worlds over long interactions without reasoning over an ever‑growing set of frames.
    RTFM: A Real-Time Frame Model | World Labs
  • RTFM circumvents this problem by modeling each frame as having a pose (position and orientation) in 3D space. We generate new frames by querying the model with the pose of the frame to be generated. The model’s memory of the world (contained in its frames) thus has a spatial structure; it uses posed frames as a spatial memory. This endows the model with a weak prior – that the world it models is a three-dimensional Euclidean space – without forcing it to explicitly predict the 3D geometry of objects in that world.
    RTFM: A Real-Time Frame Model | World Labs
  • RTFM blurs the line between reconstruction (interpolating between existing views) and generation (creating new content not visible in input views) which have historically been treated as separate problems in computer vision. When RTFM is provided with many input views it leans toward reconstruction since the task is more constrained; when provided with fewer input views it is forced to extrapolate beyond them.
    RTFM: A Real-Time Frame Model | World Labs
  • network reads from this representation (via attention) to create new views of the world consistent with the input views
    RTFM: A Real-Time Frame Model | World Labs
  • Its input frames are converted to neural network activations (the KV cache) which implicitly represent the world
    RTFM: A Real-Time Frame Model | World Labs