flâneur

Kalie C

1 followers · 1 following · 304 views

on the atlas — 12

highlights — 23

  • The standard view of visual perception, embraced in many quarters of science, is one in which we as perceiving subjects observe a determinate external reality on the basis of internally generated representations
    The Perception of Art and the Science of Perception
  • The objects in the painting are ‘there’, but only because we are ‘here’.
    The Perception of Art and the Science of Perception
  • As JJ Gibson puts it: ‘…perceiving the environment includes the ego as part of the total process. In order to localize any object there must be a point of reference. An impression of ‘there’ implies an impression of ‘here’, and neither could exist without the other.
    The Perception of Art and the Science of Perception
  • The importance of this local disorder, according to Koenderink and van Doorn, is that it encodes information in a different way from blur. Disorder, or ‘scramble’, reduces the distinctness of an image but retains far more useful information about the scene than the equivalent blurring of the image would do.
    The Perception of Art and the Science of Perception
  • Visual agnosia is an unfortunate neurological condition in which the sufferer is able to ‘see’ quite clearly, i.e. their eyes and much of the visual brain remains in tact, but they lose the capacity to recognize what they see.
    The Perception of Art and the Science of Perception
  • Our visual system creates an internal representation, or model, of objects and events in the external world, and it is this model we subjectively experience.
    The Perception of Art and the Science of Perception
  • In order to evaluate the abilities of FROMAGe to process multimodal contextual information, we assess its perfor- mance in retrieving the appropriate image conditioned on a sequence of interleaved image-text inputs from the Vi- sual Storytelling (VIST) dataset (Huang et al., 2016).
    [2301.13823] Grounding Language Models to Images for Multimodal Generation
  • Finally, to map the model’s output representations to visual space, we train a linear mapping W t ∈ R p × q . This maps the hidden representation of [RET] from the last hidden layer of the LLM, h θ ( x i ) ∈ R p , into a vector space for retrieval, where q is a dimension smaller than p .
    [2301.13823] Grounding Language Models to Images for Multimodal Generation
  • We learn translation parameters (parameterized as linear layers) to cast images into text space, and text embeddings into visual space
    [2301.13823] Grounding Language Models to Images for Multimodal Generation
  • We start from a frozen pretrained LLM, and a frozen pretrained visual encoder, and train with a multi- task objective for (1) image captioning (learning to process interleaved multimodal inputs) and (2) image-text retrieval (learning to produce interleaved multimodal outputs)
    [2301.13823] Grounding Language Models to Images for Multimodal Generation
  • To this end, we formulate a language-grounded pre-training by mapping learned 3D features to pre-trained language embeddings with a con- trastive loss. This enables a more robust 3D representation learning under im- balanced and limited 3D observations.
    Language-Grounded Indoor 3D Semantic Segmentation in the Wild
  • Compared with the best competitor [25] using the same training setting, MVT outperforms the state-of-the- art method by +11 . 2% , from 43 . 9% to 55 . 1% . Even when compared with SAT [33], which utilizes extra 2D semantics assisted its training, our MVT still outperforms it by 5 . 9% absolute value
    Multi-View Transformer for 3D Visual Grounding
  • The models are evaluated by the accuracy, which measures the percentage of successful matches be- tween the predicted proposal and the ground truth proposal
    Multi-View Transformer for 3D Visual Grounding
  • We propose the auxiliary task of language-guided classification by leveraging object category labels as the text supervision
    Multi-View Transformer for 3D Visual Grounding
  • The aggrega- tion function should be order-independent as in Eq. (4), here we choose the average aggregation by default, given as: g i = N X j =1 f j i N , (11) where G = { g 1 ,...,g M } is the final multi-modal feature of each object. Finally we obtain the grounding score by applying two FC layers on G .
    Multi-View Transformer for 3D Visual Grounding
  • As shown at the bottom of Fig. 3, in each layer of the decoder, object features O j first pass through a self-attention module [29], where interactions between objects are modeled. Then object features O j and language features L
    Multi-View Transformer for 3D Visual Grounding
  • herefore, we decoupled the 3D object feature extraction process into two steps, i.e., computing point clouds features and multi-view positional encoding . This decoupling can significantly save the computational cost since different views can share the same point cloud features outputted from the first step. Finally, the object features can be calculated by simply combining both
    Multi-View Transformer for 3D Visual Grounding
  • MV3D [5] encodes the sparse 3D point cloud with a compact multi-view (e.g., bird’s eye view and front view) representation for the 3D detection task [12]. MVCNN [28] completes the 3d object classification by rendering a large number of 2d pictures from 3d objects.
    Multi-View Transformer for 3D Visual Grounding
  • Referit3D [1] uses GRU [6] to extract text features, and makes use of GNN [27] to model the relationship between objects.
    Multi-View Transformer for 3D Visual Grounding
  • Our approach simul- taneously models the position information within the 3D scene under different views and aggregates all information to learn a view-robust representation by eliminating the dependence on the starting view.
    Multi-View Transformer for 3D Visual Grounding
  • Especially, the robot or the intelligent agent should be able to ground to the target object no matter how its view is different from the speaker’s
    Multi-View Transformer for 3D Visual Grounding
  • This discrepancy commonly exists in real applications: when a robot or an intelligent agent is navigating in the 3D scene, its view is usually different from the speaker’s or commander’s view
    Multi-View Transformer for 3D Visual Grounding
  • Compared to 2D visual grounding, the 3D task has more complex input data (e.g., sparse point clouds) and more variant spatial relationships, switching the output from grounding to 2D regions to 3D objects.
    Multi-View Transformer for 3D Visual Grounding