flâneur

Creating In-Video Search - DEV Community

dev.to · 1,354 words · saved by 1 readers

In-video search is ability to search for a specific content within a video. This can include searching for particular words spoken, objects shown or description of a scene. With the current advancement in transformers the process of in-video search have become more accurate and fairly simple. Although most of the transformers doesn’t have a joint embedding space for multiple modalities but there are few models like Meta’s ImageBind that a joint embedding space between text, image, audio, depth, thermal and IMU, or OpenAI’s CLiP model have joint embedding space between text and image. We can use these models to create a relatively quick and accurate in-video service. A typical video would have the following data with them -: Frames -: A video is basically a series of the frames. Some of these frames are complete frames called i-frames and others will be partial frames, also known as p-frames and b-frames, which contain only the changes from the previous frame. Audio -: A video also migh

In-video search is ability to search for a specific content within a video. This can include searching for particular words spoken, objects shown or description of a scene. With the current advancement in transformers the process of in-video search have become more accurate and fairly simple. Although most of the transformers doesn’t have a joint embedding space for multiple modalities but there are few models like Meta’s ImageBind that a joint embedding space between text, image, audio, depth, thermal and IMU, or OpenAI’s CLiP model have joint embedding space between text and image. We can us

saved by

related reading