LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling

Transformers (Vaswani et al., 2017), originally designed for natural language tasks, have recently been applied to the computer vision domain to model images and videos (Dosovitskiy et al., 2021; Liu et al., 2021; Touvron et al., 2021; Wang et al., 2021; Han et al., 2021). ViT (Dosovitskiy et al., 2021) is one of the most representative vision transformers, which processes each image as a sequence of image patches, and achieves remarkable performances on various image tasks.

Compared with image tasks, video understanding is more challenging because the additional temporal dimension brings a more complicated spatial-temporal information. To model the intertwined dependency of the spatial and temporal dimensions efficiently, video Transformers TimeSformer (Bertasius et al., 2021) and ViViT (Arnab et al., 2021) use the parameters of a well-trained image Transformer for initialization, and further design different variants of spatial-temporal attention mechanisms to capture the spatial-temporal dependencies.

The core of video-language models lies in modeling the interaction between the two modalities. Depending on whether using video-text pairs for pre-training, existing video-language models can be divided into two categories.

The main branch of works explicitly designs the spatial-temporal structure in video encoders, and pre-train with abundant video-text pairs to directly align videos and texts. Among these works, ALPRO (Li et al., 2022a), Frozen (Bain et al., 2021), and BridgeFormer (Ge et al., 2022) use WebVid2M (Bain et al., 2021) which contains 2.5M video-text pairs collected from the web for pre-training. Image-text pairs like CC3M (Sharma et al., 2018) and VG (Krishna et al., 2016) are also often used to enhance the spatial information in this alignment. NoiseEST (Amrani et al., 2021), VideoCLIP (Xu et al., 2021), and CLIP4Clip (Luo et al., 2021) pretrain the model with a large-scale dataset HowTo100M (Fan et al., 2019) which contains 136M video-text pairs. The other branch of works does not rely on video-text pre-training. Instead, they extend a pre-trained image-text model to extract video features, and directly fine-tune on downstream tasks. In ClipBERT (Lei et al., 2021), BLIP (Li et al., 2022b), and X-Pool (Gorti et al., 2022), each video is viewed as a collection of images, whose representations obtained from an image encoder are then used to represent the video for later interaction with the text.

As is shown in the Figure 1, our framework contains three parts: a video encoder, a text encoder, and a video-grounded text encoder. We initialize our framework based on the recently proposed image-language model BLIP (Li et al., 2022b) trained over massive image-text pairs.

We wish to preserve the spatial representation and its alignment with the text encoder learned by the image-language pre-trained model, as well as learn temporal expressivity for video-language tasks. As will be shown in Table 6, directly using TimeSformer yields better results than the original ViT. To provide more sufficient temporal expressiveness of the video encoder, we propose to learn a set of scalings that dynamically adjust the newly inserted temporal attention modules according to each specific task, as shown in Figure 1(b).

Specifically, denote the output feature after the temporal attention at the $l$-th Transformer layer as ${\bf V}_{l}^{\text{TAttn}}\in\mathbb{R}^{S\times T\times D}$. For $t$-th frame $[{\bf V}_{l}^{\text{TAttn}}]_{t}\in\mathbb{R}^{S\times D}$, we add a learnable scaling factor $\alpha_{l,t}\in\mathbb{R}$ with a tanh-gating mechanism as:

where $\gamma_{l,t}$ is a learnable scalar initialized at $0$. Then the scaled output of temporal attention ${\bf V}_{l}^{\text{TAttn}}$ is calculated as:

before the residual connection. Note the [CLS] token is kept but not involved in the computation of scaling. The choice of tanh-gating ensures that $\alpha_{l,t}$ ranges from 0 to 2. Initially, our model is equivalent to TimeSformer (i.e., $\alpha_{l,t}$=1, treat each frame equally), and then explicitly reweight the frames in each transformer block during the fine-tuning stage. When $\alpha_{l,t}$ reduces to 0, the video encoder degenerates to the ViT used in the original BLIP model, which does not consider any temporal dependency in extracting the video features.

Before interacting with the textual modality via cross-attention or self-attention, previous methods directly concatenate the features from all frames with equal importance (Li et al., 2022b; Ge et al., 2022), or aggregate the video features with heuristic mean/max pooling methods spatially or temporally (Luo et al., 2021; Li et al., 2022a). However, not all frames or spatial positions are equally representative for the whole video, and different frames or positions have different semantic similarities to the textual query (e.g., textual description in text-video retrieval tasks or textual question in video question answering tasks. For example, given a video description “a golf player is trying to hit the ball into the pit”, the video encoder is expected to focus on the object of interest (i.e., ball) and the motion of hitting across the frames.

As illustrated in Figure 2, we design a non-parametric text-dependent pooling to reweight the video features spatially and temporally depending on the corresponding textual query, enabling fine-grained video-text alignment.

Specifically, given a video with $T$ frames, each frame is pachified into $S$ patches, and a [CLS] token is inserted before the $ST$ patches. Denote the original output embedding of the video encoder as ${\bf V}_{L}\in\mathbb{R}^{(1+ST)\times D}$. ${\bf V}_{f_{t}}\in\mathbb{R}^{T\times D}$ and ${\bf V}_{f_{s}}\in\mathbb{R}^{S\times D}$ are the video features pooled by averaging ${\bf V}_{L}$ along the spatial and temporal dimension, respectively. Note that the feature of [CLS] token is not involved in averaging. Denote ${\bf t}_{cls}\in\mathbb{R}^{D}$ as the output embedding of the [CLS] token obtained from the text encoder.

Intuitively, the more similar a visual feature is to the text description, the more representative it is for understanding the content of the whole video. Thus we compute the similarity between the $\ell_{2}$ normalized features of each frame in ${\bf V}_{f_{t}}^{\text{norm}}$ and the text feature ${\bf t}_{cls}^{\text{norm}}$, and reweight the features in ${\bf V}_{f_{t}}$ as:

where $\odot$ means element-wise multiplication, and $\tau$ is the temperature which controls the sharpness of the weight distribution. We multiply the weights from the softmax function by the number of frames $T$, such that the sum of total weights keeps the same as direct concatenation. Similarly, we compute the similarity between the $\ell_{2}$ normalized features of each spatial position in ${\bf V}_{f_{s}}^{\text{norm}}$ and the text feature ${\bf t}_{cls}^{\text{norm}}$, and reweight the features in ${\bf V}_{f_{s}}$ as:

The final aggregated video feature to be fed to the video-grounded text encoder is a concatenation of $\tilde{{\bf V}}_{f_{t}},\tilde{{\bf V}}_{f_{s}}$, and the original video feature ${\bf V}_{L}$:

After obtaining the aggregated video features ${\bf V}_{f}$, we feed them to each cross-attention layer of the video-grounded text encoder. Consider a training batch with $B$ video-text pairs. For the $k$-th video-text pair, denote the $\ell_{2}$ normalized output embeddings of the [CLS] tokens from the video encoder and the text encoder as ${\bf v}^{k}_{cls}$ and ${\bf t}^{k}_{cls}$ $^{1}$, respectively. The output embedding of the [Encode] token of the video-grounded text encoder is denoted as ${\bf t}^{k}_{enc}$.

In this work, we choose BLIP to initialize our proposed model mainly because (i) it performs well on various downstream image-language tasks; and (ii) it can be regarded as a single-stream and dual-stream hybrid structure. Its dual-stream part allows efficient inference for cross-modal retrieval tasks, while its cross-attention allows deep cross-modal interaction for tasks like VQA.

On the other hand, the proposed dynamic temporal scaling and text-dependent pooling can also be applied to the dual-stream model like CLIP (Radford et al., 2021). For this setting, we also conduct a simple experiment. For CLIP, we use the proposed text-dependent pooling on top of the video features. As CLIP relies on the global features for retrieval, instead of concatenation in Eq.(5), we compute a weighted average of the reweighted features. Compared with a recent work CLIP4Clip which also extends CLIP for video retrieval, CLIP with our proposed method improves the best CLIP4Clip-meanP method by 1.9% and 1.7%, for the R1 and R10 on MSRVTT-7k, respectively.

Since the test set sizes for all three retrieval datasets used in this work are relatively small, we compute a pairwise VTM score $s_{\mathrm{vtm}}$ for all video-text pairs during inference. However, the speed of inference in this approach will be slow when the size of dataset is huge in real-world scenarios.

In this section, we provide a more efficient retrieval solution. Specifically, we first compute the video-text similarity score $s_{\mathrm{vtc}}$ for all video-text pairs, then we take the top-$k$ candidates and calculate their VTM score for ranking. Such method can speed up inference, because the $k$ can be set to be very small compared with the test set size. Table 8 shows that using this efficient two-stage retrieval solution has negligible performance degradation.