{ "figures": [ { "number": 1, "id": "S1.F1", "file": "fig-01-intro.png", "url": "https://arxiv.org/html/2502.11664v4/intro.png", "caption": "Figure 1: Comparison of RoPE, RoPE-3D, and our VRoPE in video positional encoding. (a) Positional Unbiasedness: RoPE and RoPE-3D exhibit spatial biased attention, particularly towards later tokens or specific frame regions, while VRoPE ensures more uniform attention. (b) Seamless Video-Text Transition: RoPE-3D causes a discontinuity when transitioning from video to text tokens, which VRoPE smooths for better cross-modal dependency modeling.", "width": "476", "height": "501" }, { "number": 2, "id": "S3.F2", "file": "fig-02-attn.png", "url": "https://arxiv.org/html/2502.11664v4/attn.png", "caption": "Figure 2: Attention weight visualization of RoPE, RoPE-3D, and VRoPE. We compute average text-to-video frame attention weights on VideoMME Fu et al. (2024) benchmark (lighter color indicates higher attention). (a) RoPE exhibits row-wise attention decay within frames. (b) RoPE-3D shows a similar decay from the bottom-right to the top-left, introducing positional bias that skews attention toward spatially closer frame tokens. (c) VRoPE mitigates this bias, leading to a more balanced attention distribution.", "width": "429", "height": "137" }, { "number": 3, "id": "S3.F3", "file": "fig-03-vrope.png", "url": "https://arxiv.org/html/2502.11664v4/vrope.png", "caption": "Figure 3: Left : the overall architecture of a typical Video-LLM. In this work, our improvements primarily target the positional embedding component of the LLM to enhance its video understanding capability. Right : method illustration of VRoPE. (a) We first apply symmetric arrangement to mitigate positional bias in video frames. The RoPE frequencies are uniformly allocated to the four dimensions. (b) We propose to use temporal centered arrangement in video frames to form a seamless video-text transition, which enables video input of arbitrary length without causing discontinuity.", "width": "476", "height": "156" }, { "number": 4, "id": "S5.F4", "file": "fig-04-needle.png", "url": "https://arxiv.org/html/2502.11664v4/needle.png", "caption": "Figure 4: Visualization of long video retrieval results on Video-NIAH Zhao et al. (2024) . Our VRoPE consistently achieves high accuracy across varying background lengths and needle depths, showing strong retrieval capability in long videos.", "width": "476", "height": "106" }, { "number": 5, "id": "A2.F5", "file": "fig-05-example.png", "url": "https://arxiv.org/html/2502.11664v4/example.png", "caption": "Figure 5: Attention weight visualization of RoPE, RoPE-3D, and VRoPE. The visualization reveals that VRoPE exhibits stronger attention activation within critical frames (highlighted by red boxes), demonstrating its accurate focus on pivotal spatiotemporal regions. In contrast, RoPE and RoPE-3D display attenuated attention responses in these corresponding areas, indicating insufficient awareness of key events. This attention misalignment consequently leads to erroneous predictions, as evidenced by their incorrect interpretations of the visual content.", "width": "429", "height": "256" } ], "tables": [ { "number": 1, "id": "S3.T1", "caption": "Table 1: Average attention weights at the video-text boundary on Video-MME. We use the subsequent text instruction as the query and video/text tokens as keys. Note that text-to-video attention weights of RoPE-3D are an order of magnitude lower than other methods, indicating its positional discontinuity between video and text.", "html": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MethodText-to-TextText-to-Video
RoPE1.41e-22.08e-4
RoPE-3D1.27e-25.12e-5
VRoPE (Ours)1.32e-23.70e-4
", "text": "Method Text-to-Text Text-to-Video RoPE 1.41e-2 2.08e-4 RoPE-3D 1.27e-2 5.12e-5 VRoPE (Ours) 1.32e-2 3.70e-4" }, { "number": 2, "id": "S5.T2", "caption": "Table 2: Performance comparison of RoPE variants on video benchmarks across different LLM backbones. Results across tasks, including general video understanding (Video-MME), video temporal understanding (MVBench, TempCompass), and long video understanding (MLVU, LongVideoBench, EgoSchema).", "html": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MethodVideo-MMEMLVUMVBenchLongVideoBenchTempCompassEgoSchemaAvg.
(w/o subs)@M-Avg@Val@Multi-Choice@Test
Video-Vicuna-7B
   w/ RoPE38.547.0043.9041.6653.1035.9243.35
   w/ RoPE-3D38.0 (↓\\downarrow0.5)46.30 (↓\\downarrow0.7)44.55 (↑\\uparrow0.65)40.16 (↓\\downarrow1.5)54.94 (↑\\uparrow1.84)39.79 (↑\\uparrow3.87)43.96 (↑\\uparrow0.61)
   w/ VRoPE38.9 (↑\\uparrow0.4)47.37 (↑\\uparrow0.37)45.18 (↑\\uparrow1.28)40.69 (↓\\downarrow0.97)54.05 (↑\\uparrow0.95)40.71 (↑\\uparrow4.79)44.48 (↑\\uparrow1.13)
Video-Qwen2-1.5B
   w/ RoPE39.051.1551.1546.6356.9648.5048.90
   w/ RoPE-3D39.3 (↑\\uparrow0.3)51.19 (↑\\uparrow0.04)50.45 (↓\\downarrow0.70)48.01 (↑\\uparrow1.38)57.97 (↑\\uparrow1.01)49.00 (↑\\uparrow0.50)49.32 (↑\\uparrow0.42)
   w/ VRoPE42.4 (↑\\uparrow3.4)51.76 (↑\\uparrow0.61)50.78 (↓\\downarrow0.37)47.79 (↑\\uparrow1.16)57.15 (↑\\uparrow0.19)49.90 (↑\\uparrow1.40)49.96 (↑\\uparrow1.06)
Video-Qwen2-7B
   w/ RoPE50.154.8754.3349.3663.7357.1454.92
   w/ RoPE-3D49.5 (↓\\downarrow0.6)56.06 (↑\\uparrow1.19)54.23 (↓\\downarrow0.1)49.55 (↑\\uparrow0.19)64.49 (↑\\uparrow0.76)58.80 (↑\\uparrow1.66)55.44 (↑\\uparrow0.52)
   w/ VRoPE50.6 (↑\\uparrow0.5)57.81 (↑\\uparrow2.94)54.70 (↑\\uparrow0.37)50.48 (↑\\uparrow1.12)65.88 (↑\\uparrow2.15)58.60 (↑\\uparrow1.46)56.35 (↑\\uparrow1.43)
", "text": "Method Video-MME MLVU MVBench LongVideoBench TempCompass EgoSchema Avg. (w/o subs) @M-Avg @Val @Multi-Choice @Test Video-Vicuna-7B w/ RoPE 38.5 47.00 43.90 41.66 53.10 35.92 43.35 w/ RoPE-3D 38.0 ( ↓ \\downarrow 0.5 ) 46.30 ( ↓ \\downarrow 0.7 ) 44.55 ( ↑ \\uparrow 0.65 ) 40.16 ( ↓ \\downarrow 1.5 ) 54.94 ( ↑ \\uparrow 1.84 ) 39.79 ( ↑ \\uparrow 3.87 ) 43.96 ( ↑ \\uparrow 0.61 ) w/ VRoPE 38.9 ( ↑ \\uparrow 0.4 ) 47.37 ( ↑ \\uparrow 0.37 ) 45.18 ( ↑ \\uparrow 1.28 ) 40.69 ( ↓ \\downarrow 0.97 ) 54.05 ( ↑ \\uparrow 0.95 ) 40.71 ( ↑ \\uparrow 4.79 ) 44.48 ( ↑ \\uparrow 1.13 ) Video-Qwen2-1.5B w/ RoPE 39.0 51.15 51.15 46.63 56.96 48.50 48.90 w/ RoPE-3D 39.3 ( ↑ \\uparrow 0.3 ) 51.19 ( ↑ \\uparrow 0.04 ) 50.45 ( ↓ \\downarrow 0.70 ) 48.01 ( ↑ \\uparrow 1.38 ) 57.97 ( ↑ \\uparrow 1.01 ) 49.00 ( ↑ \\uparrow 0.50 ) 49.32 ( ↑ \\uparrow 0.42 ) w/ VRoPE 42.4 ( ↑ \\uparrow 3.4 ) 51.76 ( ↑ \\uparrow 0.61 ) 50.78 ( ↓ \\downarrow 0.37 ) 47.79 ( ↑ \\uparrow 1.16 ) 57.15 ( ↑ \\uparrow 0.19 ) 49.90 ( ↑ \\uparrow 1.40 ) 49.96 ( ↑ \\uparrow 1.06 ) Video-Qwen2-7B w/ RoPE 50.1 54.87 54.33 49.36 63.73 57.14 54.92 w/ RoPE-3D 49.5 ( ↓ \\downarrow 0.6 ) 56.06 ( ↑ \\uparrow 1.19 ) 54.23 ( ↓ \\downarrow 0.1 ) 49.55 ( ↑ \\uparrow 0.19 ) 64.49 ( ↑ \\uparrow 0.76 ) 58.80 ( ↑ \\uparrow 1.66 ) 55.44 ( ↑ \\uparrow 0.52 ) w/ VRoPE 50.6 ( ↑ \\uparrow 0.5 ) 57.81 ( ↑ \\uparrow 2.94 ) 54.70 ( ↑ \\uparrow 0.37 ) 50.48 ( ↑ \\uparrow 1.12 ) 65.88 ( ↑ \\uparrow 2.15 ) 58.60 ( ↑ \\uparrow 1.46 ) 56.35 ( ↑ \\uparrow 1.43 )" }, { "number": 3, "id": "S5.T3", "caption": "Table 3: We assess various RoPE designs to validate the necessity of the three desired properties: Spatiotemporal Structure Modeling (S.S.M), Positional Unbiasedness (P.U.), and Seamless Video-Text Transition (S.V.T.). The results indicate that the model attains optimal performance when all properties are fully incorporated.", "html": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MethodS.S.M.P.U.S.V.T.Video-MMEEgoSchemaLongVideoBenchAvg.
RoPE✘✘✔39.048.5046.6344.71
RoPE-2D✔✘✔43.2 (↑\\uparrow4.2)47.60 (↓\\downarrow0.90)46.33 (↓\\downarrow0.30)45.71 (↑\\uparrow1.00)
RoPE-3D✔✘✘39.3 (↑\\uparrow0.3)49.00 (↑\\uparrow0.50)48.01 (↑\\uparrow1.38)45.44 (↑\\uparrow0.73)
RoPE-Share✘✔✔39.7 (↑\\uparrow0.7)48.66 (↑\\uparrow0.16)45.10 (↓\\downarrow1.53)44.49 (↓\\downarrow0.22)
RoPE-Compact✔✘✔38.1 (↓\\downarrow0.9)50.77 (↑\\uparrow2.27)45.96 (↓\\downarrow0.67)44.94 (↑\\uparrow0.23)
VRoPE✔✔✔42.4 (↑\\uparrow3.4)49.90 (↑\\uparrow1.40)47.79 (↑\\uparrow1.16)46.70 (↑\\uparrow1.99)
", "text": "Method S.S.M. P.U. S.V.T. Video-MME EgoSchema LongVideoBench Avg. RoPE ✘ ✘ ✔ 39.0 48.50 46.63 44.71 RoPE-2D ✔ ✘ ✔ 43.2 ( ↑ \\uparrow 4.2 ) 47.60 ( ↓ \\downarrow 0.90 ) 46.33 ( ↓ \\downarrow 0.30 ) 45.71 ( ↑ \\uparrow 1.00 ) RoPE-3D ✔ ✘ ✘ 39.3 ( ↑ \\uparrow 0.3 ) 49.00 ( ↑ \\uparrow 0.50 ) 48.01 ( ↑ \\uparrow 1.38 ) 45.44 ( ↑ \\uparrow 0.73 ) RoPE-Share ✘ ✔ ✔ 39.7 ( ↑ \\uparrow 0.7 ) 48.66 ( ↑ \\uparrow 0.16 ) 45.10 ( ↓ \\downarrow 1.53 ) 44.49 ( ↓ \\downarrow 0.22 ) RoPE-Compact ✔ ✘ ✔ 38.1 ( ↓ \\downarrow 0.9 ) 50.77 ( ↑ \\uparrow 2.27 ) 45.96 ( ↓ \\downarrow 0.67 ) 44.94 ( ↑ \\uparrow 0.23 ) VRoPE ✔ ✔ ✔ 42.4 ( ↑ \\uparrow 3.4 ) 49.90 ( ↑ \\uparrow 1.40 ) 47.79 ( ↑ \\uparrow 1.16 ) 46.70 ( ↑ \\uparrow 1.99 )" }, { "number": 4, "id": "S5.T4", "caption": "Table 4: Average retrieval accuracy across different input frame length intervals on Video-NIAH Zhao et al. (2024) . Compared to RoPE, the performance advantage of VRoPE becomes more pronounced at longer video lengths.", "html": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Method256-512512-768768-10241024-1216
RoPE94.8487.0373.2854.84
RoPE-3D88.9080.9469.6972.81
VRoPE98.2895.1690.3187.03
", "text": "Method 256-512 512-768 768-1024 1024-1216 RoPE 94.84 87.03 73.28 54.84 RoPE-3D 88.90 80.94 69.69 72.81 VRoPE 98.28 95.16 90.31 87.03" }, { "number": 5, "id": "S5.T5", "caption": "Table 5: Ablation study on VRoPE components. We evaluate the impact of Symmetric Bias Mitigation (Symmetric) and Temporal Centered Arrangement (Continuity). The model achieves the best performance when both components are applied together.", "html": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ContinuitySymmetricVideo-MMELongVideoBench
✘✘39.046.63
✔✘42.346.30
✘✔41.347.27
✔✔42.447.79
", "text": "Continuity Symmetric Video-MME LongVideoBench ✘ ✘ 39.0 46.63 ✔ ✘ 42.3 46.30 ✘ ✔ 41.3 47.27 ✔ ✔ 42.4 47.79" }, { "number": 6, "id": "A2.T6", "caption": "Table 6: Performance comparison of RoPE variants on event-based EventBench Du et al. (2024) .", "html": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MethodEventBench
Video-Vicuna-7B
   w/ RoPE38.97
   w/ RoPE-3D39.33 (↑\\uparrow0.36)
   w/ VRoPE40.38 (↑\\uparrow1.41)
Video-Qwen2-1.5B
   w/ RoPE53.31
   w/ RoPE-3D52.76 (↓\\downarrow0.55)
   w/ VRoPE54.23 (↑\\uparrow0.92)
Video-Qwen2-7B
   w/ RoPE59.25
   w/ RoPE-3D58.61 (↓\\downarrow0.64)
   w/ VRoPE60.35 (↑\\uparrow1.1)
", "text": "Method EventBench Video-Vicuna-7B w/ RoPE 38.97 w/ RoPE-3D 39.33 ( ↑ \\uparrow 0.36 ) w/ VRoPE 40.38 ( ↑ \\uparrow 1.41 ) Video-Qwen2-1.5B w/ RoPE 53.31 w/ RoPE-3D 52.76 ( ↓ \\downarrow 0.55 ) w/ VRoPE 54.23 ( ↑ \\uparrow 0.92 ) Video-Qwen2-7B w/ RoPE 59.25 w/ RoPE-3D 58.61 ( ↓ \\downarrow 0.64 ) w/ VRoPE 60.35 ( ↑ \\uparrow 1.1 )" }, { "number": 7, "id": "A2.T7", "caption": "Table 7: Detailed performance comparison of RoPE variants on Video-MME Fu et al. (2024) .", "html": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MethodShortMediumLong
Video-Vicuna-7B
   w/ RoPE46.438.031.0
   w/ RoPE-3D46.0 (↓\\downarrow0.4)37.5 (↓\\downarrow0.5)30.6 (↓\\downarrow0.4)
   w/ VRoPE46.4 (-)38.3 (↑\\uparrow0.3)31.8 (↑\\uparrow0.8)
Video-Qwen2-1.5B
   w/ RoPE47.437.632.2
   w/ RoPE-3D47.1 (↓\\downarrow0.3)37.0 (↓\\downarrow0.6)33.8 (↑\\uparrow1.6)
   w/ VRoPE50.1 (↑\\uparrow2.7)39.3 (↑\\uparrow1.7)37.8 (↑\\uparrow5.6)
Video-Qwen2-7B
   w/ RoPE60.247.642.5
   w/ RoPE-3D60.0 (↓\\downarrow0.2)46.7 (↓\\downarrow0.9)41.7 (↓\\downarrow0.8)
   w/ VRoPE60.4 (↑\\uparrow0.2)47.6 (-)43.9 (↑\\uparrow1.4)
", "text": "Method Short Medium Long Video-Vicuna-7B w/ RoPE 46.4 38.0 31.0 w/ RoPE-3D 46.0 ( ↓ \\downarrow 0.4 ) 37.5 ( ↓ \\downarrow 0.5 ) 30.6 ( ↓ \\downarrow 0.4 ) w/ VRoPE 46.4 (-) 38.3 ( ↑ \\uparrow 0.3 ) 31.8 ( ↑ \\uparrow 0.8 ) Video-Qwen2-1.5B w/ RoPE 47.4 37.6 32.2 w/ RoPE-3D 47.1 ( ↓ \\downarrow 0.3 ) 37.0 ( ↓ \\downarrow 0.6 ) 33.8 ( ↑ \\uparrow 1.6 ) w/ VRoPE 50.1 ( ↑ \\uparrow 2.7 ) 39.3 ( ↑ \\uparrow 1.7 ) 37.8 ( ↑ \\uparrow 5.6 ) Video-Qwen2-7B w/ RoPE 60.2 47.6 42.5 w/ RoPE-3D 60.0 ( ↓ \\downarrow 0.2 ) 46.7 ( ↓ \\downarrow 0.9 ) 41.7 ( ↓ \\downarrow 0.8 ) w/ VRoPE 60.4 ( ↑ \\uparrow 0.2 ) 47.6 (-) 43.9 ( ↑ \\uparrow 1.4 )" }, { "number": 8, "id": "A2.T8", "caption": "Table 8: Results on Video-MME Du et al. (2024) under lower frame rates (8 frames).", "html": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MethodAcc.
Video-Qwen2-1.5B
   w/ RoPE38.9
   w/ RoPE-3D37.2 (↓\\downarrow1.7)
   w/ VRoPE40.9 (↑\\uparrow2.0)
", "text": "Method Acc. Video-Qwen2-1.5B w/ RoPE 38.9 w/ RoPE-3D 37.2 ( ↓ \\downarrow 1.7 ) w/ VRoPE 40.9 ( ↑ \\uparrow 2.0 )" }, { "number": 9, "id": "A2.T9", "caption": "Table 9: Performance comparison of RoPE variants on larger models and datasets. Results across tasks, including general video understanding (Video-MME), video temporal understanding (MVBench, TempCompass), and long video understanding (MLVU, LongVideoBench, EgoSchema).", "html": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
MethodVideo-MMEMLVUMVBenchLongVideoBenchTempCompassEgoSchemaAvg.
(w/o subs)@M-Avg@Val@Multi-Choice@Test
Video-Qwen3-8B
   w/ RoPE61.0064.9659.6860.8168.6756.4161.92
   w/ RoPE-3D61.44 (↑\\uparrow0.44)64.50 (↓\\downarrow0.46)59.34 (↓\\downarrow0.34)61.00 (↑\\uparrow0.19)69.11 (↑\\uparrow0.44)56.03 (↓\\downarrow0.38)61.90 (↓\\downarrow0.02)
   w/ VRoPE62.56 (↑\\uparrow1.56)65.36 (↑\\uparrow0.40)59.23 (↓\\downarrow0.45)61.48 (↑\\uparrow0.67)68.67 (-)57.07 (↑\\uparrow0.66)62.40 (↑\\uparrow0.48)
", "text": "Method Video-MME MLVU MVBench LongVideoBench TempCompass EgoSchema Avg. (w/o subs) @M-Avg @Val @Multi-Choice @Test Video-Qwen3-8B w/ RoPE 61.00 64.96 59.68 60.81 68.67 56.41 61.92 w/ RoPE-3D 61.44 ( ↑ \\uparrow 0.44 ) 64.50 ( ↓ \\downarrow 0.46 ) 59.34 ( ↓ \\downarrow 0.34 ) 61.00 ( ↑ \\uparrow 0.19 ) 69.11 ( ↑ \\uparrow 0.44 ) 56.03 ( ↓ \\downarrow 0.38 ) 61.90 ( ↓ \\downarrow 0.02 ) w/ VRoPE 62.56 ( ↑ \\uparrow 1.56 ) 65.36 ( ↑ \\uparrow 0.40 ) 59.23 ( ↓ \\downarrow 0.45 ) 61.48 ( ↑ \\uparrow 0.67 ) 68.67 (-) 57.07 ( ↑ \\uparrow 0.66 ) 62.40 ( ↑ \\uparrow 0.48 )" } ], "equations": [ { "id": "S3.E1", "mathml": "RoPE​(𝐱,m)=𝐱​ei​m​𝜽\\text{RoPE}(\\mathbf{x},m)=\\mathbf{x}e^{\\mathrm{i}m\\boldsymbol{\\theta}}", "text": "RoPE ​ ( 𝐱 , m ) = 𝐱 ​ e i ​ m ​ 𝜽 \\text{RoPE}(\\mathbf{x},m)=\\mathbf{x}e^{\\mathrm{i}m\\boldsymbol{\\theta}} (1)" }, { "id": null, "mathml": "RoPE​(𝐱,m)=𝐱​ei​m​𝜽\\text{RoPE}(\\mathbf{x},m)=\\mathbf{x}e^{\\mathrm{i}m\\boldsymbol{\\theta}}", "text": "RoPE ​ ( 𝐱 , m ) = 𝐱 ​ e i ​ m ​ 𝜽 \\text{RoPE}(\\mathbf{x},m)=\\mathbf{x}e^{\\mathrm{i}m\\boldsymbol{\\theta}} (1)" }, { "id": "S3.E2", "mathml": "𝜽j=base−2​jd\\boldsymbol{\\theta}_{j}=\\text{base}^{\\frac{-2j}{d}}", "text": "𝜽 j = base − 2 ​ j d \\boldsymbol{\\theta}_{j}=\\text{base}^{\\frac{-2j}{d}} (2)" }, { "id": null, "mathml": "𝜽j=base−2​jd\\boldsymbol{\\theta}_{j}=\\text{base}^{\\frac{-2j}{d}}", "text": "𝜽 j = base − 2 ​ j d \\boldsymbol{\\theta}_{j}=\\text{base}^{\\frac{-2j}{d}} (2)" }, { "id": "S3.E3", "mathml": "𝐀(m,n)=ℜ⁡[𝐪m⋅𝐤n∗​ei⁡(m−n)​𝜽]\\mathbf{A}_{(m,n)}=\\Re\\left[\\mathbf{q}_{m}\\cdot\\mathbf{k}_{n}^{*}e^{\\mathrm{i}(m-n)\\boldsymbol{\\theta}}\\right]", "text": "𝐀 ( m , n ) = ℜ ⁡ [ 𝐪 m ⋅ 𝐤 n ∗ ​ e i ⁡ ( m − n ) ​ 𝜽 ] \\mathbf{A}_{(m,n)}=\\Re\\left[\\mathbf{q}_{m}\\cdot\\mathbf{k}_{n}^{*}e^{\\mathrm{i}(m-n)\\boldsymbol{\\theta}}\\right] (3)" }, { "id": null, "mathml": "𝐀(m,n)=ℜ⁡[𝐪m⋅𝐤n∗​ei⁡(m−n)​𝜽]\\mathbf{A}_{(m,n)}=\\Re\\left[\\mathbf{q}_{m}\\cdot\\mathbf{k}_{n}^{*}e^{\\mathrm{i}(m-n)\\boldsymbol{\\theta}}\\right]", "text": "𝐀 ( m , n ) = ℜ ⁡ [ 𝐪 m ⋅ 𝐤 n ∗ ​ e i ⁡ ( m − n ) ​ 𝜽 ] \\mathbf{A}_{(m,n)}=\\Re\\left[\\mathbf{q}_{m}\\cdot\\mathbf{k}_{n}^{*}e^{\\mathrm{i}(m-n)\\boldsymbol{\\theta}}\\right] (3)" }, { "id": "S3.E4X", "mathml": "RoPE-3Dj(𝐱,w,h,t)={RoPEj​(𝐱,w),j∈DwRoPEj​(𝐱,h),j∈DhRoPEj​(𝐱,t),j∈Dt\\displaystyle\\text{RoPE-3D}_{j}(\\mathbf{x},w,h,t)=\\left\\{\\begin{aligned} &\\text{RoPE}_{j}(\\mathbf{x},w),j\\in D_{w}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},h),j\\in D_{h}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},t),j\\in D_{t}\\end{aligned}\\right.", "text": "RoPE-3D j ( 𝐱 , w , h , t ) = { RoPE j ​ ( 𝐱 , w ) , j ∈ D w RoPE j ​ ( 𝐱 , h ) , j ∈ D h RoPE j ​ ( 𝐱 , t ) , j ∈ D t \\displaystyle\\text{RoPE-3D}_{j}(\\mathbf{x},w,h,t)=\\left\\{\\begin{aligned} &\\text{RoPE}_{j}(\\mathbf{x},w),j\\in D_{w}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},h),j\\in D_{h}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},t),j\\in D_{t}\\end{aligned}\\right. (4)" }, { "id": "S3.E5", "mathml": "RoPE-3Dj​(𝐱,m,m,m)≡RoPEj​(𝐱,m)\\text{RoPE-3D}_{j}(\\mathbf{x},m,m,m)\\equiv\\text{RoPE}_{j}(\\mathbf{x},m)", "text": "RoPE-3D j ​ ( 𝐱 , m , m , m ) ≡ RoPE j ​ ( 𝐱 , m ) \\text{RoPE-3D}_{j}(\\mathbf{x},m,m,m)\\equiv\\text{RoPE}_{j}(\\mathbf{x},m) (5)" }, { "id": null, "mathml": "RoPE-3Dj​(𝐱,m,m,m)≡RoPEj​(𝐱,m)\\text{RoPE-3D}_{j}(\\mathbf{x},m,m,m)\\equiv\\text{RoPE}_{j}(\\mathbf{x},m)", "text": "RoPE-3D j ​ ( 𝐱 , m , m , m ) ≡ RoPE j ​ ( 𝐱 , m ) \\text{RoPE-3D}_{j}(\\mathbf{x},m,m,m)\\equiv\\text{RoPE}_{j}(\\mathbf{x},m) (5)" }, { "id": "S4.E6", "mathml": "(u1u2u3u4)=(w+hw−h−w−h−w+h).\\begin{pmatrix}u_{1}\\\\\nu_{2}\\\\\nu_{3}\\\\\nu_{4}\\end{pmatrix}=\\begin{pmatrix}w+h\\\\\nw-h\\\\\n-w-h\\\\\n-w+h\\end{pmatrix}.", "text": "( u 1 u 2 u 3 u 4 ) = ( w + h w − h − w − h − w + h ) . \\begin{pmatrix}u_{1}\\\\\nu_{2}\\\\\nu_{3}\\\\\nu_{4}\\end{pmatrix}=\\begin{pmatrix}w+h\\\\\nw-h\\\\\n-w-h\\\\\n-w+h\\end{pmatrix}. (6)" }, { "id": null, "mathml": "(u1u2u3u4)=(w+hw−h−w−h−w+h).\\begin{pmatrix}u_{1}\\\\\nu_{2}\\\\\nu_{3}\\\\\nu_{4}\\end{pmatrix}=\\begin{pmatrix}w+h\\\\\nw-h\\\\\n-w-h\\\\\n-w+h\\end{pmatrix}.", "text": "( u 1 u 2 u 3 u 4 ) = ( w + h w − h − w − h − w + h ) . \\begin{pmatrix}u_{1}\\\\\nu_{2}\\\\\nu_{3}\\\\\nu_{4}\\end{pmatrix}=\\begin{pmatrix}w+h\\\\\nw-h\\\\\n-w-h\\\\\n-w+h\\end{pmatrix}. (6)" }, { "id": "S4.E7", "mathml": "(v1v2v3v4)=(u1u2+H−1u3+H+W−2u4+W−1)+ps​t​a​r​t.\\begin{pmatrix}v_{1}\\\\\nv_{2}\\\\\nv_{3}\\\\\nv_{4}\\end{pmatrix}=\\begin{pmatrix}u_{1}\\\\\nu_{2}+H-1\\\\\nu_{3}+H+W-2\\\\\nu_{4}+W-1\\end{pmatrix}+p_{start}.", "text": "( v 1 v 2 v 3 v 4 ) = ( u 1 u 2 + H − 1 u 3 + H + W − 2 u 4 + W − 1 ) + p s ​ t ​ a ​ r ​ t . \\begin{pmatrix}v_{1}\\\\\nv_{2}\\\\\nv_{3}\\\\\nv_{4}\\end{pmatrix}=\\begin{pmatrix}u_{1}\\\\\nu_{2}+H-1\\\\\nu_{3}+H+W-2\\\\\nu_{4}+W-1\\end{pmatrix}+p_{start}. (7)" }, { "id": null, "mathml": "(v1v2v3v4)=(u1u2+H−1u3+H+W−2u4+W−1)+ps​t​a​r​t.\\begin{pmatrix}v_{1}\\\\\nv_{2}\\\\\nv_{3}\\\\\nv_{4}\\end{pmatrix}=\\begin{pmatrix}u_{1}\\\\\nu_{2}+H-1\\\\\nu_{3}+H+W-2\\\\\nu_{4}+W-1\\end{pmatrix}+p_{start}.", "text": "( v 1 v 2 v 3 v 4 ) = ( u 1 u 2 + H − 1 u 3 + H + W − 2 u 4 + W − 1 ) + p s ​ t ​ a ​ r ​ t . \\begin{pmatrix}v_{1}\\\\\nv_{2}\\\\\nv_{3}\\\\\nv_{4}\\end{pmatrix}=\\begin{pmatrix}u_{1}\\\\\nu_{2}+H-1\\\\\nu_{3}+H+W-2\\\\\nu_{4}+W-1\\end{pmatrix}+p_{start}. (7)" }, { "id": "S4.E8", "mathml": "vjt=vj+t⁡(H+W−1),v_{j}^{t}=v_{j}+t(H+W-1),", "text": "v j t = v j + t ⁡ ( H + W − 1 ) , v_{j}^{t}=v_{j}+t(H+W-1), (8)" }, { "id": null, "mathml": "vjt=vj+t⁡(H+W−1),v_{j}^{t}=v_{j}+t(H+W-1),", "text": "v j t = v j + t ⁡ ( H + W − 1 ) , v_{j}^{t}=v_{j}+t(H+W-1), (8)" }, { "id": "S4.E9X", "mathml": "VRoPEj​(CLOSE\\displaystyle\\text{VRoPE}_{j}(", "text": "VRoPE j ​ ( CLOSE \\displaystyle\\text{VRoPE}_{j}( OPEN 𝐱 , v 1 t , v 2 t , v 3 t , v 4 t ) \\displaystyle\\mathbf{x},v_{1}^{t},v_{2}^{t},v_{3}^{t},v_{4}^{t}) (9)" }, { "id": "S4.E9Xa", "mathml": "={RoPEj​(𝐱,v1t)j=4​kRoPEj​(𝐱,v2t)j=4​k+1RoPEj​(𝐱,v3t)j=4​k+2RoPEj​(𝐱,v4t)j=4​k+3\\displaystyle=\\left\\{\\begin{aligned} &\\text{RoPE}_{j}(\\mathbf{x},v_{1}^{t})_{j=4k}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},v_{2}^{t})_{j=4k+1}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},v_{3}^{t})_{j=4k+2}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},v_{4}^{t})_{j=4k+3}\\end{aligned}\\right.", "text": "= { RoPE j ​ ( 𝐱 , v 1 t ) j = 4 ​ k RoPE j ​ ( 𝐱 , v 2 t ) j = 4 ​ k + 1 RoPE j ​ ( 𝐱 , v 3 t ) j = 4 ​ k + 2 RoPE j ​ ( 𝐱 , v 4 t ) j = 4 ​ k + 3 \\displaystyle=\\left\\{\\begin{aligned} &\\text{RoPE}_{j}(\\mathbf{x},v_{1}^{t})_{j=4k}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},v_{2}^{t})_{j=4k+1}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},v_{3}^{t})_{j=4k+2}\\\\\n&\\text{RoPE}_{j}(\\mathbf{x},v_{4}^{t})_{j=4k+3}\\end{aligned}\\right." } ], "headings": [ "VRoPE: Rotary Position Embedding for Video Large Language Models", "Abstract", "1 Introduction", "2 Related Work", "2.1 Video Large Language Models", "2.2 Multimodal Position Embedding", "3 Motivation", "3.1 Preliminary: Rotary Position Embedding", "3.2 RoPE for Video-LLMs", "3.3 RoPE-3D for Video-LLMs", "3.4 Problem Analysis", "(1) Positional Attention Bias.", "(2) Cross-Modal Positional Discontinuity.", "4 Method: VRoPE", "4.1 Symmetric Bias Mitigation", "4.2 Temporal Centered Arrangement", "5 Experiments", "5.1 Experimental Setup", "Implementation Details.", "Training Data.", "Evaluation Benchmarks.", "5.2 Main Results", "5.3 Results on Long Video Retrieval", "5.4 Ablation Studies", "Comparison of RoPE Variants.", "Ablation on VRoPE Components.", "6 Conclusion", "7 Acknowledgments", "8 Limitations", "References", "Appendix A Discussion", "Dimensional Adaptability.", "Appendix B More Results", "B.1 Results on EventBench", "B.2 Results on Video-MME with varying lengths", "B.3 Results under Challenging Conditions", "B.4 Results of Larger Models and Datasets", "Appendix C Visualization Analysis", "Appendix D Detailed Illustration of Other RoPE Variants", "RoPE-Share.", "RoPE-Compact.", "Appendix E License Statement" ] }