VGI-Bench
VGI-Bench (Video General Intelligence Bench, by Seldon) asks multiple-choice questions about long-form videos: what appeared first, what was on screen at the same time as something else, what a narrator said while a specific shot was up, and questions whose premise never happens. Each sample below is a real run through OpenRouter. The model received the full video and the question in one turn at temperature 0, and the graded answer is the letter on its final Answer: line.
Last benchmark run
Model comparison
Accuracy
Representative-run accuracy, best first.
Cost per question
Average cost per question, cheapest first.
Time per question
Average wall-clock time per question, fastest first.
Cost efficiency
Accuracy vs. cost (Pareto frontier)
One point per model, using default routing (not pinned to a provider) when available. The line is the Pareto frontier: no model beats these on both accuracy and cost.
Leaderboard
Accuracy over the public split (439 questions, runs need at least 395 answered to be listed), with cost, time, and input and output tokens per question for each provider endpoint the model was run on.
| # | Model | Std dev | |||||
|---|---|---|---|---|---|---|---|
| 1 | Seed | 68.6% | -- | $0.066 | 2.1m | 228k | 4.55k |
| 2 | ByteDance Seed: Seed 1.6 Pareto | 67.3% | -- | $0.041 | 68s | 148k | 2.21k |
| 3 | Alibaba | 66.7% | -- | $0.058 | 4.7m | 413k | 11.3k |
| 4 | Alibaba | 63.6% | -- | $0.62 | 8.9m | 584k | 19.9k |
| 5 | 61.4% | ±0.0pp | $0.27 | 2.6m | 231k | 17.2k | |
| 6 | Google AI Studio | 60.7% | -- | $0.059 | 2.1m | 341 | 5.88k |
| 7 | Alibaba | 58.0% | -- | $0.33 | 8.2m | 579k | 13.6k |
| 8 | Google AI Studio | 55.4% | -- | $0.14 | 3.8m | 108k | 40.1k |
| 9 | 54.7% | -- | $0.25 | 2.9m | 152k | 22.8k | |
| 10 | Alibaba | 54.2% | -- | $0.26 | 5.8m | 1.48M | 11k |
| 11 | 51.6% | -- | $0.15 | 2.0m | 153k | 16.5k | |
| 12 | Alibaba | 51.0% | -- | $0.21 | 7.4m | 517k | 19.5k |
| 13 | Google: Gemini 2.5 Flash Pareto Google AI Studio | 49.8% | -- | $0.023 | 2.3m | 341 | 9.17k |
| 14 | 48.9% | ±1.4pp | $0.030 | 2.3m | 37.7k | 6.93k | |
| 15 | Alibaba | 47.4% | -- | $0.10 | 4.8m | 1.5M | 20.1k |
| 16 | Xiaomi: MiMo-V2.5 Pareto Xiaomi | 47.2% | ±2.0pp | $0.020 | 5.0m | 208k | 13.1k |
| 17 | Perceptron | 46.0% | -- | $0.008 | 1.9m | 45.8k | 1.01k |
| 18 | Alibaba | 46.0% | -- | $1.14 | 6.8m | 1.49M | 19.9k |
| 19 | 45.6% | -- | $0.14 | 5.6m | 368k | 16.8k | |
| 20 | 45.6% | -- | $0.027 | 4.6m | 77.4k | 14.7k | |
| 21 | 45.4% | -- | $0.034 | 5.2m | 337k | 11.9k | |
| 22 | 44.8% | -- | $0.17 | 8.4m | 517k | 21.4k | |
| 23 | Xiaomi | 44.3% | -- | $0.040 | 4.7m | 75.7k | 22.1k |
| 24 | 43.4% | -- | $0.34 | 3.2m | 77k | 21.1k | |
| 25 | 40.5% | -- | $0.024 | 1.9m | 211k | 7.3k | |
| 26 | 34.6% | ±2.7pp | $0.028 | 8.3m | 58.2k | 43.9k | |
| 27 | 28.3% | ±0.8pp | $0.10 | 13.4m | 52.4k | 97.3k | |
| 28 | Reka Edge Pareto Reka | 21.2% | -- | $0.00015 | 37s | 1.48k | 29.6 |
| 29 | 13.8% | -- | $0.025 | 14.9m | 11k | 25.1k | |
| 30 | 11.8% | -- | $0.084 | 10.3m | 114k | 2.33k | |
| 31 | Alibaba | 11.6% | ±0.5pp | $0.029 | 3.9m | 30.5k | 5.69k |
| 32 | 9.4% | -- | $0.00034 | 38s | 2.6k | 4.61k | |
| 33 | BaseTen | 5.3% | ±5.9pp | $0.028 | 36s | 4.49k | 1.46k |
| 34 | Darkbloom | 2.1% | -- | $0.003 | 11.4m | 612 | 27.9k |
| 35 | Z.AI | 1.4% | -- | $0.004 | 61s | 22k | 227 |
Trajectories
The reasoning trace is the model's perception: the moments it says it saw and the timestamps it attaches to them. Reading it against the video shows how the answer was reached, and on the failed runs, where the description parts from the footage.
correct
Which window the pointer hovers over most
Happy path: The model locates the moment and reads it correctly.
Question 1187 global-graph/interaction_partner Qwen3.8 27B
The full video the model was sent, served from the OpenRouter mirror used by the harness.
Which option does the mouse pointer hover over most often?
- A)Windhawk application windowkey, model
- B)Install button
- C)Adobe Premiere Pro window
- D)Windows taskbar
answered A ✓key A
The trace tracks the pointer across the whole recording rather than one frame, names the Premiere Pro cameo near the end as a distractor, and lands on the window that dominates the session.
The question asks which option the mouse pointer hovers over most often. Let me think about the video content. The video is a tutorial about Windhawk, a customization tool for Windows. Throughout the video, the user is constantly interacting with the Windhawk application window - browsing mods, installing them, configuring settings, etc.
Let me consider each option:
- A) Windhawk application window - This is the main focus of the video. The user opens Windhawk, browses mods, clicks on details, settings, installs mods, etc. The mouse is frequently over this window.
- B) Install button - The install button is clicked a few times but not hovered over most often.
- C) Adobe Premiere Pro window - This appears only briefly near the end (around 13:20-13:40) as a joke about the title bar color.
- D) Windows taskbar - The taskbar is shown but the mouse isn't constantly hovering over it.
The dominant interaction is with the Windhawk application window. The mouse pointer is over it for the majority of the video as the user navigates through mods, settings, etc.
So the answer is A.
Let me double check by reviewing timestamps. From 1:20 onwards, the Windhawk window is open and the mouse moves around it constantly - clicking Mods, Explore, scrolling, clicking Details, Settings, etc. This continues through most of the video. The taskbar is shown at times but the mouse hovers over the Windhawk window more.
Thus A is correct.
incorrect
Object co-present with a garden shed
Failure mode: Salient object over the key
Question 1862 global-graph/co_presence Qwen3.8 27B
The full video the model was sent, served from the OpenRouter mirror used by the harness.
Which option is repeatedly visible at the same time as the wooden garden shed during the video?
- A)presenter with short curly hairkey
- B)metal shed hinge
- C)wagon wheelmodel
- D)glass doors
answered C ✗key A
The trace cites eight timestamps for a wagon wheel leaning against the shed and dismisses the presenter as incidental. The human-validated key is the presenter. The model chose the most distinctive object in the frame over the one the question was scored on.
The question asks which option is repeatedly visible at the same time as the wooden garden shed during the video.
Let me look at the frames where a wooden garden shed appears. At around 3:57-4:00, there's a wooden garden shed with a wagon wheel leaning against it, and the presenter is sitting in front of it. At 4:57-5:15, the same wooden shed with the wagon wheel is visible while the presenter talks about the slab base.
So the wooden garden shed appears together with the wagon wheel in multiple segments. Let me check the other options:
- A) presenter with short curly hair - the presenter is visible in many places, but the question is about what's visible at the same time as the wooden shed. The presenter is there too, but the wagon wheel is a distinctive object attached to the shed scene.
- B) metal shed hinge - not particularly visible.
- C) wagon wheel - yes, clearly visible leaning against the wooden shed in multiple frames (3:57, 4:00, 4:57, 5:00, 5:03, 5:07, 5:11, 5:15).
- D) glass doors - those are on a different building (the dark shed with white doors).
The wagon wheel is repeatedly visible alongside the wooden garden shed. The answer is C.
Let me double check the frames. At 3:57 the presenter sits against a wooden shed with a large wagon wheel on the right. At 4:57-5:15 same scene. So wagon wheel is repeatedly visible with the wooden shed.
Thus answer C.
incorrect
A false-premise question about a measuring spoon
Failure mode: Narrates a moment that never happens
Question 1096 contrastive-hard-negative/false-premise-visual-attribute Qwen3.8 27B
The full video the model was sent, served from the OpenRouter mirror used by the harness.
When the speaker says she used a full tablespoon of salt, where is the red measuring spoon positioned relative to the cheese curds?
- A)This does not happen in the video.key
- B)It is resting beside the bowl of cheese curds.
- C)It is positioned just in front of the cheese curds.model
- D)It is laid across the top of the cheese curds.
answered C ✗key A
The key says this moment does not occur. The trace narrates a 2:38 to 2:44 sequence with the spoon set down in front of the curds, rejects the 'does not happen' option outright, and answers with detail the key does not support.
The user wants me to look at the video and determine the position of the red measuring spoon relative to the cheese curds when the speaker mentions using a full tablespoon of salt.
Let me trace through the video. Around 2:38-2:42, the speaker is adding salt to the cheese curds in a cloth-lined pot. She holds a red measuring spoon with salt in it. At 2:42, she sets the red spoon down. Let me look at where it ends up.
At 2:42-2:44, after adding the salt, she places the red spoon down in front of the bowl/curds. Looking at the frame at 2:44, the red spoon is lying on the counter just in front of the cheese curds (the cloth-lined pot). Then she picks up the wooden spoon to stir.
Let me check the options:
A) Does not happen - but it does happen, she uses a tablespoon of salt.
B) Resting beside the bowl - "beside" could be left/right.
C) Positioned just in front of the cheese curds.
D) Laid across the top.
Looking at the frame at 2:44, the red spoon is at the bottom of the frame, in front of the curds. So "just in front of the cheese curds" matches C.
Let me double check the timestamp. At 2:38 she holds the red spoon with salt. At 2:42 she tips it in. At 2:44 the red spoon is resting on the counter at the front edge, in front of the curds. Yes.
Thus answer C.
I need to provide reasoning then final answer format.
incorrect
The sound of an M.2 heatsink locking
Failure mode: Answers an audio question from world knowledge
Question 2852 contrastive-hard-negative/visual_audio Qwen3.8 27B
The full video the model was sent, served from the OpenRouter mirror used by the harness.
When the two M.2 slots are highlighted in purple, what click is heard as the top heatsink locks back into place?
- A)This does not happen in the videokey
- B)A sharp metallic snap
- C)A firm double-click
- D)A soft plastic clickmodel
answered D ✗key A
The trace says 'I need to recall the audio' and then reasons from what PC build videos 'typically' sound like, never from this one. The key is 'does not happen'. The final response presents the guessed click as something heard in the video.
The user is asking about a specific audio detail in the video. Let me look at the relevant section. Around 02:20-02:30, the video shows the M.2 slots being highlighted in purple (the diagram at 02:20 shows the M.2 slots). Then the person is working with the heatsink.
Let me think about the sequence. At 02:20 there's a diagram showing M.2 slots highlighted. Then at 02:30 the person is holding a black heatsink piece. At 02:40-02:50 they're working on the motherboard with the M.2 area.
The question asks: when the two M.2 slots are highlighted in purple, what click is heard as the top heatsink locks back into place?
I need to recall the audio. In these PC build videos, when you press down on the M.2 heatsink, there's typically a plastic click as it locks. The options are: sharp metallic snap, firm double-click, soft plastic click, or doesn't happen.
Given the context of an M.2 heatsink locking into place, the typical sound is a soft plastic click. The heatsink is plastic/metal but the locking mechanism is usually a plastic clip.
Let me consider the diagram at 02:20 - it shows the M.2 slots with purple/magenta highlighting. Then the person installs the SSD and the heatsink. The click when the heatsink locks would be a soft plastic click.
I'll go with D, soft plastic click.
Let me double check by reasoning about the video flow. The diagram with purple highlights appears at 02:20. Then at 02:30-02:50 the person is handling the heatsink and the M.2 area. The locking click of an M.2 heatsink is generally a soft plastic click.
Thus answer D.
incorrect
Background behind a PSU cable being plugged in
Failure mode: Locates a near-duplicate moment
Question 1539 near-duplicate-moments Qwen3.8 27B
The full video the model was sent, served from the OpenRouter mirror used by the harness.
When the hand inserts a black power cable into the Seasonic power supply at the rear of the black PC case, what surface is behind it?
- A)A dark, textured surfacekey
- B)A gray carpeted surface
- C)A metal mesh surface
- D)A plain white surfacemodel
answered D ✗key A
The question pins a moment that recurs with small differences. The trace settles on a rear view at 15:30 to 15:40 with a white wall behind it. The key is a dark textured surface, so the model matched the action to a similar shot and read the background off that one.
The user wants me to identify the surface behind the action of plugging a black power cable into the Seasonic PSU at the rear of the case.
Let me look at the relevant timestamps. Around 15:30-15:40, the video shows the rear of the case with the Seasonic PSU installed, and a hand plugging in a black power cable (the IEC power cord) into the PSU.
Looking at the frames around 15:30-15:40, the background behind the case appears to be a plain white wall/surface. There's a white wall with some light switches visible in earlier frames (around 7:50-8:00), and at the rear view (15:30-15:40) the background is a plain white surface.
Let me verify by examining the frames. At 15:30, the case is shown from the rear with the Seasonic logo, and the background is white. At 15:40, the hand plugs the cable in, and the background is a plain white surface.
So the answer is D) A plain white surface.
Let me double check the other options: A) dark textured surface - that's the table surface where components were laid out earlier. B) gray carpeted surface - similar to the table. C) metal mesh surface - that's part of the case itself. The background behind the case at the rear view is white.
Thus D.
incorrect
Where a whiskey bottle sits when a line is spoken
Failure mode: Never locates the spoken cue
Question 2894 contrastive-hard-negative/transcript_visual Qwen3.8 27B
The full video the model was sent, served from the OpenRouter mirror used by the harness.
When he says, “This whole time he's back there drinking whiskey,” is the whiskey bottle to the left or right of the laptop on Craig’s bench?
- A)This does not happen in the video.key
- B)To the left of the laptop.model
- C)To the right of the laptop.
- D)Directly behind the laptop.
answered B ✗key A
The cue is a spoken line. The trace never finds it, says it 'probably occurs during the conversation at the bench', and answers from a frame at 60 to 70 seconds. The key is 'does not happen'. Without the transcript anchor, the model substituted the nearest matching scene.
The user is asking about a specific moment in the video where someone says "This whole time he's back there drinking whiskey." I need to find this moment and look at the position of the whiskey bottle relative to the laptop on Craig's bench.
Let me look at the early part of the video. Around 60-70 seconds, there's a scene with a man sitting at a bench with a laptop, and there's a bottle on the bench. Let me look carefully.
At around 60-70 seconds, there's a man in a light blue shirt sitting in a chair at a bench with a laptop. On the bench, to the left of the laptop (from the viewer's perspective), there's a bottle (looks like a whiskey bottle) and a mug.
Wait, I need to find the audio line "This whole time he's back there drinking whiskey." This is likely during the scene where two people are at the bench with the laptop. Let me look at the timestamps around 60-130 seconds.
At 60-70s: man in blue shirt at bench with laptop, bottle on left side of bench.
At 120-130s: close up of laptop showing a 3D model of a table, with a hand typing.
The line about drinking whiskey probably occurs during the conversation at the bench. The bottle is on the bench. Looking at the frame at 60-70s, the bottle is to the left of the laptop (viewer's left). The laptop is on the right side of the bench, the bottle and mug are on the left side.
So from the viewer's perspective, the whiskey bottle is to the left of the laptop.
Thus the answer is B.
Let me double check the orientation. The question asks "to the left or right of the laptop on Craig's bench?" In the image at 60-70s, the laptop is on the right portion of the bench, the bottle is on the left portion. So the bottle is to the left of the laptop.
Therefore answer B.
I'm fairly confident. The line "This whole time he's back there drinking whiskey" matches the scene where one person is at the laptop and another is "back there" drinking. The bottle visible on the bench is to the left of the laptop.
Thus B.