VideoGameBench

We selected games based on relative difficulty and diversity of gameplay characteristics, which we've roughly highlighted in the list below. Certain games involve completing the entire single player mode (e.g. Super Mario Land, Legend of Zelda: Link's Awakening) while others only involve completing a single campaign or game due to their lengths (e.g. Civ 1). For certain games we also included sequels due to sufficient variety in world exploration.

MS-DOS 💻

1. Doom 3D shooter
2. Doom II 3D shooter
3. Quake 3D shooter
4. Sid Meier's Civilization 1 2D strategy turn-based
5. Warcraft II: Tides of Darkness (Orc Campaign) 2.5D strategy
6. Oregon Trail Deluxe (1992) 2D strategy turn-based
7. X-COM UFO Defense 2D strategy
8. The Incredible Machine (1993) 2D puzzle
9. Prince of Persia 2D platformer
10. The Need for Speed 3D racer
11. Age of Empires (1997) 2D strategy

Game Boy 🎮

1. Pokemon Red (GB) 2D grid-world turn-based
2. Pokemon Crystal (GBC) 2D grid-world turn-based
3. Legend of Zelda: Link's Awakening (DX for GBC) 2D open-world
4. Super Mario Land 2D platformer
5. Kirby's Dream Land (DX Mod for GBC) 2D platformer
6. Mega Man: Dr. Wily's Revenge 2D platformer
7. Donkey Kong Land 2 2D platformer
8. Castlevania Adventure 2D platformer
9. Scooby-Doo! - Classic Creep Capers 2D detective

In our experience, current state-of-the-art VLMs substantially struggle to play video games because of high inference latency. When an agent takes a screenshot(s) and queries the VLM about what action to take, by the time the response comes back, the game state has changed significantly and the action is no longer relevant (for example, an enemy shooting at the agent when the screenshot was taken might have now moved to standing in front of the agent).

Subset of VideoGameBench-Lite Games

1. Doom II 3D shooter
2. Quake 3D shooter
3. Prince of Persia 2D platformer
4. Legend of Zelda: Link's Awakening (DX for GBC) 2D open-world
5. Super Mario Land 2D platformer
6. Kirby's Dream Land (DX Mod for GBC) 2D platformer

It becomes apparent after running an agent on any of these games that VLM agents are not close to solving an entire game, let alone even the first level of most games. We have observed some interesting progress in games, e.g. our agent getting to the first mini-boss of Kirby's Dream Land that is interesting to observe. This section is primarily focused on highlighting a few qualitative observations rather than rigorous quantitative experimentation. All experiments use a basic VideoGameAgent that uses ReAct [25] with memory given a sequence of 5-10 frames and issues sequences of key-presses and mouse movements as actions.

Organizing thoughts and goal oriented-ness. There have been several prior works that successfully use LLMs for planning in games [13] and some others that have attempted to solve text games [8,15,24]. In the multimodal setting, we found that agents often misinterpret events from the game screen -- this gap between visual event interpretation and the language understanding causes unintended behavior, which can also affect the internal goals proposed by the agent. In the example below, the agent wastes its ammo thinking dead enemies are alive.

The granularity of an "action". An immediate issue when running a VLM on any real-time video game is the 3-5 second inference delay between querying the VLM for an action given the current game screen and when the action is actually returned by the VLM. Inference latency of large models is unavoidable, so an interesting question is what granularity a VLM "action" takes -- is it a single key press, a sequence of key presses, code, or even a simple mini policy?

Controller, Mouse + Keyboard Precision. We found several instances of the agent / VLM struggling to translate the effect of actions (e.g. move right) on the screen. The most obvious across all frontier models we tried (e.g. GPT-4o, Claude Sonnet 3.7, Gemini 2.5 Pro) was the inability to accurately move the position of the mouse for games like Civilization and Warcraft II, where frequent mouse movements are necessary.

Figure 7. Our agent (using GPT-4o) plays Warcraft II in real-time on VideoGameBench. The agent struggles to move the mouse to the correct position, leading to it clicking "load game" instead of "new game" each.

Unintuitive Game Mechanics. Unless explicitly prompted with proper instructions, many game mechanics are not obvious to VLMs. While seemingly obvious, this fact becomes more important when benchmarking on "secret" games, where game-specific prompts may be impossible and the model has to learn these mechanics from scratch.

Reinforcement Learning (RL) on Games

The premise of this question is a bit mis-leading, since certain data-driven RL methods have already proven to be able to fully solve games better than humans. For starters, RL has been able to solve Atari games [1] for almost a decade now. In terms of superhuman game-playing AI, DeepMind's AlphaGo [22] was a neural network capable of defeating the human world champion, but it was arguable that the game's state and action spaces are discrete. It was then believed that video games were infinitely more complicated, but it turned out with DeepMind's AlphaStar [2] and OpenAI Five [3] that proper featurization of the game environment made it roughly equivalent to the AlphaGo problem. Even in 3D game settings where this featurization is significantly more difficult, there have been strong efforts such as the Dreamer papers [4] that are actively building agents on games like Minecraft. Recently, a larger question with RL on video games is whether language-heavy games can be solved. These methods rely on using language models to replace / model some module of the reinforcement learning pipeline, e.g. the value function. One particularly interesting modern example are new agents on the popular turn-based battle simulator Pokemon Showdown, which have now achieved human-competitive performance using a mix of language models and RL-style policies [5]. Another prominent example is the CICERO agent [26] for the multi-player strategy game Diplomacy.

Pure VLMs and LMs playing Video Games.

There exists a class of unsolved video games that are extremely difficult for both RL methods and VLMs to solve. These games tend to involve some form of language component, long-form exploration / objectives, and spatial reasoning puzzles – the BALROG benchmark contains a suite of these games for VLMs to solve with progress indicators [6]. One of the most popular case studies of AI for games has been NetHack [6,8,9] which is a grid world terminal game with an extraordinarily complex combat, item, and dungeoning system (+ randomization) that makes it difficult for humans to complete.

VideoGameBench deviates slightly from these benchmarks in that a possible solution to most of the games is to learn a policy through "sufficiently large" RL-style exploration (e.g. Pokemon Red has been solved multiple times using RL [10,11], shooters like Doom have RL training platforms [16]). Unlike RL algorithms, while certain information about video games are most likely present in the training data of VLMs, it is significantly smaller than what is provided to RL methods. Recently, there has been a lot more interest in evaluating the capabilities of newer frontier models and agent approaches for solving the aforementioned games such as Claude plays Pokemon and now Gemini plays Pokemon (which uses a more interesting agentic approach). Another recent major effort has been by the Hao AI lab to build a platform for VLM agents to play games like Mario, Sokoban, and Candy Crush in real-time (their work is awesome so please check them out!) [12].

VideoGameBench and VideoGameBench-Lite similarly focus on real video games, but use a fixed and diverse set of challenging games (e.g. platformers, shooters, RTS, RPGs, 2D, 2.5D, 3D, etc.) with a standard common interface. The common environment is also designed to eventually enable plugging in various emulators without any extra friction.

It is easy to run other Game Boy or MS-DOS games on our platform by downloading the ROM / js-dos link and adding a new config with specifications (e.g. defining a game-specific prompt, a preset list of actions to run when a game loads like pre-selecting the difficulty, and game settings). We provide explicit instructions in our GitHub README. Extending to other emulators is also relatively simple, but we currently do not support this in a no-code fashion. We want to add diverse support to maintain our benchmark, so we welcome open-source contributions – we generally would like to work in the open! Finally, we picked a specific set of games for our benchmark, but we encourage people to run and experiment with other games on our environment.

We want to emphasize that the development of VideoGameBench was made possible by leveraging existing open-source emulators. We are grateful to the developers of PyBoy, DOSBox, JS-DOS, and Playwright. This benchmark uses proprietary games, please make sure to purchase these games before running the benchmark.

Benchmarking frontier models and agents on real video games is both flashy and experimentally useful for measuring reasoning capabilities. Unlike extremely complicated domains like unsolved math proofs and olympiad-level math problems, playing video games is not a superhuman reasoning task, yet models still struggle to solve them. Furthermore, while most progress has been on LLM and text-only reasoning, few benchmarks evaluate reasoning capabilities over multimodal domains in an interpretable and widely understandable setting.

[Open-source] We have worked on many open-source projects and gladly welcome contributions from the community to expand the suite of games and supported emulators. Details on contributing can be found in the project repository: https://github.com/alexzhang13/videogamebench.