Tencent Unveils GameGen-O: AI Model That Generates 'Black Myth'-Style Game Videos with One Click

What? Could large language models soon generate AAA titles like Black Myth: Wukong?!

Journey to the West is now on the table, complete with a soundtrack. It’s got that authentic vibe (doge).

This is GameGen-O, the Transformer model for generating open-world video games recently unveiled by Tencent.

In simple terms, this model can simulate various game engine functions to generate game characters, dynamic environments, complex animations, and more.

It also supports interactive control, allowing users to manipulate game content via text, operational signals, and video prompts.

The announcement immediately triggered a wave of excitement on social media (formerly Twitter), with netizens lining up to express their amazement:

The co-founder and CTO of game studio Azra Games went so far as to say:

GameGen-O will be the ChatGPT moment for game studios.

“Game Studios Enter Their ChatGPT Moment”

Specifically, this project was launched by Tencent in collaboration with Hong Kong University of Science and Technology (HKUST) and the University of Science and Technology of China (USTC).

The inferred goal is to use AI models to replace certain game development processes. For instance, the currently demonstrated features include character creation, environment generation, animation generation, event generation, and various interactive controls.

Let’s take a closer look at each feature:

With GameGen-O, users can directly generate various characters—Western cowboys, astronauts, wizards, guards—with a single click.

If budget constraints make real-world filming difficult, there is now a Plan B.

Show off impressive moves to teammates; generating animations from various camera perspectives is now easily handled.

A staple of gaming—occasionally throwing some difficulty at players. Tsunamis, tornadoes, and fire events are now just a prompt away (doge).

Meanwhile, GameGen-O supports open-domain generation, meaning it is not limited by style, environment, or scene.

Finally, interaction is achieved using text, operational signals, and video prompts: move left, move right, walk toward the dawn…

Goodness, everyone knows how expensive game development is. Now, ordinary players can also create games using GameGen-O.

One AI architect netizen even declared:

Using GPT-4o for Data Annotation

To develop this model, the team stated they primarily undertook two tasks:

• Constructing a proprietary dataset, OGameData, using GPT-4o for data annotation
• Undergoing a two-stage training process

Specifically, the team first proposed a dataset construction pipeline.

The team collected 32,000 raw videos from the internet. These videos were sourced from hundreds of open-world games, ranging in length from minutes to hours, and covering genres such as role-playing, first-person shooters, racing, and action-puzzle games.

Human experts then identified and filtered these videos, resulting in approximately 15,000 usable clips.

Next, the filtered videos were segmented into clips using scene detection technology. These video segments underwent strict sorting and filtering based on aesthetics, optical flow, and semantic content.

Subsequently, over 4,000 hours of high-quality video clips, with resolutions ranging from 720p to 4K, were meticulously annotated using GPT-4o.

To enable interactive control, the team selected the highest-quality segments from the annotated dataset and performed decoupled labeling.

This labeling design describes changes in the state of clip content, ensuring that the training dataset is more refined and interactive.

Regarding this collaborative approach between human experts and GPT-4o, some netizens noted:

This represents a form of recursive self-improvement (human experts ensure annotation accuracy and help GPT-4o improve itself through feedback mechanisms).

After completing data preparation, the team trained GameGen-O through two processes: base pre-training and instruction tuning.

In the base training phase, the GameGen-O model used a 2+1D VAE (Variational Autoencoder, such as Magvit-v2) to compress video clips.

To adapt the VAE for the gaming domain, the team fine-tuned the VAE decoder specifically for this field.

The team adopted a mixed training strategy with varying frame rates and resolutions to enhance generalization across different frame rates and resolutions.

Additionally, the model’s overall architecture followed the principles of the Latte and OpenSora V1.2 frameworks.

By employing masked attention mechanisms, GameGen-O acquired dual capabilities: text-to-video generation and video continuation.

The team explained:

This training method, combined with the OGameData dataset, enables the model to stably and high-quality generate open-domain video game content, laying the foundation for subsequent interactive control capabilities.

Following this, the pre-trained model was frozen, and fine-tuning was performed using a trainable InstructNet. This allows the model to generate subsequent frames based on multimodal structural instructions.

InstructNet is primarily designed to accept various multimodal inputs, including structured text, operational signals, and video prompts.

During the adjustment of the InstructNet branch, current content was used as a condition, establishing a mapping relationship between current clip content and future clip content under multimodal control signals.

The result is that during inference, GameGen-O allows users to continuously generate and control the next generated segment based on the current one.

Currently, GameGen-O has created an official GitHub repository, although the code has not yet been uploaded.

Those interested can bookmark it for now.