Authors: Louis Keryhuel & Khanh Nguyen

Introduction

In this post, we'll be discussing one of our latest projects, the Game Tags Classifier. We were able to create this ambitious proof of concept in just one month, from brainstorm to delivery. If you're interested in learning how AI can help you predict game genres automatically to monitor the market more efficiently, keep reading!

Part 1: The Context and the Achievement

Every week, thousands of new game apps are submitted to the App Store and PlayStore. As a game publisher and studio, it's crucial for Homa to understand basic app metadata such as the genre, mechanics, camera angles, and in-game objects. While third-party services can provide this information, they often lack flexibility. For instance, they may not allow us to define our own genre categories or tag smaller apps from lesser-known developers. It is crucial for us to keep an eye on our competitors' prototypes and follow any shift in market trends. To solve this problem, we decided to build our own metadata prediction tool, the Game Tags Classifier.

The Game Tags Classifier (GTC)’s goal is simple: it allows Homa to identify the genre of hypercasual games available on app stores based on just two inputs: the title and the description of the app.

Despite the success of the project, it was not easy to build. We faced challenges in choosing the right methodology for classifying games, selecting the most suitable technology, and determining the source of truth for our input data.

Part 2: The Technology

There are two main considerations when choosing which technology to use for the project.

Type of Data Input for Prediction

When looking at the public data available on the app store pages, there are two types of possible inputs. Using only text-based information such as the title and description falls into natural language processing while making use of the screenshot images available would rely heavily on computer vision algorithms to decode the game's images.

While images can provide a lot of information about a game's genre, we've decided to explore training an AI model just by using text-based information.

This is because text inputs are much easier to work with compared to images. Following our literature research, we’ve understood that using images to predict a game’s genre only brings a marginal increase in prediction performance compared to using only text input.

Jiang, Y., & Zheng, L. (2020). Deep learning for video game genre classification. arXiv. https://doi.org/10.48550/arXiv.2011.12143

Prediction Algorithm

For our genre prediction algorithm, we decided to use OpenAI's GPT3. At its core, it is an algorithm allowing us to predict a sentence’s next words based on its beginning: Google search’s auto-complete is a great illustration. If a user types in "Homa is a mobile game," the top search suggestion will add the word "company."

We chose this algorithm due to the following advantages:

🤖 It is a state-of-the-art algorithm that is very performant for many standard language tasks, such as translation, question-answering, or predictive text. This is because GPT3 is built using transformers, which are neural-network architectures that understand natural languages remarkably well.

🤖 Yet, it is flexible enough to be modified based on our own labeled data (games whose genres were manually tagged by our internal team). Afterward, when we provide the AI model with the title and description of a given game, the model can "autocomplete" the genre of the game, so to speak.

🤖 Lastly, it has a very convenient API for training and prediction, which allows us to save a lot of time and effort to deploy the model compared to building our own models in-house (using deep learning libraries like TensorFlow or PyTorch).

# Train model
response = openai.FineTune.create(training_file=file_id, model="ada")

# Predict genre of new games using trained model
response = openai.Completion.create(
            model=self.openai_fine_tuned_model_id,
            prompt=input_data,
            max_tokens=1,
            temperature=0,
            logprobs=5,
        )

Part 3: The Reliability & the Outcome

Our auto-tagging technology can predict game genres with 92% accuracy.

We managed to achieve this high-accuracy level by following three main steps:

👉 STEP 1: We trained our model with more than 10,000 human-tagged apps to define the game genres we wanted to predict. At Homa, we were lucky to already have a large 3-year dataset of manually tagged apps thanks to the past tedious work from our internal market intelligence team. Leveraging this asset helped us tremendously in training our model. Using OpenAI to prepare our training dataset only cost us about $5; we were not going to find a better deal to test our prototype.

👉 STEP 2: This trained model was then used for predicting hundreds of untagged games belonging to several genres of the hyper-casual industry such as 'Arcade Idle', 'Platformer', 'Defense', or 'Runner', for instance. The cost for tagging an app was also very affordable — less than $0.1 per 1,000 tagged apps.

👉 STEP 3: We conducted a 'human tagging’ assessment consisting of validating the game genre predicted by the Game Tags Classifier. Comparing both sources helped us gain confidence in the output generated by GTC.

Even if some genres were predicted with a higher accuracy level than others, this proof of concept went way beyond our expectations.

GTC’s impact on Homa is still felt nine months later as it has fully automated the tedious task of tagging games, allowing our in-house Market Intelligence tools to scale our efforts in identifying market trends while keeping a close eye on the game genres our competitors are working on.

Summary

The Technological Choice Was a Primary Factor for Succeeding in This Project

When selecting the technology that would fuel this project, we had a very pragmatic approach not focusing on overly complex models or trying to achieve the 100% prediction accuracy; we've decided to focus more on ease of use: using text input instead of images and using GPT3's API to train and deploy our AI models.

High Impact, Fast Execution

Like everything in Homa, impact mixed with speed of execution is a key value of ours as we were able to build an MVP in under a month, go through small iterations, and build the data pipelines necessary to serve our model's predictions to both internal dashboards and our Homa Lab data platform.

The Power of Unexplored Data

Working on this project, we realized Homa since its inception has accumulated a lot of data just waiting to be analyzed. For example, without the painstaking efforts of our Market Intelligence team to manually tag more than 10,000 games over the first three years of the company, we would have never been able to train our models to build this auto-tagging AI technology.

To give some examples of extensive internal datasets with little use today related to marketability & user behavior:

💡 The thousands of games that we tested along the way

💡 The nearly 100,000 creatives lovingly crafted by our creative teams

💡 The more than 1 billion events we receive every single day from our analytics data pipeline

In short, as we’re trying to fully exploit these datasets and more, we’re convinced we can bring many more insights and impact, keeping our competitive edge over the rest of the market.