DeNA held a lecture on reinforcement learning (RL) AI technology at CEDEC 2026 today (the 24th). The speaker, engineer Kosuke Sakimi, joined DeNA in 2019 and has since served as an AI system engineer for 'Pokémon Trading Card Game Pocket,' co-developed with The Pokémon Company and Creatures. Having been a speaker at the 'Reinforcement Learning AI for Live Service Games' session at CEDEC 2025, he returned to explain the process of enabling RL AI to play the complex TCG environment of Pokémon and how it can be applied across various domains.

Reinforcement Learning AI is Essential for Navigating the Complex TCG and Pokémon Environment

Pokémon Trading Card Game Pocket is a mobile adaptation of the Pokémon TCG that allows users to collect cards and battle players worldwide. Since its launch in October 2024, it surpassed a major milestone in global cumulative downloads as of May 2026. The game involves building a 20-card deck consisting of 'Pokémon' placed on the field and 'Trainer' cards (items, tools, supports) to assist in battle. Players engage in 1v1 matches, earning one point for every opponent's Pokémon they knock out, with the first to reach three points winning the game.

While this format is familiar to veteran card game and Pokémon players, it hides vast and complex logic from the perspective of reinforcement learning, which relies on trial and error. Beyond the individual logic of each card, there are complex interactions, such as 'Metamon,' which copies the moves of the opponent's active Pokémon.

Furthermore, outcomes are not fixed due to probabilistic elements, such as damage determined by coin tosses or the 'Monster Ball' card, which randomly adds a Basic Pokémon from the deck to the hand. Additionally, the AI must make decisions under conditions of imperfect information, where it cannot see the opponent's hand or the order of cards in either player's deck.

The battle AI developed for this environment is used for CPU opponents and for automated battles using a player's deck. It needed to handle not only the approximately 540 pre-defined decks but also any combination a user might create. Rule-based AI, which judges pre-defined conditional branches, struggled with the near-infinite combinations. Search-tree AI, which expands nodes and evaluates outcomes to choose the best move, was also impractical due to the game's imperfect information, forcing reliance on heuristics for state evaluation.

A greater challenge was the live service nature of the game, which requires regular additions of new cards. Traditional methods would lead to an exponential increase in the effort required for code management and synergy verification with existing cards, making RL AI essential. Because RL AI learns through trial and error, it adapts flexibly when cards are added or rules change, significantly reducing the need for manual code modifications.

Additionally, while search trees have inconsistent node expansion and computation times, a trained neural network offers the advantage of constant computation regardless of the complexity of the state. Furthermore, by automatically running a vast number of matches, the AI could discover game logic errors and rare edge-case bugs.

How Can an RL Agent Handle Gameplay Involving Probability and Evolution?

To implement reinforcement learning, the team had to design the AI model architecture, learning infrastructure, and data flow. Sakimi initially designed a system where the game engine converts the current state into features and passes them to the battle AI, which then decides on an action and returns it to the engine.

The features provided by the engine included global information like turn count and stadium cards, as well as card information (HP, energy, moves) and player data (points, hand/deck size). Private information held by the opponent was masked to prevent it from overly influencing the AI's inference.

This created a loop where the engine executes the action, updates the field, and provides new features until the match ends. After the match, the AI's decision-making criteria were updated by providing feedback on how much each action improved the state evaluation (expected win rate).

The team utilized the open-source library 'HandyRL,' which features support for multi-agent/imperfect information, high sample-efficiency off-policy learning, and multi-instance distributed processing. The resulting RL AI completed 50 million episodes (matches) per model, with each episode taking an average of one second. Computing resources included six t2d-standard-32 machines distributed across three regions to mitigate preemption risks. The model used a Transformer architecture with approximately 2 million parameters and adopted a stateless structure that does not rely on past action history.

For the deck-building process, basic Pokémon were grouped as candidates, with pre-evolution and post-evolution forms treated as a single unit. To account for card synergy, energy requirements were integrated and adjusted, and scores were assigned to ensure complex or high-energy cards were frequently included in decks, encouraging the AI to learn how to use difficult cards. Random candidates were then extracted based on these scores to complete the deck. The processes of candidate creation, updating, and prioritization were pre-calculated to boost speed during actual generation.

Card synergy also required management. For example, 'Rookie Researcher' is a card that returns a 'Mew ex' from the field to the hand; it is useless without a Mew ex. To address this, synergies were managed in a parent-child tree structure. Mew ex, which can be included in any deck, was set as the parent, and the child (Rookie Researcher) would only appear in the selection pool if the parent was included in the deck.

Decomposing Complex Rules and Operations for Step-by-Step Reinforcement Learning

Even so, there were further hurdles. For instance, 'Evolution' involves a broad range of actions: selecting the evolution process, choosing the target (e.g., Pikachu), and deciding on the evolution form (e.g., Raichu). Bundling these into a single action is fine for simple cases, but as the number of possibilities grows, the 'action space'—the set of all possible actions in a given state—explodes, making learning impossible.

To address this, Sakimi decomposed each action into the smallest possible units, 'atomic actions,' to be selected sequentially. For evolution, the selection of the action, the target, and the evolution form were separated. An 'Action Builder' was placed between the RL AI and the game engine to act as an intermediary. When the engine passes the state to the AI via the Action Builder, the AI selects the 'Evolution' atomic action. The Action Builder then determines if the action is complete; if not, it prompts the AI to process the next step, such as selecting the pre-evolution Pokémon. Once the process is complete, the Action Builder bundles them into a single 'Evolution Action' and sends it to the game engine.

This sequential mediation prevented the action space from expanding and allowed inference to proceed step-by-step, while the game engine only updated upon receiving a completed action, reducing system conflicts. It also narrowed the scope of logic. However, because the field did not update during the selection of atomic actions, a 'sub-evaluation' problem occurred where the change in evaluation value was calculated as zero.

To solve this, the team established a calculation method where the evaluation improvement at the point the final action was completed and the field was updated was applied as identical target feedback to all atomic actions that constituted that move. For steps where the field was not updated, they used a TD error formula that did not apply gamma and lambda discount rates.

Meanwhile, in obvious situations where only one atomic action was available, the engine performed the action automatically without AI inference. This accelerated episode generation, reduced inference computation, and excluded unnecessary states from the learning data, effectively lengthening the look-ahead (TD steps).

The battle AI model was also structurally separated into a model for processing field states and a model for selecting atomic actions. This allowed the reuse of field inference results, which do not change during atomic action selection, drastically reducing computational costs.

Refining the RL AI Through Constraints and Improvements

In development, theoretical implementations often face practical issues. The RL AI was no exception. The first issue identified was unnatural card usage. For example, 'Speed Up' reduces the energy required for a Pokémon to retreat by 1. However, the AI would play the card but then fail to retreat that turn.

Analysis revealed that, unlike LLMs, the RL AI does not read or understand card text but learns purely from trial-and-error data. Combined with the game's 3500 cards, probabilistic elements like coin tosses, and delayed rewards, the AI struggled when the balance between correct and incorrect usage data was disrupted.

The team modified the loss function by adding a penalty term. Previously, applying a direct negative reward caused the AI to perceive the state itself as 'bad.' For example, 'Windie ex' deals 120 damage to the opponent but 20 to itself. If a negative reward was given to prevent using the move when HP is 20 or less, the AI might mistakenly perceive the presence of Windie ex on the field as inherently bad. Instead, they added a penalty term that suppresses only the self-destructive action without affecting the overall state evaluation.

For cards like 'Speed Up,' the team increased their probability of inclusion during deck generation to ensure the AI encountered them more often. They also added auxiliary information to the features, such as whether retreat energy was actually reduced, to assist in conditional judgment. If a specific combination occurred—like using Speed Up but not retreating—a negative reward was immediately fed back to lower the selection probability. Following these adjustments, the rate of correct card usage (leading to a retreat after using Speed Up) rose significantly to 96.2%.

While RL AI can adapt flexibly once built, it requires careful adjustment to account for complex processes. Sakimi is now considering the next steps for the AI, such as regret minimization methods to mitigate delayed rewards, and 'AlphaZero'-style approaches that compare and learn from various scenario distributions within the same state without extra search. He concluded the lecture by stating that finding ways to increase efficiency through such research is the current challenge for engineers.