Learning to Beat ByteRL: Exploitability of Collectible Card Game Agents
Radovan Haluska Martin Schmid


fancy


Deep Hard X-ray Survey of the M81 Field Based on INTEGRAL Data
    
R. A. Krivonos1krivonos@cosmos.ru,
I. A. Mereminskiy1,
S. Yu. Sazonov1,
1Space Research Institute, Russian Academy of Sciences, Moscow, 117997 Russia

    
===========================================================================================================================================================





§ INTRODUCTION



Games have been at the forefront of artificial intelligence research since the very
beginning <cit.>. They provide a well-defined set of rules for players to follow,
yet they are complex enough to pose a challenge for humans and computer agents alike. Furthermore,
they allow efficient software implementations that are currently crucial for pushing the boundaries
of artificial intelligence. The field of artificial intelligence has seen many successful attempts
at beating humans in board games, such as Checkers <cit.>,
Chess <cit.>, Go <cit.>,
Shogi <cit.> and many others. These games have one thing in common: they
are games of perfect information, meaning that both players have access to complete information
about the game at any time. In the real world, we rarely have access to the whole picture. More
often than not, we are working with partial information. Games with this characteristic feature
have seen fewer successful attempts at beating humans than their perfect information counterparts.
The most notable successes are beating humans in Limit Heads-up Texas Hold'em
Poker <cit.>, No-Limit Heads-up Texas Hold'em Poker <cit.>, Stratego <cit.> and
Diplomacy <cit.>.

A collectible card game (CCG) is an imperfect information card game that mixes two stages:
a deck-building or a drafting stage and a battle stage. In the drafting stage, each player builds a
deck of cards unknown to the opponent, which is then used against their opponent in the battle
stage. The goal of the game is to decrease the opponent's health to zero. There are many popular
collectible card games, such as Magic: The Gathering <cit.>, Hearthstone <cit.>,
The Elder Scrolls: Legends <cit.> and many others. A trait that makes collectible
card games appealing to human players and challenging for AI agents is the broad range of ways to
mix and match available cards into decks. Even small collectible card games with tens of available
cards can offer more potential decks than the total number of atoms in the
universe <cit.>. Despite the popularity of
Hearthstone,[https://activeplayer.io/hearthstone] which receives substantially more
traffic than the world's largest Poker site (PokerStars), collectible card games received very
little attention from AI researchers in previous years.

This paper looks at a collectible card game named Legends of Code and Magic
(LOCM) <cit.> and its state-of-the-art agent
ByteRL <cit.>. Legends of Code and Magic is a game that was
specifically created to promote AI research in collectible card games. It is loosely inspired by
The Elder Scrolls: Legends <cit.>, but it is much smaller than the most
popular CCGs, allowing quick iteration of ideas in the research phase. Even though LOCM makes a
couple of simplifying assumptions, such as no randomness in the cards, researching and training
successful agents in LOCM translates to successful agents in Hearthstone, as shown by the team at
ByteDance <cit.>. We show that ByteRL is easily
exploitable in specific cases by training an adversarial agent using behaviour cloning and
subsequently fine-tuning it with the help of reinforcement learning.

In the rest of the paper, we look at the preliminary results of our experiments in which we search
for the best response against ByteRL. Section <ref> briefly introduces
reinforcement learning and behaviour cloning, and it describes Legends of Code and Magic in greater
detail. Section <ref> provides an overview of the literature and approaches
that have been applied to LOCM and other CCGs, focusing primarily on ByteRL.
Section <ref> describes the setup of the first part of our experiments
and comments on the results achieved so far. Section <ref> continues with
the second part of our experiments that focus on RL training. Lastly,
Section <ref> concludes the paper with final remarks and future plans.





§ BACKGROUND



This section briefly describes reinforcement learning and behaviour cloning and provides pointers
to the literature for a more in-depth treatment of the topics. The rest of the section introduces
Legends of Code and Magic. It describes the two available versions, 1.2 and 1.5, and discusses
their differences. Finally, it dives deeper into LOCM 1.5, which is the main focus of this paper.



 §.§ Reinforcement Learning



We consider partially observable sequential decision tasks modelled by Partially Observable Markov
Decision Processes. Partially Observable Markov Decision Process is a six-tuple (S, A, P, R,
Ω, O), where S is the set of states, A is the set of actions, P: S × A × S →ℝ describes conditional transition probabilities, R: S × A →ℝ is the
reward function, Ω is the set of observations, and O: S × A ×Ω→ℝ describes the probability of an observation. At each timestep t, an agent is in state
s_t ∈ S, it observes observation o_t ∈Ω, and based on this observation, it selects
action a_t ∈ A. After executing action a_t ∈ A, it transitions into a new state s_t+1∈ S according to P, receives a new observation o_t+1∈Ω according to O, and
finally it receives a reward r_t+1∈ℝ according to R. This process repeats for t
= 1 … T, where T is the maximum length of an episode.

An agent acting in an environment follows a policy π(· | s_t) that returns a probability of
taking each valid action in state s_t. The goal of the agent is to find an optimal policy π^*
that maximizes the expected return G_t = ∑_k=t^T r_k.

Agent's interaction with the environment can be summarized as a trajectory. Internally, a
trajectory is a sequence of states and actions; however, from the perspective of an agent acting in
the environment, a trajectory is a sequence of observations and actions, where each state is a
complete snapshot of the environment at a specific moment, each observation is a snapshot of the
environment available to the agent and each action is the agent's response to the observation it
received. A more in-depth treatment of the topics presented here is given in the book by Sutton and
Barto <cit.>.

We model Legends of Code and Magic, a two-player zero-sum game, as a POMDP by fixing the opponent
and thus viewing it as part of the environment. Fixing the opponent makes the environment
single-agent from our perspective. Finding the optimal policy in such an environment corresponds to
the best response against the opponent in the original
environment <cit.>.



 §.§ Behaviour Cloning



Behaviour cloning is the most straightforward approach to imitation learning, which falls under the
category of offline reinforcement learning. In offline RL, we do not interact with an environment,
either because it is prohibitively expensive or dangerous, for example, in robotics or autonomous
driving <cit.>. Instead, we collect samples from expert demonstrations and
store them in a static dataset as trajectories prior to learning. In the simplest form, each
trajectory contains a sequence of state-action pairs.

Behaviour cloning aims to copy the target behaviour using supervised
learning <cit.>. We treat each state as input and each action as either a
class label in the case of discrete actions or as a real number in the case of continuous actions.
We learn the new policy by minimizing the appropriate loss function between the target's policy
actions and the new policy's outputs.



 §.§ Legends of Code and Magic



Legends of Code and Magic (LOCM) <cit.> is a two-player collectible
card game created to promote AI research. Like many CCGs, it consists of two stages: a draft stage
and a battle stage. In the draft stage, both players build their decks from a pool of available
cards. Once both players have built their decks, the game proceeds with the battle stage, in which
the players use their decks to decrease the opponent's health to zero. Unlike some other CCGs, card
effects in LOCM are completely deterministic with no random effects. The only non-determinism
during the game arises from the deck ordering at the beginning of the battle stage and partial
observability.

Two versions of the game are available online, version 1.2 and 1.5. The versions have a couple of
differences, but the main difference lies in the draft stage. Version 1.2 uses a fixed pool of 160
cards, whereas version 1.5 procedurally generates a pool of 120 cards before each game. Both
versions split the draft stage into 30 rounds. In LOCM 1.2, a player is presented with three cards
in each round and selects a single card with no restrictions. In LOCM 1.5, all 120 cards are
presented together, and a player chooses a single card in each round, with the restriction that the
final deck may contain at most two copies of the same card.

Even though Legends of Code and Magic is considered a small CCG compared to Magic: The Gathering or
Hearthstone, it is by no means a small game. LOCM 1.2 contains 160 available cards and has
approximately (160 × 159 × 158)^30≈ 1.33 × 10^198 available
decks <cit.>. This number is significantly larger than the number of atoms
in the universe. For comparison, Hearthstone, as of March 2024, features roughly 4500 cards. Even
though the number of possible decks in LOCM 1.2 is tremendous, simple rule-based agents and
heuristics based on card ordering were enough to reduce the game only to the battle stage. LOCM 1.5
and its procedural generation of cards changed that. The number of possible decks is even larger
and practically infinite, and the agents now must learn to generalize and deal even with unbalanced
cards (e.g. lethal cards with zero cost).

Our work currently focuses on version 1.5 of the game, so we shall describe what a typical game in
this version looks like. The game starts with the draft stage and proceeds according to the rules
above. In each timestep, each player receives an observation vector 2040 numbers long and chooses a
single card from the pool according to the rules by returning a number between 0 and 119. After the
30 timesteps, the draft stage concludes, and the game progresses to the battle stage. The player's
decks are randomly shuffled. The first player is dealt four cards, and the second player is dealt
five cards from their respective decks. The players take turns for up to 50 rounds or until one of
the player's health decreases to zero. Each player receives an observation vector 244 numbers long
and can perform multiple actions in each round, followed by a pass action to conclude their turn.
Actions are encoded by numbers 0 through 144. Every move receives an immediate reward of 0, except
for the last move, which receives either +1 when a player wins the game or -1 when a player loses
the game.

The original Java implementation of the game and reimplementations in Rust and Nim can be found
online <cit.>. Additionally, the game has been rewritten in Python as an OpenAI Gym
environment <cit.>.





§ RELATED WORK



In this section, we discuss prior work that has been done on collectible card games, mainly on
Legends of Code and Magic and Hearthstone. We divide this section into three subsections. The first
subsection summarises the Strategy Card Game AI Competition, which used Legends of Code and Magic
and was organized by the authors of the game. The second subsection focuses on research done in
Legends of Code and Magic, and Hearthstone. The last subsection is dedicated to the
state-of-the-art agent in Legends of Code and Magic <cit.>, and
Hearthstone <cit.>.



 §.§ Strategy Card Game AI Competition



Legends of Code and Magic initially started as an online challenge on the CodinGame platform in
2018. The first two online competitions attracted hundreds of players. The creators of Legends of
Code and Magic also organized a series of competitions between 2019 and 2022 named Strategy Card
Game AI Competition, which took place at the COG and CEC conferences organized by IEEE. They have
written a comprehensive paper detailing everything there is to know about LOCM, including
descriptions of different versions of the game and descriptions of every COG and CEC
participant <cit.>. They have also publicly released the source codes
that were submitted to the competitions on their GitHub <cit.>.

We shall briefly highlight the most frequent and the most important approaches that were submitted
to the competitions. As most of the participants did not write a paper detailing their approach, we
used the LOCM summary paper <cit.> and the submitted code to figure out
the details.

During the earlier versions, including LOCM 1.2, participants mainly used handcrafted rules, such
as fixed deck ordering for the draft stage or lethal move detection during the battle stage.
Winning participants used different search techniques, such as Minimax with depth-limited search
and/or alpha-beta pruning or Monte Carlo Tree Search <cit.>. Starting
at CEC 2020, the first neural network-based agent appeared in the competition. The agent used two
neural networks during the draft stage, one for each side trained by self-play. During the battle
stage, the agent used a best-first search with handcrafted rules optimized using Bayesian
optimization <cit.>. Another notable approach that won COG 2021 was
using a flat simulation-based algorithm.

COG 2022 was the only competition to use the newer version of LOCM – 1.5. This new version brought
a completely redesigned draft stage, rendering fixed deck ordering approaches unusable. Neural
network-based submissions dominated this competition. The approaches used mainly reinforcement
learning agents utilizing two-stage training where an agent is trained separately on each stage.
Some used Q-learning combined with best-first search, while others used PPO. ByteRL was the only
agent that used end-to-end training and won this competition by a large margin. We describe ByteRL
in more depth in Section <ref>.



 §.§ Collectible Card Games



Although most of the work on Legends of Code and Magic was done as part of the SCGAI Competition,
there have also been numerous papers on LOCM. Kowalski et al. <cit.>
explored a variant of the evolutionary algorithm that uses a concept of active genes in the draft
stage of LOCM 1.2. Miernik et al. <cit.> looked at evolving evaluation
functions for CCGs using genetic algorithms and genetic programming techniques. Yang et
al. <cit.> also evolved various scoring functions using genetic algorithms,
compared the results against existing agents and analyzed the resulting decks in LOCM 1.2. Vieira
et al. published two papers that used self-play reinforcement learning to train agents for the
draft stage <cit.> and for the battle
stage <cit.> in LOCM 1.2. Vieira et
al. <cit.> also explored self-play reinforcement learning, reward
shaping and augmentation of features by adding information about the deck to the state
representation in LOCM 1.5.

Some research has also been conducted directly on Hearthstone. The Hearthstone AI
Competition <cit.> took place between 2018 and 2020. It featured two
tracks, both of which focused on the battle stage: 1) "pre-made deck" playing, which used three
publicly known decks and three unknown decks, and 2) "self-made deck" playing in which players had
to come up with their own deck ahead of time. Most of the submissions again used various search
techniques, evolutionary algorithms, heuristics or handcrafted rules. Janusz et
al. <cit.> organized the AAIA'17 Data Mining Challenge to develop a state
evaluation function for Hearthstone. The goal was to predict the probability of winning given only
a single game state generated from Hearthstone matches between two random players. Hoover et
al. <cit.> outlined many AI challenges Hearthstone poses, such as playing
the game to win, playing in a specific style, helping beginners to learn, helping professionals to
improve, balancing the decks and more. Bhatt et al. <cit.> explored the
Hearthstone deck space with evolutionary strategies by focusing on the deck-building phase.
Fontaine et al. <cit.> used an evolutionary approach in the form of a
modified MAP-Elites algorithm to create and balance decks in Hearthstone. Zhang et
al. <cit.> extended MAP-Elites further using deep surrogate modelling and
applied it to deck-building in Hearthstone. Silva et al. <cit.> used fuzzy ART
adaptive neural networks and trained an agent for the battle stage in Hearthstone. Xia et
al. <cit.> combined language modelling and self-play reinforcement learning to
train a battle-stage agent capable of outperforming the winning agent from Hearthstone AI
Competition 2020. Finally, Sakurai et al. <cit.> took a different approach to
beating some of the agents from the Hearthstone AI Competition using a modified version of the
Rolling Horizon Evolutionary Algorithm.



 §.§ ByteRL



ByteRL <cit.> is the state-of-the-art agent submitted to the last LOCM
competition held in 2022. It is the first agent that views both stages in a unified manner. It is
trained end-to-end, meaning that a single trajectory contains states from both the draft stage and
the battle stage. The authors proposed Optimistic Smooth Fictitious Self-play (OSFP) to train
ByteRL, and unlike other agents capable of playing imperfect information games, it does not use
search. Agents using online search algorithms enumerate the world state consistent with the current
agent's information state. The number of such states in collectible card games is not tractable,
and any algorithm that explicitly needs to work with all the states is not feasible.

Optimistic Smooth Fictitious Self-play is an iterative algorithm using Smooth Best
Response <cit.> and reinforcement learning algorithms as building
blocks to find Nash Equilibria. It is an extension of the Smooth Fictitious Self-play
algorithm <cit.>. While in Smooth FSP the actual sequence of policies
does not converge to a Nash Equilibrium (only the sequence of average policies does), the authors
claim that in OSFP the actual sequence of policies converges to a Nash Equilibrium. For a more
in-depth explanation, we refer the reader to their paper <cit.>. To find the
smooth best response, ByteRL uses a policy gradient reinforcement learning algorithm as a
sub-solver. Specifically, they use the V-trace <cit.> algorithm with an
auxiliary UPGO loss <cit.>.

ByteRL uses a fairly complex neural network in the background. During the draft stage, all
available cards are embedded using a one-dimensional convolution layer, concatenated with already
selected cards and passed through another one-dimensional convolution layer. Subsequently, action
masking is applied and passed to the output head. For the battle stage, various features, such as a
player's hand, a player's deck, all cards on the table and other public information, are again
embedded using one-dimensional convolutions, concatenated and passed through three fully-connected
layers. Afterwards, a single LSTM layer combines the current output of the fully-connected layers
with the hidden state carried through the whole battle stage. Finally, as in the draft stage,
action masking is applied and passed to the output head.





§ BEHAVIOUR CLONING







In this section, we describe our experiments with behaviour cloning and present the results we
obtained. Although we could have used ByteRL with its existing weights that are available
online <cit.> and fine-tuned it further, we chose to start from scratch and use
techniques that work even against black-box agents. It is important to note that although LOCM is a
two-stage game consisting of a draft stage and a battle stage, we have only focused on the battle
stage so far. The draft stage in LOCM 1.5 is not as straightforward to learn as the battle stage,
and it requires more engineering effort. Thus, whenever we evaluate an agent in the experiments, we
use ByteRL to perform the draft, providing us with a strong deck, and then we use the agent to
battle ByteRL with the drafted deck.

We started our experiments by using supervised learning to pre-train a policy network to mimic the
behaviour of ByteRL. We collected over 125 thousand matches between two instances of ByteRL playing
against each other. These matches produced roughly 3.5 million state-action pairs. We used 110
thousand matches (3 million state-action pairs) as our training data and the remaining 15 thousand
matches (500 thousand state-action pairs) as our validation data. The behaviour cloning experiments
were trained for 512 epochs using the same training and evaluation data. We used the Adam optimizer
with the default learning rate of 0.001 and a batch size of 256 to optimize the categorical
cross-entropy loss. We ran a periodic evaluation against ByteRL for 100 matches every 64 epochs. At
the end of every training, we restored the best-performing checkpoint and evaluated it against
ByteRL in 1000 matches.



 §.§ Filtering out the Pass Actions



As every game contains many pass actions, either to end the turn or to select the first possible
action to play when in doubt, we trained two identical neural networks, with one neural network
using all of the data and the other using only non-pass actions. Both neural networks consisted of
one hidden layer with 128 neurons.

Filtering out the pass actions reduced the number of state-action pairs by roughly 25%, from 3
million to 2.3 million in the training data and from 500 thousand to 330 thousand in the evaluation
data. The two experiments named  and  in
Figure <ref> show the win rates achieved in the 1000 evaluation
matches against ByteRL. Filtering out the pass actions increased the win rate by almost 7%.
Figure <ref> shows the accuracy achieved on the training and
validation sets and how the evaluation results against ByteRL developed throughout training for the
experiment .

As the environment automatically adds a pass action at the end of the list of actions returned by
an agent when needed, there is no need to keep training the future agents on datasets containing
pass actions. Thus, every subsequent experiment in this paper filters out the pass actions.



 §.§ Normalization & Standardization



The next batch of experiments looked at the preprocessing stage of the training pipeline. We tried
the following approaches: 1) using the raw input data without any preprocessing, 2) normalizing the
input data to the [0, 1] interval, and 3) standardizing the input data to have zero mean and unit
variance.

This time, we trained two architectures. The first one is the same as in the previous experiment: a
single hidden layer with 128 neurons. The second one is a scaled-up version of the first one. We
increased the number of hidden layers to two, and the layers contained 256 and 128 neurons,
respectively.

Looking at Figure <ref>, experiment  represents
the smaller neural network with normalization, and  represents the same neural
network with standardization. Both neural networks achieved a lower win rate than the neural
network that uses raw input data ().

The following three experiments are the larger networks with no processing (),
normalization () and standardization (), respectively. As we can
see, the results are the same as with the smaller network. No preprocessing achieved the best win
rate among the three networks, with normalization coming in second, slightly better than
standardization. Moreover, the larger network with no preprocessing achieved the best win rate
overall – 42.4%. It is important to note that we have achieved this win rate with no recurrence
and no memory. Figure <ref> shows the training and evaluation
curves for this network.



 §.§ Scaling up the Networks



Having decided on filtering out the pass actions and no preprocessing, we scaled the neural
networks in both the depth and the width even more. The results of these experiments are shown in
the remaining two bars. Making the network wider () resulted in an inferior win
rate compared to our best-performing network, and we started noticing overfitting on the validation
data quite early in the training; see Figure <ref> for details.
We saw the same problem with the deeper network (), which consisted of three
layers with 256, 128 and 64 neurons, respectively.

The future plans in this area are the following. We have yet to see whether increasing the training
data size would prevent overfitting and allow training larger networks and whether it would
increase the win rate. We also plan to experiment with more complex architectures, such as
recurrent neural networks, optionally combined with one-dimensional convolutions. Lastly, we plan
to include a second head for value function prediction, which is currently trained from scratch
during reinforcement learning.





§ REINFORCEMENT LEARNING FINE-TUNING



Although behaviour cloning yielded a stronger agent than we had initially expected, it still did
not match the performance of ByteRL. Thus, we started running another batch of experiments focusing
on reinforcement learning fine-tuning to learn the best response to ByteRL. Using reinforcement
learning is a common way to find an approximate best response to an agent in large games, where the
explicit computation of the best response is not tractable <cit.>. In these experiments, we start with our best pre-trained network
and fine-tune it with the help of reinforcement learning. To justify the pre-training phase, we
compare the agents trained starting from the pre-trained network and agents trained from scratch.

All of the experiments in this section use the Proximal Policy Optimization (PPO)
algorithm <cit.> from the RLlib library <cit.>. The policy network
is initialized with the weights of the network trained using behaviour cloning that achieved the
highest win rate (). The value network's weights are initialized randomly, and the
two networks do not share any weights. We periodically evaluate our agents for 100 matches every 20
training iterations. During evaluation, instead of sampling the action to take, we always take the
one with the highest probability. Evaluation happens at least five times during training, and we
stop the training at either 1000 iterations or when the average evaluation win rate from the last
five evaluation rounds surpasses 75%. We use the default hyperparameters for PPO, except for the
increased batch sizes, to better utilize the hardware and an added entropy regularizer with a
coefficient of 0.01.



 §.§ Training on Fixed Decks





For our initial experiments, we decided to simplify the game by limiting the number of procedurally
generated deck pools. Fixing the number of unique deck pools allows us to control the game's
complexity and gradually increase the number of deck pools, reaching the full game at the end
without affecting the rules in any way. We ran several experiments with deck pool sizes of up to
1024 different deck pools.

To ablate the effect of the supervised pre-training, we ran two categories of experiments. The
policy network in the first category of experiments was initialized with the weights of the best
pre-trained network (), while in the second category, the policy network was
randomly initialized. Figure <ref> shows the averaged training curves,
and Table <ref> shows the final evaluation results of these experiments.
Each combination of the initialization method and the deck pool size was run with five different
seeds, and the mean performance and 95% confidence interval were computed. However, due to time
constraints, experiments whose average win rate over the last five evaluation rounds exceeded 75%
were terminated, resulting in some runs having different lengths. In such cases, the averages and
confidence intervals are computed with the available data, and the decrease in the number of runs
being averaged is depicted using vertical lines. Dotted lines are used for experiments starting
from the pre-trained weights, and dashed lines are used for experiments starting with random
weights.

We were able to fine-tune the best pre-trained agent and beat ByteRL on up to 512 decks and match
its performance on 1024 decks during training, and we beat ByteRL in all six cases during
evaluation rounds. In the experiments on up to 256 decks, the win rate surpassed the 50% mark in
less than 100 training episodes and reached the 75% mark in less than 500 training episodes. The
fifth experiment needed a little more time, but all five runs eventually surpassed the 50% mark,
and two of those reached the 75% mark before the end of training. The last experiment matched
ByteRL's performance during training and slightly outperformed ByteRL during evaluation.



We repeated the same experiments with random initialization, plotted the results in
Figure <ref> and showed the average evaluation win rates in
Table <ref>. Apart from the second experiment, where the two curves overlap,
the rest of the experiments clearly showed that behaviour cloning prior to RL training is
beneficial. Although the first four experiments again defeated ByteRL, they required significantly
more time. From the first three experiments, we can see that the time needed to reach the win rate
of 75% or more is more than twice that of experiments which started with the pre-trained weights.
The fourth experiment surpassed the 75% win rate only in one out of five runs. The fifth
experiment got on par with ByteRL, and the final experiment did not even reach the win rate of our
pre-trained network (42.4%). Finally, looking at Table <ref>, we see that
with 512 and 1024 deck pools, the gap between the two categories of experiments grew significantly.

The next step in our experiments with reinforcement learning is to use curriculum learning to
automatically increase the number of available deck pools during training when the win rate reaches
a certain threshold. Furthermore, we plan to repeat the same experiments once we address the
shortcomings of the behaviour cloning pre-training as outlined in
Section <ref>.





§ CONCLUSION



In this paper, we made the first steps in our work on collectible card games by focusing on one
simple collectible card game called Legends of Code and Magic. We put ByteRL, the state-of-the-art
agent in Legends of Code and Magic, to the test and saw that given a strong deck for the battle
stage, ByteRL's performance in the battle stage is highly exploitable, leaving space for further
improvement. We showed that simple behaviour cloning of ByteRL's policy yielded an agent that was
almost on par with ByteRL. Further fine-tuning led to an agent capable of matching or even
improving on ByteRL's performance on hundreds of decks. Lastly, we ran a small ablation study,
which showed that behaviour cloning prior to reinforcement learning fine-tuning is beneficial.
Although these results seem favourable, we are aware of the shortcomings of our preliminary
experiments, and we plan to continue working on them.

One of the most critical next steps is losing any dependence on ByteRL during the deck-building
stage, which requires training a separate network for the draft stage that can produce similarly
strong decks. Other steps will include continuing our work on both training phases, the behaviour
cloning phase and the fine-tuning phase. As mentioned in the paper, we plan to collect more
training data for the behaviour cloning phase, scale the networks even further, and see if that
increases the win rate against ByteRL. We also plan to experiment with different neural network
architectures, and lastly, we plan to train the value function during supervised training along the
policy network. In the RL fine-tuning phase, we plan to experiment with automatic curriculum
learning during the battle stage, where we automatically increase the number of deck pools during
training. We also plan to experiment with reinforcement learning during the draft stage.




    This publication was supported by Charles Univ. project UNCE 24/SCI/008.
    Computational resources were provided by the e-INFRA CZ project (ID:90254),
    supported by the Ministry of Education, Youth and Sports of the Czech Republic.




ACM-Reference-Format


