ch20AlphaZeroTicTacToe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "653089c9",
   "metadata": {},
   "source": [
    "# Chaper 20: Iterative Self-Play and AlphaZero in Tic Tac Toe\n",
    "\n",
    "\n",
    "\n",
    "***\n",
    "***“This neural network improves the strength of the tree search, resulting in higher quality\n",
    "move selection and stronger self-play in the next iteration.”***\n",
    "\n",
    "-- David Silver et al,  Mastering the game of Go without human knowledge (Nature 2017) \n",
    "***\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "In the previous chapter, you successfully developed a streamlined version of AlphaZero for the coin game. The essence of AlphaZero's approach lies in honing a gaming strategy through self-play and deep reinforcement learning exclusively, bypassing the need for guidance from human experts or any specific knowledge beyond the rules of the game. A distinguishing feature of AlphaZero, setting it apart from its predecessor, AlphaGo, is its utilization of the actor-critic technique. This method employs two distinct networks for output: the policy network, which forecasts the most advantageous subsequent actions, and the value network, which estimates the likely victor of the match. You acquired skills in training an actor-critic agent for the coin game and integrting it with Monte Carlo tree search (MCTS) to forge an AlphaZero agent. This AlphaZero agent is capable of flawlessly solving the game and executing impeccable moves in the coin game.\n",
    "\n",
    "In this chapter, we will guide you through the process of constructing an AlphaZero agent for Tic Tac Toe. However, it's important to recognize that in straightforward games like Tic Tac Toe or Connect Four, the contribution of the value network to enhancing game strategies is minimal. Although a value network eliminates the need to play out game simulations all the way till the terminal state, employing a neural network for game state evaluation can also be time-consuming. Therefore, the effectiveness of using a value network in improving game strategy remains an empirical question. In Chapter 18, we have demonstrated that forgoing the value network in favor of rolling out games all the way to their terminal states is the most effective strategy.\n",
    "\n",
    "Given these insights, we will not develop actor-critic networks for Tic Tac Toe or Connect Four. Instead, we will use the policy-gradient method for these games, which is advantageous because it involves only a single output network, making the training process more efficient.\n",
    "\n",
    "More precisely, in this chapter, you will learn to construct an AlphaZero agent for Tic Tac Toe by initially developing a policy gradient agent. We will then integrate this policy-gradient agent with Monte Carlo Tree Search (MCTS) to develop an AlphaGo agent. This agent will select child nodes based on recommendations from the policy-gradient network in addition to the rollout value, offering a strategic blend of guided exploration and evaluation.\n",
    "\n",
    "In this chapter, we develop an AlphaZero agent that is multifunctional, capable of managing both Tic Tac Toe and Connect Four. This adaptability minimizes code duplication and streamlines the process of applying the AlphaZero algorithm across a wider array of games. As we proceed to the next chapter to create an AlphaZero agent for Connect Four, our attention can be dedicated to developing the most effective policy gradient network without the concern of integrating it with MCTS to formulate an AlphaZero algorithm.\n",
    "\n",
    "\n",
    "Now back to training the AlphaZero agent in Tic Tac Toe. In the beginning, the policy gradient network is untrained and initialized with random weights. During the training phase, the policy gradient agent competes against a more advanced version of itself: the AlphaZero agent. As training progresses, both agents gradually improve their performance.\n",
    "\n",
    "This training approach differs from the one described in Chapter 14, where the policy gradient agent faced opponents with static strategies. In the current training regimen, the policy gradient agent competes against its evolving self. This dynamic scenario presents a unique challenge, as the agent effectively faces a \"moving target,\" complicating the training process. The opponent, in this case the AlphaZero agent, employs the policy gradient network to choose child nodes during the game. Given that this network is simultaneously being trained, improvements in the agent's capabilities directly enhance the opponent's strength, as it relies on the same policy network for making decisions in Monte Carlo Tree Search (MCTS) simulations.\n",
    "\n",
    "To address the challenge of this moving target, an iterative self-play methodology is employed. Initially, we keep the weights of the policy gradient network, as utilized by the AlphaZero agent, constant, while updating the weights within the policy gradient network itself. Once the policy gradient agent achieves a specified performance level, we conclude the first iteration and proceed to update the weights in the policy gradient network used by the AlphaZero agent. This process is repeated in successive iterations until the AlphaZero agent perfects its gameplay and consistently solves the Tic Tac Toe game."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a933d3",
   "metadata": {},
   "source": [
    "# 1. An AlphaGo Agent for Multiple Games"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44908b0",
   "metadata": {},
   "source": [
    "## 1.1. Select, expand, roll out, and backpropagate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "063509d0",
   "metadata": {},
   "source": [
    "```python\n",
    "def select(priors,env,results,weight):    \n",
    "    # weighted average of priors and rollout_value\n",
    "    scores={}\n",
    "    for k,v in results.items():\n",
    "        # rollout_value for each next move\n",
    "        if len(v)==0:\n",
    "            vi=0\n",
    "        else:\n",
    "            vi=sum(v)/len(v);\n",
    "        # scale the prior by (1+N(L))\n",
    "        prior=priors[0][k-1]/(1+len(v))\n",
    "        # calculate weighted average\n",
    "        scores[k]=weight*prior+(1-weight)*vi\n",
    "    # select child node based on the weighted average     \n",
    "    return max(scores,key=scores.get) \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74702e59",
   "metadata": {},
   "source": [
    "```python\n",
    "# expand the game tree by taking a hypothetical move\n",
    "def expand(env,move):\n",
    "    env_copy=deepcopy(env)\n",
    "    state,reward,done,info=env_copy.step(move)\n",
    "    return env_copy, done, reward\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32135136",
   "metadata": {},
   "source": [
    "```python\n",
    "# roll out a game till terminal state or depth reached\n",
    "def simulate(env_copy,done,reward):\n",
    "    # if the game has already ended\n",
    "    if done==True:\n",
    "        return reward\n",
    "    # select moves based on fast policy network\n",
    "    while True:\n",
    "        move=env_copy.sample()\n",
    "        state,reward,done,info=env_copy.step(move)\n",
    "        # if terminal state is reached, returns outcome\n",
    "        if done==True:\n",
    "            return reward\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dff8ab61",
   "metadata": {},
   "source": [
    "```python\n",
    "def backpropagate(env,move,reward,results):\n",
    "    # update results\n",
    "    if env.turn==\"X\" or env.turn==\"red\":\n",
    "        results[move].append(reward)\n",
    "    # if current player is player 2,\n",
    "    # multiply outcome with -1\n",
    "    if env.turn==\"O\" or env.turn==\"yellow\":\n",
    "        results[move].append(-reward)                  \n",
    "    return results\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88f81d44",
   "metadata": {},
   "source": [
    "## 1.2 AlphaZero for Tic Tac Toe and Connect Four"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb5f027c",
   "metadata": {},
   "source": [
    "```python\n",
    "def alphazero(env,weight,PG_net,num_rollouts=100):\n",
    "    # if there is only one valid move left, take it\n",
    "    if len(env.validinputs)==1:\n",
    "        return env.validinputs[0]\n",
    "    # get the prior from the PG policy network\n",
    "    if env.turn==\"X\" or env.turn==\"O\":\n",
    "        state = env.state.reshape(-1,9)\n",
    "        conv_state = state.reshape(-1,3,3,1)\n",
    "        if env.turn==\"X\":\n",
    "            priors = PG_net([state,conv_state])\n",
    "        elif env.turn==\"O\":\n",
    "            priors = PG_net([-state,-conv_state])  \n",
    "    if env.turn==\"red\" or env.turn==\"yellow\":\n",
    "        state = env.state.reshape(-1,42)\n",
    "        conv_state = state.reshape(-1,7,6,1)\n",
    "        if env.turn==\"red\":\n",
    "            priors = PG_net([state,conv_state])\n",
    "        elif env.turn==\"yellow\":\n",
    "            priors = PG_net([-state,-conv_state])          \n",
    "    # create a dictionary results\n",
    "    results={}\n",
    "    for move in env.validinputs:\n",
    "        results[move]=[]\n",
    "    # roll out games\n",
    "    for _ in range(num_rollouts):\n",
    "        # select\n",
    "        move=select(priors,env,results,weight)\n",
    "        # expand\n",
    "        env_copy, done, reward=expand(env,move)\n",
    "        # simulate\n",
    "        reward=simulate(env_copy,done,reward)\n",
    "        # backpropagate\n",
    "        results=backpropagate(env,move,reward,results)\n",
    "    # select the most visited child node\n",
    "    visits={k:len(v) for k,v in results.items()}\n",
    "    return max(visits,key=visits.get)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eee28a4e",
   "metadata": {},
   "source": [
    "# 2. A Blueprint to Train AlphaZero in Tic Tac Toe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c36e50de",
   "metadata": {},
   "source": [
    "## 2.1. Steps to Train AlphaZero in Tic Tac Toe\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e312e9d5",
   "metadata": {},
   "source": [
    "## 2.2. A Policy Gradient Network in Tic Tac Toe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b019c91c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "num_inputs = 9\n",
    "num_actions = 9\n",
    "# The convolutional input layer\n",
    "conv_inputs = layers.Input(shape=(3,3,1))\n",
    "conv=layers.Conv2D(filters=64, kernel_size=(3,3),padding=\"same\",\n",
    "     input_shape=(3,3,1), activation=\"relu\")(conv_inputs)\n",
    "# Flatten the output from the conv layer\n",
    "flat = layers.Flatten()(conv)\n",
    "# The dense input layer\n",
    "inputs = layers.Input(shape=(9,))\n",
    "# Combine the two into a single input layer\n",
    "two_inputs = tf.concat([flat,inputs],axis=1)\n",
    "# two hidden layers\n",
    "common = layers.Dense(128, activation=\"relu\")(two_inputs)\n",
    "action = layers.Dense(32, activation=\"relu\")(common)\n",
    "# Policy output network \n",
    "action = layers.Dense(num_actions, activation=\"softmax\")(action)\n",
    "# The final model\n",
    "model = keras.Model(inputs=[inputs, conv_inputs],\\\n",
    "                    outputs=action)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "68df75e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer=keras.optimizers.Adam(learning_rate=0.00025,\n",
    "                                clipnorm=1)   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbc0a431",
   "metadata": {},
   "source": [
    "# 3. Train AlphaZero in Tic Tac Toe\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09d6d496",
   "metadata": {},
   "source": [
    "## 3.1. Train Players X and O\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5b08faaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# An old policy gradient network that doesn't update \n",
    "old_model = keras.Model(inputs=[inputs, conv_inputs],\\\n",
    "                    outputs=action)\n",
    "\n",
    "from utils.ch20util import alphazero\n",
    "\n",
    "# define an opponent for the policy gradient agent\n",
    "weight=0.5\n",
    "num_rollouts=100\n",
    "def opponent(env):\n",
    "    move=alphazero(env,weight,old_model,\n",
    "                   num_rollouts=num_rollouts) \n",
    "    return move                        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4dd9f55b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.ttt_simple_env import ttt\n",
    "import numpy as np\n",
    "\n",
    "# allow a maximum of 50 steps per game\n",
    "max_steps=50\n",
    "env=ttt()\n",
    "\n",
    "def playing_X():\n",
    "    # create lists to record game history\n",
    "    action_probs_history = []\n",
    "    wrongmoves_history = []\n",
    "    rewards_history = []\n",
    "    state = env.reset()\n",
    "    for step in range(max_steps):\n",
    "        state = state.reshape(-1,9,)\n",
    "        conv_state = state.reshape(-1,3,3,1)\n",
    "        # Predict action probabilities and future rewards\n",
    "        action_probs = model([state,conv_state])\n",
    "        # select action based on the policy network\n",
    "        action=np.random.choice(num_actions,\\\n",
    "                                p=np.squeeze(action_probs))\n",
    "        # record log probabilities\n",
    "        action_probs_history.append(\\\n",
    "                        tf.math.log(action_probs[0, action]))\n",
    "        # punish the agent if there is an illegal move\n",
    "        if action+1 not in env.validinputs:\n",
    "            rewards_history.append(0)\n",
    "            wrongmoves_history.append(-1)\n",
    "        # otherwise, place the move on the game board\n",
    "        else:              \n",
    "        # Apply the sampled action in our environment\n",
    "            state, reward, done, _ = env.step(action+1)\n",
    "            if done:\n",
    "                wrongmoves_history.append(0)\n",
    "                rewards_history.append(reward)\n",
    "                break\n",
    "            else:\n",
    "                state,reward,done,_=env.step(opponent(env))\n",
    "                rewards_history.append(reward)\n",
    "                wrongmoves_history.append(0)              \n",
    "                if done:\n",
    "                    break                \n",
    "    return action_probs_history,\\\n",
    "            wrongmoves_history,rewards_history "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "15b9c0d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma=0.95\n",
    "def discount_rs(r,wrong):\n",
    "    discounted_rs = np.zeros(len(r))\n",
    "    running_add = 0\n",
    "    for i in reversed(range(0, len(r))):\n",
    "        if wrong[i]==0:  \n",
    "            running_add = gamma*running_add + r[i]\n",
    "            discounted_rs[i] = running_add  \n",
    "    return discounted_rs.tolist() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "053a34c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def playing_O():\n",
    "    # create lists to record game history\n",
    "    action_probs_history = []\n",
    "    wrongmoves_history = []\n",
    "    rewards_history = []\n",
    "    state = env.reset()\n",
    "    # the opponent moves first\n",
    "    state, reward, done, _ = env.step(opponent(env))\n",
    "    for step in range(max_steps):\n",
    "        state = state.reshape(-1,9,)\n",
    "        conv_state = state.reshape(-1,3,3,1)\n",
    "        # predict action probabilities; multiply the board by -1\n",
    "        action_probs = model([-state,-conv_state])\n",
    "        # select action based on the policy network\n",
    "        action=np.random.choice(num_actions,\\\n",
    "                                p=np.squeeze(action_probs))\n",
    "        # record log probabilities\n",
    "        action_probs_history.append(\\\n",
    "                        tf.math.log(action_probs[0, action]))\n",
    "        # punish the agent if there is an illegal move\n",
    "        if action+1 not in env.validinputs:\n",
    "            rewards_history.append(0)\n",
    "            wrongmoves_history.append(-1)\n",
    "        # otherwise, place the move on the game board\n",
    "        else:              \n",
    "        # apply the sampled action in our environment\n",
    "            state, reward, done, _ = env.step(action+1)\n",
    "            if done:\n",
    "                wrongmoves_history.append(0)\n",
    "                rewards_history.append(-reward)\n",
    "                break\n",
    "            else:\n",
    "                state,reward,done,_=env.step(opponent(env))\n",
    "                rewards_history.append(-reward)\n",
    "                wrongmoves_history.append(0)            \n",
    "                if done:\n",
    "                    break                \n",
    "    return action_probs_history,\\\n",
    "            wrongmoves_history,rewards_history"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a156ac04",
   "metadata": {},
   "source": [
    "## 3.2. Update Parameters in the Policy Gradient Network\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "052e7981",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size=10     \n",
    "def create_batch_X(batch_size):\n",
    "    action_probs_history = []\n",
    "    wrongmoves_history = []\n",
    "    rewards_history = []\n",
    "    for i in range(batch_size):\n",
    "        aps,wms,rs = playing_X()\n",
    "        # rewards are discounted\n",
    "        returns = discount_rs(rs,wms)\n",
    "        action_probs_history += aps\n",
    "        # punishments for wrong moves are not discounted\n",
    "        wrongmoves_history += wms\n",
    "        # combine discounted rewards with punishments\n",
    "        combined=np.array(returns)+np.array(wms)\n",
    "        # add combined rewards to rewards history\n",
    "        rewards_history += combined.tolist()\n",
    "    return action_probs_history,rewards_history  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b7a83221",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_batch_O(batch_size):\n",
    "    action_probs_history = []\n",
    "    wrongmoves_history = []\n",
    "    rewards_history = []\n",
    "    for i in range(batch_size):\n",
    "        aps,wms,rs = playing_O()\n",
    "        # reward related to legal moves are discounted\n",
    "        returns = discount_rs(rs,wms)\n",
    "        action_probs_history += aps\n",
    "        # punishments for wrong moves are not discounted\n",
    "        wrongmoves_history += wms\n",
    "        # combine discounted rewards with punishments\n",
    "        combined=np.array(returns)+np.array(wms)\n",
    "        # add combined rewards to rewards history\n",
    "        rewards_history += combined.tolist()\n",
    "    return action_probs_history,rewards_history  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6b0d413",
   "metadata": {},
   "source": [
    "## 3.3. The Training Loop\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9c62f391",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.ch06util import MiniMax_ab\n",
    "\n",
    "def rule_based_AI(env):\n",
    "    move = MiniMax_ab(env)\n",
    "    return move    \n",
    "\n",
    "def test():\n",
    "    results=[]\n",
    "    for i in range(10):\n",
    "        state=env.reset()     \n",
    "        while True: \n",
    "            action = rule_based_AI(env)  \n",
    "            state, reward, done, info = env.step(action)\n",
    "            if done:\n",
    "                results.append(-abs(reward))\n",
    "                break            \n",
    "            # AlphaZero moves\n",
    "            action=alphazero(env,weight,model,\n",
    "                   num_rollouts=num_rollouts)\n",
    "            state, reward, done, info = env.step(action)\n",
    "            if done:\n",
    "                results.append(abs(reward))            \n",
    "                break              \n",
    "    return results   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f3ec3d36",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "\n",
    "tests=deque(maxlen=100) \n",
    "n = 0\n",
    "batches=0\n",
    "# Train the model\n",
    "while True:\n",
    "    with tf.GradientTape() as tape:\n",
    "        if batches%2==0:\n",
    "            action_probs_history,\\\n",
    "    rewards_history=create_batch_X(batch_size)\n",
    "        else:\n",
    "            action_probs_history,\\\n",
    "    rewards_history=create_batch_O(batch_size)                    \n",
    "        rets=tf.convert_to_tensor(rewards_history,\\\n",
    "                                  dtype=tf.float32)\n",
    "        alosses=-tf.multiply(tf.convert_to_tensor(\\\n",
    "          action_probs_history,dtype=tf.float32),rets)\n",
    "        loss_value = tf.reduce_sum(alosses) \n",
    "    grads = tape.gradient(loss_value, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads,\\\n",
    "                                  model.trainable_variables))\n",
    "    n += batch_size\n",
    "    batches += 1\n",
    "    if n % 100 == 0:   \n",
    "        results=test()\n",
    "        tests += results\n",
    "        losses = tests.count(-1)\n",
    "        print(f\"at episode {n}, lost {losses} games\")\n",
    "        if (losses<=40 and n>=1000) or n>=10000:\n",
    "            print(f\"Finished at episode {n}!\")\n",
    "            break\n",
    "model.save(\"files/zero_ttt0.h5\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8fc850b1",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "for i in range(4):\n",
    "    weight=0.5+(i+1)*0.1\n",
    "    num_rollouts=100+(i+1)*100\n",
    "    reload=keras.models.load_model(f\"files/zero_ttt{i}.h5\")\n",
    "    model.set_weights(reload.get_weights()) \n",
    "    # update weights in the opponent\n",
    "    old_model.set_weights(reload.get_weights()) \n",
    "    tests=deque(maxlen=100) \n",
    "    n = 0\n",
    "    batches=0\n",
    "    # Train the model\n",
    "    while True:\n",
    "        with tf.GradientTape() as tape:\n",
    "            if batches%2==0:\n",
    "                action_probs_history,\\\n",
    "        rewards_history=create_batch_X(batch_size)\n",
    "            else:\n",
    "                action_probs_history,\\\n",
    "        rewards_history=create_batch_O(batch_size)                    \n",
    "            rets=tf.convert_to_tensor(rewards_history,\\\n",
    "                                      dtype=tf.float32)\n",
    "            alosses=-tf.multiply(tf.convert_to_tensor(\\\n",
    "              action_probs_history,dtype=tf.float32),rets)\n",
    "            loss_value = tf.reduce_sum(alosses) \n",
    "        grads = tape.gradient(loss_value, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(grads,\\\n",
    "                                      model.trainable_variables))\n",
    "        n += batch_size\n",
    "        batches += 1\n",
    "        if n % 100 == 0:   \n",
    "            results=test()\n",
    "            tests += results\n",
    "            losses = tests.count(-1)\n",
    "            print(f\"at episode {n}, lost {losses} games\")\n",
    "            if (losses<=40-(i+1)*10 and n>=1000) or n>=10000:\n",
    "                print(f\"Finished at episode {n}!\")\n",
    "                break\n",
    "    model.save(f\"files/zero_ttt{i+1}.h5\") "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cda387a5",
   "metadata": {},
   "source": [
    "# 4. Test AlphaZero in Tic Tac Toe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "13cfa062",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The game is tied!\n",
      "The game is tied!\n",
      "The game is tied!\n",
      "The game is tied!\n",
      "The game is tied!\n",
      "The game is tied!\n",
      "The game is tied!\n",
      "The game is tied!\n",
      "The game is tied!\n",
      "The game is tied!\n"
     ]
    }
   ],
   "source": [
    "# Use the trained PG model\n",
    "reload=keras.models.load_model(\"files/zero_ttt4.h5\")\n",
    "model.set_weights(reload.get_weights()) \n",
    "env=ttt()\n",
    "for i in range(10):\n",
    "    state=env.reset()  \n",
    "    if i%2==0:\n",
    "        action=rule_based_AI(env)  \n",
    "        state,reward,done,_=env.step(action)        \n",
    "    while True: \n",
    "        # AlphaZero moves\n",
    "        action=alphazero(env,0.9,model,\n",
    "                   num_rollouts=1000) \n",
    "        state,reward,done,_=env.step(action)     \n",
    "        if done:\n",
    "            if reward==0:\n",
    "                print(\"The game is tied!\")\n",
    "            else:\n",
    "                print(\"AlphaZero wins!\")\n",
    "            break      \n",
    "        # MiniMax agent moves\n",
    "        action=rule_based_AI(env)  \n",
    "        state,reward,done,_=env.step(action)\n",
    "        if done:\n",
    "            if reward==0:\n",
    "                print(\"The game is tied!\")\n",
    "            else:\n",
    "                print(\"AlphaZero loses!\")\n",
    "            break           "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}