ch19ACcoinAlphaZero.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "653089c9",
   "metadata": {},
   "source": [
    "# Chaper 19: The Actor-Critic Method and AlphaZero \n",
    "\n",
    "***\n",
    "***“As it plays, the neural network is tuned and updated to predict moves, as well as the eventual winner of the games.”***\n",
    "\n",
    "-- --- David Silver and Demis Hassabis, 2017\n",
    "***\n",
    "\n",
    "\n",
    "\n",
    "In 2017, the DeepMind team introduced an advanced version of AlphaGo, named AlphaGo Zero (which we'll refer to as AlphaZero in this book because we apply the algorithm to various games beyond Go). This version marked a significant departure from its predecessor, the AlphaGo that triumphed over World Go Champion Lee Sedol in 2016. Unlike the earlier version, AlphaGo Zero's training relied exclusively on deep reinforcement learning, without any human-derived strategies or domain-specific knowledge, except for the basic rules of the game. It learned through self-play from scratch. This development has profound implications for AI. When confronting complex global issues, human expertise and domain knowledge often have their limitations. The challenge lies in how AI can devise robust solutions under such constraints. The success of AlphaGo Zero, however, offers a beacon of hope in this endeavor.\n",
    "\n",
    "You may be curious about what sets AlphaZero apart from AlphaGo in terms of power. A key factor is AlphaZero's use of larger and more complex neural networks, which significantly boost its learning abilities. However, in the context of the simpler games covered in this book, the additional layers in the neural networks don't substantially increase benefits. Therefore, we'll not explore this aspect of AlphaZero in this book. \n",
    "\n",
    "Another element that has elevated AlphaZero above its predecessors is its application of an advanced deep reinforcement learning technique. As explored in the previous three chapters, the AlphaGo algorithm initially trains two policy networks using data from human experts. Following this, it employs self-play deep reinforcement learning to enhance one of these policy networks, creating a policy gradient network. It also leverages game experience data from self-play to train a value network for predicting game outcomes. In contrast, AlphaZero utilizes a singular neural network with two outputs: one *policy head* for predicting the next move and a *value head* for forecasting game results, as highlighted by David Silver and Demis Hassabis in this chapter's introductory quote.\n",
    "\n",
    "In this chapter, we'll delve into the advanced deep reinforcement learning strategy known as the actor-critic method, applying it specifically to the coin game. The concept of the *actor* in this method is straightforward: it's the game player who must determine the most advantageous next move. This actor is essentially the policy network we've examined in the policy-gradient methods from Chapters 13 to 15.\n",
    "\n",
    "The *critic* refers to a different role – akin to another player – who evaluates the moves made thus far. The insights provided by the critic are instrumental in refining future moves. To operationalize this, we've added an additional component to the deep neural network, resulting in two distinct outputs: a policy network for predicting the next move and a value network for estimating the game's outcome. The policy network assumes the actor's role, selecting the subsequent move, while the value network, as the critic, monitors the agent's progress throughout the game. This feedback is essential for the training process, serving a similar function to how a game review might enhance your own learning.\n",
    "\n",
    "We'll focus on applying the actor-critic method to the coin game in this chapter. You'll learn how to develop an Actor-Critic (AC) deep reinforcement learning model. This model will feature two concurrent outputs: the policy network, serving as the 'actor', and the value network, functioning as the 'critic'. Following this, you'll learn how to train this AC deep reinforcement model. Unlike our approach with the AlphaGo algorithm, where we used rule-based AI players as experts for generating training moves, here we'll start from scratch. We'll rely solely on self-play for training, without using expert moves as a guide.\n",
    "\n",
    "Once you've successfully trained the AC deep reinforcement model, you'll integrate both the policy and value networks with Monte Carlo Tree Search (MCTS) for making decisions in actual games, mirroring the approach we took with the AlphaGo algorithm in Chapter 16. You'll observe that the AlphaZero agent, armed with this model, performs flawlessly and solves the game. It can beat the AlphaGo agent we developed in Chapter 16 if playing as Player 2."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "282ad876",
   "metadata": {},
   "source": [
    "# 1. The Actor-Critic Method\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eee28a4e",
   "metadata": {},
   "source": [
    "# 2. An Overview of the Training Process\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c36e50de",
   "metadata": {},
   "source": [
    "## 2.1. Steps to Train An AlphaZero Agent\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e312e9d5",
   "metadata": {},
   "source": [
    "## 2.2. An Actor-Critic Agent for the Coin Game\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b019c91c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "num_inputs = 22\n",
    "num_actions = 2\n",
    "# The input layer\n",
    "inputs = layers.Input(shape=(22,))\n",
    "# The common layer\n",
    "common = layers.Dense(32, activation=\"relu\")(inputs)\n",
    "common = layers.Dense(32, activation=\"relu\")(common)\n",
    "# The policy network\n",
    "action = layers.Dense(num_actions, activation=\"softmax\")(common)\n",
    "# The value network\n",
    "critic = layers.Dense(1, activation=\"tanh\")(common)\n",
    "model = keras.Model(inputs=inputs, outputs=[action, critic])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "68df75e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam(learning_rate=0.001)\n",
    "loss_func = keras.losses.MeanAbsoluteError()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbc0a431",
   "metadata": {},
   "source": [
    "# 3. Train the Actor-Critic Agent in the Coin Game\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09d6d496",
   "metadata": {},
   "source": [
    "## 3.1. Train the Two Players in the Actor-Critic Model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5b08faaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def onehot_encoder(state):\n",
    "    onehot=np.zeros((1,22))\n",
    "    onehot[0,state]=1\n",
    "    return onehot\n",
    "\n",
    "def ACplayer(env): \n",
    "    # estimate action probabilities and future rewards\n",
    "    onehot_state = onehot_encoder(env.state)\n",
    "    action_probs, _ = model(onehot_state)\n",
    "    # select action with the highest probability\n",
    "    action=np.random.choice([1,2],p=np.squeeze(action_probs))   \n",
    "    return action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4dd9f55b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.coin_simple_env import coin_game\n",
    "env=coin_game()\n",
    "def playing_1():\n",
    "    # create lists to record game history\n",
    "    action_probs_history = []\n",
    "    critic_value_history = []\n",
    "    rewards_history = []\n",
    "    state = env.reset()\n",
    "    while True:\n",
    "        # estimate action probabilities and future rewards\n",
    "        onehot_state = onehot_encoder(env.state)\n",
    "        action_probs, critic_value = model(onehot_state)\n",
    "        # select action based on policy network\n",
    "        action=np.random.choice([1,2],p=np.squeeze(action_probs))\n",
    "        # record value history\n",
    "        critic_value_history.append(critic_value[0, 0])\n",
    "        # record log probabilities\n",
    "        action_probs_history.append(\\\n",
    "                    tf.math.log(action_probs[0, action-1]))      \n",
    "        # Apply the sampled action in our environment\n",
    "        state, reward, done, _ = env.step(action)\n",
    "        if done:\n",
    "            rewards_history.append(reward)\n",
    "            break\n",
    "        else:\n",
    "            state, reward, done, _ = env.step(ACplayer(env))\n",
    "            rewards_history.append(reward)               \n",
    "            if done:\n",
    "                break                \n",
    "    return action_probs_history,critic_value_history,\\\n",
    "        rewards_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "15b9c0d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def discount_rs(r):\n",
    "    discounted_rs = np.zeros(len(r))\n",
    "    running_add = 0\n",
    "    for i in reversed(range(0, len(r))):\n",
    "        running_add = gamma*running_add + r[i]\n",
    "        discounted_rs[i] = running_add  \n",
    "    return discounted_rs.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "053a34c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def playing_2():\n",
    "    # create lists to record game history\n",
    "    action_probs_history = []\n",
    "    critic_value_history = []\n",
    "    rewards_history = []\n",
    "    state = env.reset()\n",
    "    state, reward, done, _ = env.step(ACplayer(env))\n",
    "    while True:\n",
    "        # estimate action probabilities and future rewards\n",
    "        onehot_state = onehot_encoder(env.state)\n",
    "        action_probs, critic_value = model(onehot_state)\n",
    "        # select action based on policy network\n",
    "        action=np.random.choice([1,2],p=np.squeeze(action_probs))\n",
    "        # record value history\n",
    "        critic_value_history.append(critic_value[0, 0])\n",
    "        # record log probabilities\n",
    "        action_probs_history.append(\\\n",
    "                        tf.math.log(action_probs[0, action-1]))      \n",
    "        # Apply the sampled action in our environment\n",
    "        state, reward, done, _ = env.step(action)\n",
    "        if done:\n",
    "            rewards_history.append(-reward)\n",
    "            break\n",
    "        else:\n",
    "            state, reward, done, _ = env.step(ACplayer(env))\n",
    "            rewards_history.append(-reward)                \n",
    "            if done:\n",
    "                break                \n",
    "    return action_probs_history,critic_value_history, \\\n",
    "            rewards_history "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a156ac04",
   "metadata": {},
   "source": [
    "## 3.2. Update Parameters During Training\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "052e7981",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size=10\n",
    "def create_batch(playing_func):\n",
    "    action_probs_history = []\n",
    "    critic_value_history = []\n",
    "    rewards_history = []\n",
    "    for i in range(batch_size):\n",
    "        aps,cvs,rs = playing_func()\n",
    "        # rewards are discounted\n",
    "        returns = discount_rs(rs)\n",
    "        action_probs_history += aps\n",
    "        critic_value_history += cvs\n",
    "        rewards_history += returns       \n",
    "    return action_probs_history,critic_value_history,\\\n",
    "        rewards_history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8c6e8ed7",
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma=0.95\n",
    "def train_player(playing_func):\n",
    "    # Train the model for one batch (ten games)\n",
    "    with tf.GradientTape() as tape:\n",
    "        action_probs_history,critic_value_history,\\\n",
    "            rewards_history=create_batch(playing_func)\n",
    "        # Calculating loss values to update our network        \n",
    "        tfdif=tf.convert_to_tensor(rewards_history,\\\n",
    "                                   dtype=tf.float32)-\\\n",
    "        tf.convert_to_tensor(critic_value_history,dtype=tf.float32)\n",
    "        alosses=-tf.multiply(tf.convert_to_tensor(\\\n",
    "          action_probs_history,dtype=tf.float32),tfdif)\n",
    "        closs=loss_func(tf.convert_to_tensor(rewards_history,\\\n",
    "                                             dtype=tf.float32),\\\n",
    "     tf.convert_to_tensor(critic_value_history,dtype=tf.float32))\n",
    "        # Backpropagation\n",
    "        loss_value = tf.reduce_sum(alosses) + closs\n",
    "    grads = tape.gradient(loss_value, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads,\\\n",
    "                                  model.trainable_variables)) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6b0d413",
   "metadata": {},
   "source": [
    "## 3.3. The Training Loop\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9c62f391",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def rule_based_AI(state):\n",
    "    if state%3 != 0:\n",
    "        move = state%3\n",
    "    else:\n",
    "        move = random.choice([1,2])\n",
    "    return move\n",
    "\n",
    "def test():\n",
    "    results=[]\n",
    "    for i in range(100):\n",
    "        env = coin_game()\n",
    "        state=env.reset()     \n",
    "        while True:    \n",
    "            action = rule_based_AI(state)  \n",
    "            state, reward, done, info = env.step(action)\n",
    "            if done:\n",
    "                # record -1 if rule-based AI player won\n",
    "                results.append(-1)\n",
    "                break\n",
    "            # estimate action probabilities and future rewards\n",
    "            onehot_state = onehot_encoder(state)\n",
    "            action_probs, _ = model(onehot_state)\n",
    "            # select action with the highest probability\n",
    "            action=np.argmax(action_probs[0])+1\n",
    "            state, reward, done, info = env.step(action)\n",
    "            if done:\n",
    "                # record 1 if AC agent won\n",
    "                results.append(1)            \n",
    "                break  \n",
    "    return results.count(1)      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8fc850b1",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 100, number of wins is 3/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 200, number of wins is 3/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 300, number of wins is 1/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 400, number of wins is 2/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 500, number of wins is 4/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 600, number of wins is 2/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 700, number of wins is 0/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 800, number of wins is 0/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 900, number of wins is 1/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1000, number of wins is 2/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1100, number of wins is 1/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1200, number of wins is 1/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1300, number of wins is 2/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1400, number of wins is 3/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1500, number of wins is 1/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1600, number of wins is 1/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1700, number of wins is 3/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1800, number of wins is 3/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 1900, number of wins is 4/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2000, number of wins is 6/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2100, number of wins is 3/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2200, number of wins is 4/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2300, number of wins is 1/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2400, number of wins is 3/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2500, number of wins is 4/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2600, number of wins is 0/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2700, number of wins is 0/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2800, number of wins is 0/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 2900, number of wins is 4/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3000, number of wins is 3/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3100, number of wins is 5/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3200, number of wins is 5/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3300, number of wins is 5/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3400, number of wins is 6/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3500, number of wins is 12/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3600, number of wins is 9/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3700, number of wins is 22/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3800, number of wins is 0/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 3900, number of wins is 15/100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4000, number of wins is 21/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4100, number of wins is 27/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4200, number of wins is 15/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4300, number of wins is 30/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4400, number of wins is 24/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4500, number of wins is 35/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4600, number of wins is 45/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4700, number of wins is 54/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4800, number of wins is 56/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 4900, number of wins is 42/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 5000, number of wins is 56/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 5100, number of wins is 52/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 5200, number of wins is 48/100\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n",
      "at episode 5300, number of wins is 100/100\n"
     ]
    }
   ],
   "source": [
    "n = 0\n",
    "batches=0\n",
    "# Train the model\n",
    "while True:\n",
    "    if batches%2==0:\n",
    "        train_player(playing_1)\n",
    "    else:\n",
    "        train_player(playing_2)\n",
    "    # Log details\n",
    "    n += batch_size\n",
    "    batches += 1\n",
    "    # print out progress\n",
    "    if n % 1000 == 0:\n",
    "        model.save(\"files/ac_coin.h5\")        \n",
    "        wins=test()\n",
    "        print(f\"at episode {n}, number of wins is {wins}/100\")\n",
    "        if wins==100:\n",
    "            break     "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80e2ef29",
   "metadata": {},
   "source": [
    "The model is successfully trained after 5300 episodes. Alternatively, you can download the trained model in the book's GitHub repository."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cda387a5",
   "metadata": {},
   "source": [
    "# 4. An AlphaZero Agent in the Coin Game"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a933d3",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## 4.1. Select, Expand, Simulate, and Backpropagate\n",
    "In the local module *ch19util*, we first load up the trained actor-critic model, as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d4ea2c",
   "metadata": {},
   "source": [
    "```python\n",
    "from copy import deepcopy\n",
    "import numpy as np\n",
    "from tensorflow import keras\n",
    "\n",
    "# Load the trained actor critic model\n",
    "model=keras.models.load_model(\"files/ac_coin.h5\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25e0b0e2",
   "metadata": {},
   "source": [
    "```python\n",
    "def select(priors,env,results,weight):    \n",
    "    # weighted average of priors and rollout_value\n",
    "    scores={}\n",
    "    for k,v in results.items():\n",
    "        # rollout_value for each next move\n",
    "        if len(v)==0:\n",
    "            vi=0\n",
    "        else:\n",
    "            vi=sum(v)/len(v)\n",
    "        # scale the prior by (1+N(L))\n",
    "        prior=priors[0][k-1]/(1+len(v))\n",
    "        # calculate weighted average\n",
    "        scores[k]=weight*prior+(1-weight)*vi\n",
    "    # select child node based on the weighted average     \n",
    "    return max(scores,key=scores.get)  \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c33efb1c",
   "metadata": {},
   "source": [
    "```python\n",
    "# roll out a game till terminal state or depth reached\n",
    "def simulate(env_copy,done,reward,depth):\n",
    "    # if the game has already ended\n",
    "    if done==True:\n",
    "        return reward\n",
    "    for _ in range(depth):\n",
    "        move=env_copy.sample()\n",
    "        state,reward,done,info=env_copy.step(move)\n",
    "        # if terminal state is reached, returns outcome\n",
    "        if done==True:\n",
    "            return reward\n",
    "    # depth reached but game not over, evaluate\n",
    "    onehot_state=onehot_encoder(state)\n",
    "    # use the trained actor critic model to evaluate\n",
    "    action_probs, critic_value = model(onehot_state)\n",
    "    # output is predicted game outcome  \n",
    "    return critic_value[0,0]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88f81d44",
   "metadata": {},
   "source": [
    "## 4.2 Create An AlphaZero Agent in the Coin Game\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb5f027c",
   "metadata": {},
   "source": [
    "```python\n",
    "def alphazero_coin(env,weight,depth,num_rollouts=100):\n",
    "    # if there is only one valid move left, take it\n",
    "    if len(env.validinputs)==1:\n",
    "        return env.validinputs[0]\n",
    "    # get the prior from the AC policy network\n",
    "    onehot_state = onehot_encoder(env.state)\n",
    "    priors, _ = model(onehot_state)    \n",
    "    # create a dictionary results\n",
    "    results={}\n",
    "    for move in env.validinputs:\n",
    "        results[move]=[]\n",
    "    # roll out games\n",
    "    for _ in range(num_rollouts):\n",
    "        # select\n",
    "        move=select(priors,env,results,weight)\n",
    "        # expand\n",
    "        env_copy, done, reward=expand(env,move)\n",
    "        # simulate\n",
    "        reward=simulate(env_copy,done,reward,depth)\n",
    "        # backpropagate\n",
    "        results=backpropagate(env,move,reward,results)\n",
    "    # select the most visited child node\n",
    "    visits={k:len(v) for k,v in results.items()}\n",
    "    return max(visits,key=visits.get)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "202a5d06",
   "metadata": {},
   "source": [
    "## 4.3. AlphaZero vs AlphaGo in the Coin Game\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f806366f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.\n",
      "WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.\n",
      "AlphaZero wins!\n",
      "AlphaZero wins!\n",
      "AlphaZero wins!\n",
      "AlphaZero wins!\n",
      "AlphaZero wins!\n",
      "AlphaZero wins!\n",
      "AlphaZero wins!\n",
      "AlphaZero wins!\n",
      "AlphaZero wins!\n",
      "AlphaZero wins!\n"
     ]
    }
   ],
   "source": [
    "from utils.coin_simple_env import coin_game\n",
    "from utils.ch16util import alphago_coin\n",
    "from utils.ch19util import alphazero_coin\n",
    "# initiate game environment\n",
    "env=coin_game()\n",
    "# test ten games\n",
    "for i in range(10):\n",
    "    state=env.reset()  \n",
    "    while True: \n",
    "        # The AlphaGo agent moves first\n",
    "        action=alphago_coin(env,0.95,25,num_rollouts=1000)  \n",
    "        state,reward,done,_=env.step(action)\n",
    "        if done:\n",
    "            # print out the winner\n",
    "            print(\"AlphaGo wins!\")\n",
    "            break\n",
    "        # The AlphaZero agent moves second\n",
    "        action=alphazero_coin(env,0.95,25,num_rollouts=1000) \n",
    "        state,reward,done,_=env.step(action)     \n",
    "        if done:\n",
    "            # print out the winner\n",
    "            print(\"AlphaZero wins!\")\n",
    "            break        "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}