ch13DeepReinforcementLearningCoin.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "653089c9",
   "metadata": {},
   "source": [
    "# Chaper 13: Deep Reinforcement Learning\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "***\n",
    "*“Play is our brain’s favorite way of learning.”*\n",
    "\n",
    "-- Diane Ackerman\n",
    "***\n",
    "\n",
    "\n",
    "\n",
    "AlphaGo employs self-play and deep reinforcement learning to enhance its policy network, making it even more robust. During self-play, AlphaGo accumulates game experience data, which is then used to train a powerful value network. This network predicts the outcome of the game, a critical component in AlphaGo's strategy when competing against the renowned Go player, Lee Sedol. This chapter will guide you through understanding and implementing deep reinforcement learning, particularly the policy gradient method, in a coin game context. You'll explore the concept of a policy and how deep neural networks can be utilized to train a policy for intelligent gameplay.\n",
    "\n",
    "Previously, in Chapter 12, the focus was on value-based reinforcement learning for mastering the coin game. This involved learning the Q-value for each action in a state, denoted as $Q(a|s)$. After training the reinforcement model, decisions in the game were made based on choosing actions with the highest $Q(a|s)$ value in a given state $s$.\n",
    "\n",
    "Now, this chapter introduces policy-based reinforcement learning, where decisions are driven by a policy $π(a∣s)$, directing the agent on which actions to take in each state $s$. Policies can be deterministic, prescribing actions with absolute certainty, or stochastic, offering a probability distribution for possible actions.\n",
    "\n",
    "In the policy gradients method, the agent engages in numerous game sessions to learn the optimal policy. The agent bases its actions on the model's predictions, observes the resulting rewards, and adjusts the model parameters to align predicted action probabilities with desired probabilities. If a predicted action probability is lower than desired, the model weights are tweaked to increase the prediction, and vice versa.\n",
    "\n",
    "Additionally, this chapter delves into self-play, a technique to strengthen the deep reinforcement learning agent by pitting it against an incrementally stronger version of itself. The self-play generates a wealth of game experiences, which are then used to train a value network that predicts game outcomes based on current game states, with a value of $−1$ indicating a win for the second player and 1 for the first player. You'll also learn how to leverage the trained value network for strategic gameplay."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "282ad876",
   "metadata": {},
   "source": [
    "# 1. The Policy Gradients Method"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee63f10c",
   "metadata": {},
   "source": [
    "## 1.1. What Is A Policy?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed19b2e4",
   "metadata": {},
   "source": [
    "## 1.2. What is the Policy Gradient Method?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eee28a4e",
   "metadata": {},
   "source": [
    "# 2. Use Policy Gradients to Play the Coin Game"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e312e9d5",
   "metadata": {},
   "source": [
    "### 2.1. Use a network to define the policy\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b019c91c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "num_inputs = 22\n",
    "num_actions = 2\n",
    "# The input layer\n",
    "inputs = layers.Input(shape=(22,))\n",
    "# The common layer\n",
    "common = layers.Dense(32, activation=\"relu\")(inputs)\n",
    "common = layers.Dense(32, activation=\"relu\")(common)\n",
    "# The policy layer (the output layer)\n",
    "action = layers.Dense(num_actions, activation=\"softmax\")(common)\n",
    "# Put together the policy network\n",
    "model = keras.Model(inputs=inputs, outputs=action)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "68df75e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam(learning_rate=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09d6d496",
   "metadata": {},
   "source": [
    "## 2.2. Calculate Gradients and Discounted Rewards\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5b08faaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def onehot_encoder(state):\n",
    "    onehot=np.zeros((1,22))\n",
    "    onehot[0,state]=1\n",
    "    return onehot\n",
    "# the trained strong policy model\n",
    "trained=keras.models.load_model('files/strong_coin.h5')\n",
    "def opponent(state):\n",
    "    onehot_state=onehot_encoder(state)\n",
    "    policy=trained(onehot_state)\n",
    "    return np.random.choice([1,2],p=np.squeeze(policy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "601d5832",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.coin_simple_env import coin_game\n",
    "import tensorflow as tf\n",
    "\n",
    "env=coin_game()\n",
    "def playing():\n",
    "    # create lists to record game history\n",
    "    action_probs_history = []\n",
    "    rewards_history = []\n",
    "    state = env.reset()\n",
    "    episode_reward = 0\n",
    "    # the strong policy network agent moves first\n",
    "    state, reward, done, _ = env.step(opponent(state))\n",
    "    # record all game states \n",
    "    states = []    \n",
    "    while True:\n",
    "        # convert state to onehot to feed to model\n",
    "        onehot_state = onehot_encoder(state)\n",
    "        # estimate action probabilities \n",
    "        action_probs = model(onehot_state)\n",
    "        # select action based on the policy distribution\n",
    "        action=np.random.choice(num_actions, \n",
    "                            p=np.squeeze(action_probs))\n",
    "        # record log probabilities\n",
    "        action_probs_history.append(\n",
    "            tf.math.log(action_probs[0, action]))\n",
    "        # Apply the sampled action in our environment\n",
    "        # Remember to add 1 to action (change 0 or 1 to 1 or 2)\n",
    "        state, reward, done, _ = env.step(action+1)\n",
    "        states.append(state)\n",
    "        if done:\n",
    "            # PG player is player 2, -1 means PG player wins\n",
    "            reward = -reward\n",
    "            rewards_history.append(reward)\n",
    "            episode_reward += reward \n",
    "            break\n",
    "        else:\n",
    "            state, reward, done, _ = env.step(opponent(state))\n",
    "            reward = -reward\n",
    "            rewards_history.append(reward)\n",
    "            episode_reward += reward                 \n",
    "            if done:\n",
    "                break\n",
    "    return action_probs_history,\\\n",
    "            rewards_history, episode_reward, states, reward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "15b9c0d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def discount_rs(r):\n",
    "    discounted_rs = np.zeros(len(r))\n",
    "    running_add = 0\n",
    "    for i in reversed(range(0, len(r))):\n",
    "        running_add = gamma*running_add + r[i]\n",
    "        discounted_rs[i] = running_add  \n",
    "    return discounted_rs.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a156ac04",
   "metadata": {},
   "source": [
    "## 2.3. Update Parameters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "052e7981",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size=10\n",
    "allstates=[]\n",
    "alloutcome=[]\n",
    "def create_batch(batch_size):\n",
    "    action_probs_history = []\n",
    "    rewards_history = []\n",
    "    episode_rewards = []\n",
    "    for i in range(batch_size):\n",
    "        aps,rs,er,ss,outcome = playing()\n",
    "        returns = discount_rs(rs)\n",
    "        action_probs_history += aps\n",
    "        rewards_history += returns\n",
    "        episode_rewards.append(er)\n",
    "        # record game history for the next section\n",
    "        allstates.append(ss)\n",
    "        alloutcome.append(outcome)\n",
    "    return action_probs_history,\\\n",
    "        rewards_history,episode_rewards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "919c8b1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running reward: -0.980000 at episode 100\n",
      "running reward: -0.940000 at episode 200\n",
      "running reward: -0.960000 at episode 300\n",
      "running reward: -1.000000 at episode 400\n",
      "running reward: -0.980000 at episode 500\n",
      "running reward: -0.920000 at episode 600\n",
      "running reward: -0.840000 at episode 700\n",
      "running reward: 0.100000 at episode 800\n",
      "running reward: 0.920000 at episode 900\n",
      "Solved at episode 960!\n",
      "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n"
     ]
    }
   ],
   "source": [
    "from collections import deque\n",
    "\n",
    "running_rewards=deque(maxlen=100)\n",
    "gamma = 0.95  \n",
    "episode_count = 0\n",
    "# Train the model\n",
    "while True:\n",
    "    with tf.GradientTape() as tape:\n",
    "        action_probs_history,\\\n",
    "        rewards_history,episode_rewards=create_batch(batch_size)\n",
    "        # Calculating loss values to update our network\n",
    "        history = zip(action_probs_history, rewards_history)\n",
    "        actor_losses = []\n",
    "        for log_prob, ret in history:\n",
    "            # Calculate actor loss\n",
    "            actor_losses.append(-log_prob * ret)\n",
    "        # Adjust model parameters\n",
    "        loss_value = sum(actor_losses) \n",
    "    grads = tape.gradient(loss_value, model.trainable_variables)\n",
    "    optimizer.apply_gradients(zip(grads,\n",
    "                                  model.trainable_variables))\n",
    "\n",
    "    # Log details\n",
    "    episode_count += batch_size\n",
    "    for r in episode_rewards:\n",
    "        running_rewards.append(r)\n",
    "    running_reward=np.mean(np.array(running_rewards)) \n",
    "    # print out progress\n",
    "    if episode_count % 100 == 0:\n",
    "        template = \"running reward: {:.6f} at episode {}\"\n",
    "        print(template.format(running_reward, episode_count))   \n",
    "    # Stop if the game is solved\n",
    "    if running_reward > 0.999 and episode_count>100:  \n",
    "        print(\"Solved at episode {}!\".format(episode_count))\n",
    "        break\n",
    "model.save(\"files/PG_coin.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bcbdf00",
   "metadata": {},
   "source": [
    "# 3.  Train A Value Network\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8e0b2e3",
   "metadata": {},
   "source": [
    "## 3.1. Plans to Train a Value Network\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd8c5b19",
   "metadata": {},
   "source": [
    "## 3.2. Process the Game Experience Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fc2f8827",
   "metadata": {},
   "outputs": [],
   "source": [
    "Xs=[]\n",
    "ys=[]\n",
    "for states, result in zip(allstates,alloutcome):\n",
    "    for state in states:\n",
    "        onehot_state=onehot_encoder(state)\n",
    "        Xs.append(onehot_state)\n",
    "        if result==1:\n",
    "            # player 2 wins\n",
    "            ys.append(np.array([1,0]))\n",
    "        if result==-1:\n",
    "            # player 2 loses\n",
    "            ys.append(np.array([0,1]))       \n",
    "\n",
    "Xs=np.array(Xs).reshape(-1,22)\n",
    "ys=np.array(ys).reshape(-1,2)      "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39e27a43",
   "metadata": {},
   "source": [
    "## 3.3. Train A Value Network\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1579eddb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "value_network = Sequential()\n",
    "value_network.add(Dense(units=64,activation=\"relu\",\n",
    "                 input_shape=(22,)))\n",
    "value_network.add(Dense(32, activation=\"relu\"))\n",
    "value_network.add(Dense(16, activation=\"relu\"))\n",
    "value_network.add(Dense(2, activation='softmax'))\n",
    "value_network.compile(loss='categorical_crossentropy',\n",
    "   optimizer=keras.optimizers.Adam(learning_rate=0.0001),\n",
    "   metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6faf686c",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "195/195 [==============================] - 1s 631us/step - loss: 0.6507 - accuracy: 0.7284\n",
      "Epoch 2/100\n",
      "195/195 [==============================] - 0s 598us/step - loss: 0.5505 - accuracy: 0.7414\n",
      "Epoch 3/100\n",
      "195/195 [==============================] - 0s 635us/step - loss: 0.4156 - accuracy: 0.7454\n",
      "Epoch 4/100\n",
      "195/195 [==============================] - 0s 642us/step - loss: 0.2963 - accuracy: 0.8577\n",
      "Epoch 5/100\n",
      "195/195 [==============================] - 0s 561us/step - loss: 0.2328 - accuracy: 0.8996\n",
      "Epoch 6/100\n",
      "195/195 [==============================] - 0s 681us/step - loss: 0.2066 - accuracy: 0.9023\n",
      "Epoch 7/100\n",
      "195/195 [==============================] - 0s 578us/step - loss: 0.1955 - accuracy: 0.9009\n",
      "Epoch 8/100\n",
      "195/195 [==============================] - 0s 635us/step - loss: 0.1909 - accuracy: 0.9023\n",
      "Epoch 9/100\n",
      "195/195 [==============================] - 0s 617us/step - loss: 0.1875 - accuracy: 0.9022\n",
      "Epoch 10/100\n",
      "195/195 [==============================] - 0s 604us/step - loss: 0.1856 - accuracy: 0.9023\n",
      "Epoch 11/100\n",
      "195/195 [==============================] - 0s 572us/step - loss: 0.1845 - accuracy: 0.9023\n",
      "Epoch 12/100\n",
      "195/195 [==============================] - 0s 643us/step - loss: 0.1834 - accuracy: 0.9022\n",
      "Epoch 13/100\n",
      "195/195 [==============================] - 0s 565us/step - loss: 0.1832 - accuracy: 0.9014\n",
      "Epoch 14/100\n",
      "195/195 [==============================] - 0s 651us/step - loss: 0.1823 - accuracy: 0.9023\n",
      "Epoch 15/100\n",
      "195/195 [==============================] - 0s 611us/step - loss: 0.1819 - accuracy: 0.9023\n",
      "Epoch 16/100\n",
      "195/195 [==============================] - 0s 635us/step - loss: 0.1817 - accuracy: 0.9023\n",
      "Epoch 17/100\n",
      "195/195 [==============================] - 0s 597us/step - loss: 0.1812 - accuracy: 0.9023\n",
      "Epoch 18/100\n",
      "195/195 [==============================] - 0s 619us/step - loss: 0.1813 - accuracy: 0.9023\n",
      "Epoch 19/100\n",
      "195/195 [==============================] - 0s 590us/step - loss: 0.1808 - accuracy: 0.9023\n",
      "Epoch 20/100\n",
      "195/195 [==============================] - 0s 621us/step - loss: 0.1810 - accuracy: 0.9023\n",
      "Epoch 21/100\n",
      "195/195 [==============================] - 0s 633us/step - loss: 0.1804 - accuracy: 0.9023\n",
      "Epoch 22/100\n",
      "195/195 [==============================] - 0s 581us/step - loss: 0.1810 - accuracy: 0.9023\n",
      "Epoch 23/100\n",
      "195/195 [==============================] - 0s 633us/step - loss: 0.1806 - accuracy: 0.9012\n",
      "Epoch 24/100\n",
      "195/195 [==============================] - 0s 645us/step - loss: 0.1806 - accuracy: 0.9023\n",
      "Epoch 25/100\n",
      "195/195 [==============================] - 0s 645us/step - loss: 0.1804 - accuracy: 0.9023\n",
      "Epoch 26/100\n",
      "195/195 [==============================] - 0s 586us/step - loss: 0.1805 - accuracy: 0.9023\n",
      "Epoch 27/100\n",
      "195/195 [==============================] - 0s 629us/step - loss: 0.1805 - accuracy: 0.9023\n",
      "Epoch 28/100\n",
      "195/195 [==============================] - 0s 622us/step - loss: 0.1804 - accuracy: 0.9023\n",
      "Epoch 29/100\n",
      "195/195 [==============================] - 0s 609us/step - loss: 0.1805 - accuracy: 0.9023\n",
      "Epoch 30/100\n",
      "195/195 [==============================] - 0s 597us/step - loss: 0.1803 - accuracy: 0.9023\n",
      "Epoch 31/100\n",
      "195/195 [==============================] - 0s 623us/step - loss: 0.1804 - accuracy: 0.9023\n",
      "Epoch 32/100\n",
      "195/195 [==============================] - 0s 550us/step - loss: 0.1797 - accuracy: 0.9025\n",
      "Epoch 33/100\n",
      "195/195 [==============================] - 0s 597us/step - loss: 0.1805 - accuracy: 0.8994\n",
      "Epoch 34/100\n",
      "195/195 [==============================] - 0s 609us/step - loss: 0.1804 - accuracy: 0.9023\n",
      "Epoch 35/100\n",
      "195/195 [==============================] - 0s 546us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 36/100\n",
      "195/195 [==============================] - 0s 664us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 37/100\n",
      "195/195 [==============================] - 0s 544us/step - loss: 0.1798 - accuracy: 0.9023\n",
      "Epoch 38/100\n",
      "195/195 [==============================] - 0s 594us/step - loss: 0.1802 - accuracy: 0.9023\n",
      "Epoch 39/100\n",
      "195/195 [==============================] - 0s 617us/step - loss: 0.1802 - accuracy: 0.9023\n",
      "Epoch 40/100\n",
      "195/195 [==============================] - 0s 556us/step - loss: 0.1804 - accuracy: 0.9014\n",
      "Epoch 41/100\n",
      "195/195 [==============================] - 0s 653us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 42/100\n",
      "195/195 [==============================] - 0s 619us/step - loss: 0.1798 - accuracy: 0.8996\n",
      "Epoch 43/100\n",
      "195/195 [==============================] - 0s 591us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 44/100\n",
      "195/195 [==============================] - 0s 566us/step - loss: 0.1800 - accuracy: 0.9014\n",
      "Epoch 45/100\n",
      "195/195 [==============================] - 0s 614us/step - loss: 0.1802 - accuracy: 0.9023\n",
      "Epoch 46/100\n",
      "195/195 [==============================] - 0s 655us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 47/100\n",
      "195/195 [==============================] - 0s 626us/step - loss: 0.1803 - accuracy: 0.9023\n",
      "Epoch 48/100\n",
      "195/195 [==============================] - 0s 599us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 49/100\n",
      "195/195 [==============================] - 0s 551us/step - loss: 0.1801 - accuracy: 0.9012\n",
      "Epoch 50/100\n",
      "195/195 [==============================] - 0s 659us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 51/100\n",
      "195/195 [==============================] - 0s 610us/step - loss: 0.1803 - accuracy: 0.9023\n",
      "Epoch 52/100\n",
      "195/195 [==============================] - 0s 604us/step - loss: 0.1798 - accuracy: 0.9023\n",
      "Epoch 53/100\n",
      "195/195 [==============================] - 0s 603us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 54/100\n",
      "195/195 [==============================] - 0s 634us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 55/100\n",
      "195/195 [==============================] - 0s 592us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 56/100\n",
      "195/195 [==============================] - 0s 544us/step - loss: 0.1802 - accuracy: 0.9018\n",
      "Epoch 57/100\n",
      "195/195 [==============================] - 0s 608us/step - loss: 0.1804 - accuracy: 0.9022\n",
      "Epoch 58/100\n",
      "195/195 [==============================] - 0s 624us/step - loss: 0.1798 - accuracy: 0.9023\n",
      "Epoch 59/100\n",
      "195/195 [==============================] - 0s 656us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 60/100\n",
      "195/195 [==============================] - 0s 630us/step - loss: 0.1799 - accuracy: 0.9015\n",
      "Epoch 61/100\n",
      "195/195 [==============================] - 0s 592us/step - loss: 0.1801 - accuracy: 0.9018\n",
      "Epoch 62/100\n",
      "195/195 [==============================] - 0s 624us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 63/100\n",
      "195/195 [==============================] - 0s 561us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 64/100\n",
      "195/195 [==============================] - 0s 684us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 65/100\n",
      "195/195 [==============================] - 0s 609us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 66/100\n",
      "195/195 [==============================] - 0s 603us/step - loss: 0.1802 - accuracy: 0.9018\n",
      "Epoch 67/100\n",
      "195/195 [==============================] - 0s 608us/step - loss: 0.1802 - accuracy: 0.9023\n",
      "Epoch 68/100\n",
      "195/195 [==============================] - 0s 604us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 69/100\n",
      "195/195 [==============================] - 0s 616us/step - loss: 0.1799 - accuracy: 0.9007\n",
      "Epoch 70/100\n",
      "195/195 [==============================] - 0s 571us/step - loss: 0.1804 - accuracy: 0.9023\n",
      "Epoch 71/100\n",
      "195/195 [==============================] - 0s 635us/step - loss: 0.1800 - accuracy: 0.9010\n",
      "Epoch 72/100\n",
      "195/195 [==============================] - 0s 623us/step - loss: 0.1798 - accuracy: 0.9017\n",
      "Epoch 73/100\n",
      "195/195 [==============================] - 0s 597us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 74/100\n",
      "195/195 [==============================] - 0s 610us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 75/100\n",
      "195/195 [==============================] - 0s 558us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 76/100\n",
      "195/195 [==============================] - 0s 666us/step - loss: 0.1799 - accuracy: 0.9023\n",
      "Epoch 77/100\n",
      "195/195 [==============================] - 0s 613us/step - loss: 0.1802 - accuracy: 0.9023\n",
      "Epoch 78/100\n",
      "195/195 [==============================] - 0s 606us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 79/100\n",
      "195/195 [==============================] - 0s 622us/step - loss: 0.1799 - accuracy: 0.9023\n",
      "Epoch 80/100\n",
      "195/195 [==============================] - 0s 588us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 81/100\n",
      "195/195 [==============================] - 0s 601us/step - loss: 0.1799 - accuracy: 0.9023\n",
      "Epoch 82/100\n",
      "195/195 [==============================] - 0s 666us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 83/100\n",
      "195/195 [==============================] - 0s 588us/step - loss: 0.1802 - accuracy: 0.9023\n",
      "Epoch 84/100\n",
      "195/195 [==============================] - 0s 637us/step - loss: 0.1798 - accuracy: 0.9023\n",
      "Epoch 85/100\n",
      "195/195 [==============================] - 0s 622us/step - loss: 0.1802 - accuracy: 0.9023\n",
      "Epoch 86/100\n",
      "195/195 [==============================] - 0s 566us/step - loss: 0.1799 - accuracy: 0.9023\n",
      "Epoch 87/100\n",
      "195/195 [==============================] - 0s 659us/step - loss: 0.1799 - accuracy: 0.9023\n",
      "Epoch 88/100\n",
      "195/195 [==============================] - 0s 624us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 89/100\n",
      "195/195 [==============================] - 0s 673us/step - loss: 0.1798 - accuracy: 0.9023\n",
      "Epoch 90/100\n",
      "195/195 [==============================] - 0s 615us/step - loss: 0.1804 - accuracy: 0.9023\n",
      "Epoch 91/100\n",
      "195/195 [==============================] - 0s 613us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 92/100\n",
      "195/195 [==============================] - 0s 588us/step - loss: 0.1797 - accuracy: 0.9006\n",
      "Epoch 93/100\n",
      "195/195 [==============================] - 0s 653us/step - loss: 0.1797 - accuracy: 0.9023\n",
      "Epoch 94/100\n",
      "195/195 [==============================] - 0s 653us/step - loss: 0.1801 - accuracy: 0.9020\n",
      "Epoch 95/100\n",
      "195/195 [==============================] - 0s 620us/step - loss: 0.1801 - accuracy: 0.9023\n",
      "Epoch 96/100\n",
      "195/195 [==============================] - 0s 617us/step - loss: 0.1802 - accuracy: 0.9023\n",
      "Epoch 97/100\n",
      "195/195 [==============================] - 0s 564us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 98/100\n",
      "195/195 [==============================] - 0s 649us/step - loss: 0.1800 - accuracy: 0.9023\n",
      "Epoch 99/100\n",
      "195/195 [==============================] - 0s 619us/step - loss: 0.1799 - accuracy: 0.9023\n",
      "Epoch 100/100\n",
      "195/195 [==============================] - 0s 590us/step - loss: 0.1802 - accuracy: 0.9023\n"
     ]
    }
   ],
   "source": [
    "# Train the model for 100 epochs\n",
    "value_network.fit(Xs, ys, epochs=100, verbose=1)\n",
    "value_network.save('files/value_coin.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14805aac",
   "metadata": {},
   "source": [
    "# 4. Play the Coin Game with the Value Network\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "765a00ae",
   "metadata": {},
   "source": [
    "## 4.1. Best Moves Based on the Value Network\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2efd4a20",
   "metadata": {},
   "outputs": [],
   "source": [
    "# design game strategy\n",
    "from copy import deepcopy\n",
    "\n",
    "def best_move(env):\n",
    "    # Set the initial value of bestoutcome        \n",
    "    bestoutcome=-2;\n",
    "    bestmove=None    \n",
    "    #go through all possible moves hypothetically \n",
    "    for move in env.validinputs:\n",
    "        env_copy=deepcopy(env)\n",
    "        state,reward,done,info=env_copy.step(move)\n",
    "        onehot_state=onehot_encoder(state)\n",
    "        ps=value_network.predict(onehot_state,verbose=0)\n",
    "        # output is prob(2 wins) \n",
    "        win_prob=ps[0][0]\n",
    "        if win_prob>bestoutcome:\n",
    "            # Update the bestoutcome\n",
    "            bestoutcome = win_prob\n",
    "            # Update the best move\n",
    "            bestmove = move\n",
    "    return bestmove"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b73f926d",
   "metadata": {},
   "source": [
    "## 4.2. Against the Rule-Based AI\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3ab56e2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def rule_based_AI(env):\n",
    "    if env.state%3 != 0:\n",
    "        move = env.state%3\n",
    "    else:\n",
    "        move = random.choice([1,2])\n",
    "    return move "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f806366f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n"
     ]
    }
   ],
   "source": [
    "# test ten games\n",
    "for i in range(10):\n",
    "    state=env.reset()  \n",
    "    # The AI player moves firsts \n",
    "    action=rule_based_AI(env)\n",
    "    state,reward,done,_=env.step(action)\n",
    "    while True: \n",
    "        # move recommended by the value network\n",
    "        action=best_move(env)  \n",
    "        state,reward,done,_=env.step(action)\n",
    "        if done:\n",
    "            print(\"The value network wins!\")\n",
    "            break\n",
    "        # The AI player moves\n",
    "        action=rule_based_AI(env)\n",
    "        state,reward,done,_=env.step(action)     \n",
    "        if done:\n",
    "            print(\"The value network loses!\")\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13c2649a",
   "metadata": {},
   "source": [
    "## 4.3. Against Random Moves\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "13cfa062",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n",
      "The value network wins!\n"
     ]
    }
   ],
   "source": [
    "# does the strategy work if the agent moves first?\n",
    "# test ten games against random moves\n",
    "for i in range(10):\n",
    "    state=env.reset()  \n",
    "    while True: \n",
    "        # move recommended by the value network\n",
    "        action=best_move(env)  \n",
    "        state,reward,done,_=env.step(action)\n",
    "        if done:\n",
    "            print(\"The value network wins!\")\n",
    "            break\n",
    "        # The random player moves\n",
    "        action=random.choice(env.validinputs)\n",
    "        state,reward,done,_=env.step(action)     \n",
    "        if done:\n",
    "            print(\"The value network loses!\")\n",
    "            break"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}