ch09DeepLearningCoinGame.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "653089c9",
   "metadata": {},
   "source": [
    "# Chapter 9: Deep Learning in the Coin Game\n",
    "\n",
    "\n",
    "\n",
    "***\n",
    "*“My CPU is a neural net processor, a learning computer. The more contact I have with humans, the more I learn.”*\n",
    "\n",
    "-- The Terminator, in Terminator 2: Judgement Day\n",
    "***\n",
    "\n",
    "\n",
    "\n",
    "What you'll learn in this chapter:\n",
    "\n",
    "* The architecture of a neural network\n",
    "* How deep learning is related to machine learning and artificial intelligence\n",
    "* Steps and Components of the AlphaGo algorithm\n",
    "* Building and training a fast policy network and a strong policy network in the coin game\n",
    "* Implementing an MCTS game strategy with policy rollouts\n",
    "\n",
    "Starting from this chapter, you’ll learn a new AI\n",
    "paradigm: machine learning (ML). Instead of hard\n",
    "coding in the rules, ML algorithms take in input-output pairs and figure out the\n",
    "relation between the inputs (which we call features) and outputs (the labels). One\n",
    "field of ML, deep learning, has attracted much attention recently. The algorithm used\n",
    "by AlphaGo is based on deep reinforcement learning, which is a combination of deep\n",
    "learning and reinforcement learning (a type of ML we’ll cover later in this book). In\n",
    "this chapter, you’ll learn what deep learning is and how it’s related to AI and ML.\n",
    "\n",
    "Deep learning is a type of ML method that’s based on artificial neural networks. A\n",
    "neural network is a computational model inspired by the structure of neural networks\n",
    "in the human brain. It’s designed to recognize patterns in data, and it contains layers\n",
    "of interconnected nodes, or neurons. In this chapter, you’ll learn to use deep neural\n",
    "networks to design game strategies for the coin game. In particular, you’ll follow the\n",
    "steps in AlphaGo and create two policy networks. We’ll use these networks later in\n",
    "the book to create an AlphaGo agent to play the coin game.\n",
    "\n",
    "Specifically, the AlphaGo algorithm follows the following steps. We first gather a\n",
    "large number of games played by Go experts and use deep learning to train two\n",
    "policy networks to predict the moves of the Go experts: a fast policy network and a\n",
    "strong policy network. In the second step, we use self-play deep reinforcement learning\n",
    "to further train and improve the strong policy network. At the same time, we train a\n",
    "value network to predict game outcomes by using the game experience data from the\n",
    "self-plays. Finally, we design a game strategy based on an improved version of MCTS. Instead of using the upper confidence bounds for trees (UCT) formula to select the\n",
    "next move, AlphaGo uses a combination of the UCT formula, the improved strong\n",
    "policy network, and the value network. Further, instead of randomly selecting moves\n",
    "in game rollouts, AlphaGo uses the fast policy network to roll out games.\n",
    "\n",
    "In this chapter, you’ll implement the first step in the AlphaGo algorithm in the coin\n",
    "game. Specifically, you’ll use the rule-based AI we developed in Chapter 1 to generate\n",
    "expert moves.We then create two neural networks and use the generated expert moves\n",
    "to train the two networks to predict moves. You’ll then implement policy rollouts in\n",
    "MCTS, where games are played based on the probability distribution from the fast\n",
    "policy network, leading to a more intelligent MCTS agent compared to the traditional\n",
    "one."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48598568",
   "metadata": {},
   "source": [
    "# 1. Deep Learning, ML, and AI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "282ad876",
   "metadata": {},
   "source": [
    "# 2. What Are Neural Networks?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bcbdf00",
   "metadata": {},
   "source": [
    "# 3.  Two Policy Networks in the Coin Game\n",
    "# 4. Train Two Networks in the Coin game"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fc2f8827",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "\n",
    "def expert(env):\n",
    "    if env.state%3 != 0:\n",
    "        move = env.state%3\n",
    "    else:\n",
    "        move = random.choice([1,2])\n",
    "    return move    \n",
    "\n",
    "def non_expert(env):\n",
    "    if env.state%3 != 0 and np.random.rand()<0.5:\n",
    "        move = env.state%3\n",
    "    else:\n",
    "        move = random.choice([1,2])\n",
    "    return move  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6db4ea45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(20, 2), (17, 2), (13, 1), (11, 2), (8, 2), (5, 2), (1, 1)]\n"
     ]
    }
   ],
   "source": [
    "from utils.coin_simple_env import coin_game\n",
    "import time\n",
    "\n",
    "# Initiate the game environment\n",
    "env=coin_game()\n",
    "# Define the one_game() function\n",
    "def one_game(episode):\n",
    "    history=[]\n",
    "    state=env.reset()  \n",
    "    # The nonexpert moves firsts half the time\n",
    "    if episode%2==0:\n",
    "        action=non_expert(env)\n",
    "        state,reward,done,_=env.step(action)\n",
    "    while True:   \n",
    "        action=expert(env)  \n",
    "        history.append((state,action))\n",
    "        state,reward,done,_=env.step(action)\n",
    "        if done:\n",
    "            break\n",
    "        action=non_expert(env)\n",
    "        state,reward,done,_=env.step(action)     \n",
    "        if done:\n",
    "            break\n",
    "    return history\n",
    "\n",
    "# Simulate one game and print out results\n",
    "history=one_game(0)\n",
    "print(history)        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8a6a600a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# simulate the game 10000 times \n",
    "results = []        \n",
    "for episode in range(10000):\n",
    "    history=one_game(episode)\n",
    "    results+=history   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "88782a8d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(20, 2), (17, 2), (14, 2), (11, 2), (8, 2), (5, 2), (2, 2), (21, 1), (18, 1), (15, 1)]\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "# save the simulation data on your computer\n",
    "with open('files/games_coin.p', 'wb') as fp:\n",
    "    pickle.dump(results,fp)\n",
    "# read the data and print out the first 10 observations       \n",
    "with open('files/games_coin.p', 'rb') as fp:\n",
    "    games = pickle.load(fp)\n",
    "print(games[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bd8eed1",
   "metadata": {},
   "source": [
    "## 4.2. Create Two Neural Networks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1579eddb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "fast_model = Sequential()\n",
    "fast_model.add(Dense(units=32,activation=\"relu\",\n",
    "                 input_shape=(22,)))\n",
    "fast_model.add(Dense(2, activation='softmax'))\n",
    "fast_model.compile(loss='categorical_crossentropy',\n",
    "                   optimizer='adam', \n",
    "                   metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d4b6a220",
   "metadata": {},
   "outputs": [],
   "source": [
    "strong_model = Sequential()\n",
    "strong_model.add(Dense(units=64,activation=\"relu\",\n",
    "                 input_shape=(22,)))\n",
    "strong_model.add(Dense(32, activation=\"relu\"))\n",
    "strong_model.add(Dense(16, activation=\"relu\"))\n",
    "strong_model.add(Dense(2, activation='softmax'))\n",
    "strong_model.compile(loss='categorical_crossentropy',\n",
    "                   optimizer='adam', \n",
    "                   metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "441286c7",
   "metadata": {},
   "source": [
    "## 4.3. Train the Neural Networks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "96e2ab86",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]\n",
      " [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "states=[20,1]\n",
    "one_hot=to_categorical(states,22)\n",
    "print(one_hot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "16d6ea3b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1. 0.]\n",
      " [0. 1.]]\n"
     ]
    }
   ],
   "source": [
    "actions=[1,2]\n",
    "# change actions 1 and 2 to 0 and 1.\n",
    "actions=np.array(actions)-1\n",
    "# change actions to one-hot actions\n",
    "one_hot_actions=to_categorical(actions,2)\n",
    "print(one_hot_actions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "31cfc31b",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('files/games_coin.p','rb') as fp:\n",
    "    games=pickle.load(fp)\n",
    "\n",
    "states = []\n",
    "actions = []\n",
    "for x in games:\n",
    "    state=to_categorical(x[0],22)\n",
    "    action=to_categorical(x[1]-1,2)\n",
    "    states.append(state)\n",
    "    actions.append(action)\n",
    "\n",
    "X = np.array(states).reshape((-1, 22))\n",
    "y = np.array(actions).reshape((-1, 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a3fc541a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the models for 25 epochs\n",
    "fast_model.fit(X, y, epochs=25, verbose=1)\n",
    "fast_model.save('files/fast_coin.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bf390256",
   "metadata": {},
   "outputs": [],
   "source": [
    "strong_model.fit(X, y, epochs=25, verbose=1)\n",
    "strong_model.save('files/strong_coin.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14805aac",
   "metadata": {},
   "source": [
    "# 5. MCTS with Policy Rollouts in the Coin Game\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a934805a",
   "metadata": {},
   "source": [
    "## 5.1. Policy-Based MCTS in the Coin Game\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "207914a0",
   "metadata": {},
   "source": [
    "Go to book's GitHub repository to download the file *ch09util.py* and place it in the folder /Desktop/ags/utils/ on your computer. In the file, we define a *DL_stochastic()* function as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0275ee6b",
   "metadata": {},
   "source": [
    "```python\n",
    "def onehot_encoder(state):\n",
    "    onehot=np.zeros((1,22))\n",
    "    onehot[0,state]=1\n",
    "    return onehot\n",
    "\n",
    "def DL_stochastic(env): \n",
    "    state = env.state\n",
    "    onehot_state = onehot_encoder(state)\n",
    "    action_probs = model(onehot_state)\n",
    "    return np.random.choice([1,2], \n",
    "            p=np.squeeze(action_probs))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db74ec75",
   "metadata": {},
   "source": [
    "```python\n",
    "def policy_simulate(env_copy,done,reward,model):\n",
    "    # if the game has already ended\n",
    "    if done==True:\n",
    "        return reward\n",
    "    while True:\n",
    "        move=DL_stochastic(env_copy,model)\n",
    "        state,reward,done,info=env_copy.step(move)\n",
    "        if done==True:\n",
    "            return reward\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f4b749c",
   "metadata": {},
   "source": [
    "```python\n",
    "def policy_mcts_coin(env,model,num_rollouts=100,temperature=1.4):\n",
    "    # if there is only one valid move left, take it\n",
    "    if len(env.validinputs)==1:\n",
    "        return env.validinputs[0]\n",
    "    # create three dictionaries counts, wins, losses\n",
    "    counts={}\n",
    "    wins={}\n",
    "    losses={}\n",
    "    for move in env.validinputs:\n",
    "        counts[move]=0\n",
    "        wins[move]=0\n",
    "        losses[move]=0\n",
    "    # roll out games\n",
    "    for _ in range(num_rollouts):\n",
    "        # selection\n",
    "        move=select(env,counts,wins,losses,temperature)\n",
    "        # expansion\n",
    "        env_copy, done, reward=expand(env,move)\n",
    "        # simulation\n",
    "        reward=policy_simulate(env_copy,done,reward,model)\n",
    "        # backpropagate\n",
    "        counts,wins,losses=backpropagate(\\\n",
    "            env,move,reward,counts,wins,losses)\n",
    "    # make the move\n",
    "    return next_move(counts,wins,losses)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4535cf7",
   "metadata": {},
   "source": [
    "## 5.2. The Effectiveness of the Policy MCTS Agent\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f3fa6933",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.ch08util import mcts\n",
    "from utils.ch09util import policy_mcts_coin\n",
    "\n",
    "env=coin_game()\n",
    "results=[]\n",
    "for i in range(100):\n",
    "    state=env.reset() \n",
    "    # Half the time, the UCT MCTS agent moves first\n",
    "    if i%2==0:\n",
    "        action=mcts(env,num_rollouts=100)\n",
    "        state, reward, done, info=env.step(action)\n",
    "    while True:\n",
    "        action=policy_mcts_coin(env,model,num_rollouts=100) \n",
    "        state, reward, done, info=env.step(action)\n",
    "        if done:\n",
    "            # result is 1 if the policy MCTS agent wins\n",
    "            results.append(1)    \n",
    "            break  \n",
    "        action=mcts(env,num_rollouts=100)\n",
    "        state, reward, done, info=env.step(action)\n",
    "        if done:\n",
    "            # result is -1 if the policy MCTS agent loses\n",
    "            results.append(-1)   \n",
    "            break  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e6e7fd07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the policy MCTS agent has won 100 games\n",
      "the policy MCTS agent has lost 0 games\n"
     ]
    }
   ],
   "source": [
    "wins=results.count(1)\n",
    "print(f\"the policy MCTS agent has won {wins} games\")\n",
    "losses=results.count(-1)\n",
    "print(f\"the policy MCTS agent has lost {losses} games\")   "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}