ch11PolicyNetworkConnectFour.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "653089c9",
   "metadata": {},
   "source": [
    "# Chapter 11: A Policy Netwok in Connect Four\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "***\n",
    "***“We first trained the policy network on 30 million moves from games played by human experts, until it could predict the human move 57% of the time (the previous record before AlphaGo was 44%).”***\n",
    "\n",
    "-- David Silver and Demis Hassabis, Google DeepMind \n",
    "***\n",
    "\n",
    "\n",
    "\n",
    "In this chapter, you’ll continue with what you learned in Chapters 9 and 10 and apply deep learning to another game: Connect Four. \n",
    "\n",
    "Since the Coin game and Tic Tac Toe are solved games, generating expert moves is relatively straightforward. Connect Four is also a solved game, but implementing expert moves using the perfect rule-based algorithm (e.g., by using the method in Victor Allis' 1988 master thesis) is complicated. Further, machine learning is most useful in unsolved games such as Chess or Go that cannot rely on rule-based algorithms alone. Therefore, it's important to master machine learning skills to help unsolved games, as the DeepMind team did. As a result, we treat Connect Four as an unsolved game and use the AlphaGo algorithm to come up with strong agents. \n",
    "\n",
    "To generate expert moves in Connect Four, we first design a computer player who choose moves by using the MCTS algorithm 50% of the time. It chooses moves by using the MiniMax algorithm the other 50% of the time. By randomizing between two algorithms, we avoid identical games in simulations so we can cover different game scenarios. We let the above computer player play against itself for 10,000 games. In each game, the winner's moves are considered expert moves while the loser's moves are discarded. \n",
    "\n",
    "You'll use these expert moves to train a deep neural network. To save time, we train one network to serve as both the fast policy network and the strong policy network. The output layer has seven neurons, representing the seven possible moves the expert can take: column 1 through column 7. Essentially we are conducting a multi-category classification problem. The neural network we create includes both dense layers and a convolutional layer. You'll learn to treat the Connect Four game board as a two-dimensional image and extract spatial features from the board (several game pieces in a row horizontally, vertically, or diagonally, for example) and associate these features with expert moves. After the network is trained, you'll use it to design a game strategy to play Connect Four by using the mixed MCTS algorithm (similar to what we did in Chapter 11). Specifically, instead of selecting moves based on UCT scores alone, you'll select moves based on both UCT scores and the probability distribution from the policy network we trained. We call the agent who selects moves this way the mixed MCTS agent. Finally, you'll show that the mixed MCTS agent is more intelligent than the UCT MCTS agent that we developed in Chapter 8."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bcbdf00",
   "metadata": {},
   "source": [
    "# 1.  Deep Learning in Connect Four\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8e0b2e3",
   "metadata": {},
   "source": [
    "## 1.1. Steps to Train the Policy Netork in Connect Four\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd8c5b19",
   "metadata": {},
   "source": [
    "## 1.2. Generate Expert Moves in Connect Four\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fc2f8827",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.conn_simple_env import conn\n",
    "from utils.ch08util import mcts\n",
    "import numpy as np\n",
    "from utils.ch06util import MiniMax_conn \n",
    "\n",
    "# Initiate the game environment\n",
    "env=conn()\n",
    "# Define a player\n",
    "def player(env):\n",
    "    if len(env.occupied[3])==0:\n",
    "        action=4\n",
    "    elif len(env.validinputs)==1:\n",
    "        action=env.validinputs[0]\n",
    "    else:\n",
    "        # Use Minimax 50% of the time\n",
    "        if np.random.uniform(0,1,1)<=0.5:\n",
    "            action=MiniMax_conn(env,depth=3)\n",
    "        # Use MCTS 50% of the time            \n",
    "        else: \n",
    "            action=mcts(env,num_rollouts=1000)\n",
    "    return action "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6db4ea45",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[array([[0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0]]), 4, 'red'], [array([[0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [1, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0]]), 3, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 4, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  1,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 4, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 3, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 5, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 5, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 4, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1, -1,  0,  0],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 3, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  0,  0,  0],\n",
      "       [ 1,  1, -1, -1,  0,  0],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 4, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  0,  0,  0],\n",
      "       [ 1,  1, -1, -1, -1,  0],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 4, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  0,  0,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 5, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  0,  0,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 7, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  0,  0,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 5, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  0,  0,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1, -1, -1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 3, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  1,  0,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1, -1, -1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 3, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  1, -1,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1, -1, -1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 1, 'red'], [array([[ 1,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  1, -1,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1, -1, -1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 5, 'yellow'], [array([[ 1,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  1, -1,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1, -1, -1, -1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 5, 'red'], [array([[ 1,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  1, -1,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1, -1, -1, -1,  1],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 6, 'yellow'], [array([[ 1,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  1,  1, -1,  0],\n",
      "       [ 1,  1, -1, -1, -1,  1],\n",
      "       [-1,  1, -1, -1, -1,  1],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 6, 'red']]\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "# Define the one_game() function\n",
    "def one_game():\n",
    "    history = []\n",
    "    state=env.reset()   \n",
    "    while True:   \n",
    "        action=player(env) \n",
    "        history.append([np.array(state).reshape(7,6),\n",
    "                        action, env.turn])    \n",
    "        state, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            break             \n",
    "    return history, reward\n",
    "\n",
    "# Simulate one game and print out results\n",
    "history,reward=one_game()\n",
    "print(history)\n",
    "print(reward)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8a6a600a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the first game is (1, [[array([[0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0]]), 4, 'red'], [array([[0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [1, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0],\n",
      "       [0, 0, 0, 0, 0, 0]]), 3, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0]]), 7, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 6, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 4, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  1,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 4, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 6, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 6, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 4, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 3, 'yellow'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1, -1,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 2, 'red'], [array([[ 0,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 1, 'yellow'], [array([[-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0]]), 7, 'red'], [array([[-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1,  0,  0,  0,  0],\n",
      "       [ 1,  1, -1,  1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  1,  0,  0,  0,  0]]), 3, 'yellow'], [array([[-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  0,  0,  0],\n",
      "       [ 1,  1, -1,  1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  1,  0,  0,  0,  0]]), 3, 'red'], [array([[-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  1,  0,  0,  0,  0]]), 7, 'yellow'], [array([[-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  0,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0]]), 4, 'red'], [array([[-1,  0,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0]]), 1, 'yellow'], [array([[-1, -1,  0,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0]]), 1, 'red'], [array([[-1, -1,  1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  1, -1,  0,  0,  0]]), 7, 'yellow'], [array([[-1, -1,  1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  1, -1, -1,  0,  0]]), 7, 'red'], [array([[-1, -1,  1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1,  0,  0,  0],\n",
      "       [ 1,  1, -1, -1,  1,  0]]), 6, 'yellow'], [array([[-1, -1,  1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1, -1,  0,  0],\n",
      "       [ 1,  1, -1, -1,  1,  0]]), 6, 'red'], [array([[-1, -1,  1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1, -1,  1,  0],\n",
      "       [ 1,  1, -1, -1,  1,  0]]), 7, 'yellow'], [array([[-1, -1,  1,  0,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1, -1,  1,  0],\n",
      "       [ 1,  1, -1, -1,  1, -1]]), 1, 'red'], [array([[-1, -1,  1,  1,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [ 0,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1, -1,  1,  0],\n",
      "       [ 1,  1, -1, -1,  1, -1]]), 5, 'yellow'], [array([[-1, -1,  1,  1,  0,  0],\n",
      "       [ 1,  0,  0,  0,  0,  0],\n",
      "       [-1, -1, -1,  1,  0,  0],\n",
      "       [ 1,  1, -1,  1,  1,  0],\n",
      "       [-1,  0,  0,  0,  0,  0],\n",
      "       [-1,  1, -1, -1,  1,  0],\n",
      "       [ 1,  1, -1, -1,  1, -1]]), 5, 'red']])\n"
     ]
    }
   ],
   "source": [
    "# simulate 10000 games; take hours to run\n",
    "results = []        \n",
    "for episode in range(10000):\n",
    "    history,reward=one_game()\n",
    "    results.append((reward, history))\n",
    "\n",
    "import pickle\n",
    "# save the simulation data on your computer\n",
    "with open('files/games_conn.p', 'wb') as fp:\n",
    "    pickle.dump(results,fp)\n",
    "# read the data and print out the first 10 observations       \n",
    "with open('files/games_conn.p', 'rb') as fp:\n",
    "    games = pickle.load(fp) \n",
    "print(\"the first game is\", games[0]) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39e27a43",
   "metadata": {},
   "source": [
    "# 2. A Policy Network in Connect Four\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bd8eed1",
   "metadata": {},
   "source": [
    "## 2.1. Create A Neural Network for Connect Four\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1579eddb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Conv2D, Flatten\n",
    "\n",
    "fast_model = Sequential()\n",
    "fast_model.add(Conv2D(filters=128, \n",
    "    kernel_size=(4,4),padding=\"same\",activation=\"relu\",\n",
    "                 input_shape=(7,6,1)))\n",
    "fast_model.add(Flatten())\n",
    "fast_model.add(Dense(units=64, activation=\"relu\"))\n",
    "fast_model.add(Dense(units=64, activation=\"relu\"))\n",
    "fast_model.add(Dense(7, activation='softmax'))\n",
    "fast_model.compile(loss='categorical_crossentropy',\n",
    "                   optimizer='adam', \n",
    "                   metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "441286c7",
   "metadata": {},
   "source": [
    "## 2.2. Train the Neural Network in Connect Four\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "31cfc31b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(129776, 7, 6, 1) (129776, 7)\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "import numpy as np\n",
    "with open('files/games_conn.p','rb') as fp:\n",
    "    games=pickle.load(fp)\n",
    "\n",
    "X=[]\n",
    "y=[]\n",
    "for reward, history in games:\n",
    "    # games in which the red won the game\n",
    "    if reward>0:\n",
    "        for state, action, turn in history: \n",
    "            # we only use actions taken by red\n",
    "            if turn==\"red\":\n",
    "                X.append(state)\n",
    "                y.append(action-1)\n",
    "    # games in which the yellow won the game                \n",
    "    if reward<0:\n",
    "        for state, action, turn in history: \n",
    "            # we only use actions taken by red            \n",
    "            if turn==\"yellow\":\n",
    "                # multiply the board by -1\n",
    "                X.append(-state)\n",
    "                y.append(action-1)        \n",
    "                \n",
    "X=np.array(X).reshape((-1,7,6,1))  \n",
    "y=to_categorical(y,7)  \n",
    "print(X.shape, y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a3fc541a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "4056/4056 [==============================] - 17s 4ms/step - loss: 1.5642 - accuracy: 0.4046\n",
      "Epoch 2/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.4327 - accuracy: 0.4617\n",
      "Epoch 3/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.3910 - accuracy: 0.4821\n",
      "Epoch 4/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.3573 - accuracy: 0.4994\n",
      "Epoch 5/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.3287 - accuracy: 0.5132\n",
      "Epoch 6/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.3035 - accuracy: 0.5257\n",
      "Epoch 7/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.2820 - accuracy: 0.5346\n",
      "Epoch 8/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.2623 - accuracy: 0.5438\n",
      "Epoch 9/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.2428 - accuracy: 0.5538\n",
      "Epoch 10/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.2249 - accuracy: 0.5613\n",
      "Epoch 11/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.2074 - accuracy: 0.5683\n",
      "Epoch 12/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.1925 - accuracy: 0.5760\n",
      "Epoch 13/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.1775 - accuracy: 0.5804\n",
      "Epoch 14/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.1635 - accuracy: 0.5878\n",
      "Epoch 15/100\n",
      "4056/4056 [==============================] - 17s 4ms/step - loss: 1.1508 - accuracy: 0.5916\n",
      "Epoch 16/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.1373 - accuracy: 0.5984\n",
      "Epoch 17/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.1240 - accuracy: 0.6028\n",
      "Epoch 18/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.1127 - accuracy: 0.6084\n",
      "Epoch 19/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.1027 - accuracy: 0.6108\n",
      "Epoch 20/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.0902 - accuracy: 0.6163\n",
      "Epoch 21/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.0795 - accuracy: 0.6207\n",
      "Epoch 22/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.0702 - accuracy: 0.6259\n",
      "Epoch 23/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.0604 - accuracy: 0.6271\n",
      "Epoch 24/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.0517 - accuracy: 0.6315\n",
      "Epoch 25/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.0407 - accuracy: 0.6356\n",
      "Epoch 26/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.0335 - accuracy: 0.6379\n",
      "Epoch 27/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.0243 - accuracy: 0.6415\n",
      "Epoch 28/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.0159 - accuracy: 0.6450\n",
      "Epoch 29/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 1.0082 - accuracy: 0.6483\n",
      "Epoch 30/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 1.0003 - accuracy: 0.6511\n",
      "Epoch 31/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9916 - accuracy: 0.6540\n",
      "Epoch 32/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.9837 - accuracy: 0.6582\n",
      "Epoch 33/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9781 - accuracy: 0.6598\n",
      "Epoch 34/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9708 - accuracy: 0.6625\n",
      "Epoch 35/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9641 - accuracy: 0.6641\n",
      "Epoch 36/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9578 - accuracy: 0.6671\n",
      "Epoch 37/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9497 - accuracy: 0.6698\n",
      "Epoch 38/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.9445 - accuracy: 0.6722\n",
      "Epoch 39/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9383 - accuracy: 0.6741\n",
      "Epoch 40/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9323 - accuracy: 0.6771\n",
      "Epoch 41/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9255 - accuracy: 0.6793\n",
      "Epoch 42/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.9206 - accuracy: 0.6811\n",
      "Epoch 43/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.9149 - accuracy: 0.6829\n",
      "Epoch 44/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9085 - accuracy: 0.6853\n",
      "Epoch 45/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.9024 - accuracy: 0.6878\n",
      "Epoch 46/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8989 - accuracy: 0.6886\n",
      "Epoch 47/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8934 - accuracy: 0.6903\n",
      "Epoch 48/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8881 - accuracy: 0.6931\n",
      "Epoch 49/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8812 - accuracy: 0.6951\n",
      "Epoch 50/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8789 - accuracy: 0.6962\n",
      "Epoch 51/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8740 - accuracy: 0.6983\n",
      "Epoch 52/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.8704 - accuracy: 0.6984\n",
      "Epoch 53/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8653 - accuracy: 0.7006\n",
      "Epoch 54/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8597 - accuracy: 0.7025\n",
      "Epoch 55/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.8569 - accuracy: 0.7046\n",
      "Epoch 56/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.8515 - accuracy: 0.7057\n",
      "Epoch 57/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.8483 - accuracy: 0.7077\n",
      "Epoch 58/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.8419 - accuracy: 0.7089\n",
      "Epoch 59/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.8394 - accuracy: 0.7107\n",
      "Epoch 60/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.8363 - accuracy: 0.7112\n",
      "Epoch 61/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.8316 - accuracy: 0.7120\n",
      "Epoch 62/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8292 - accuracy: 0.7137\n",
      "Epoch 63/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8231 - accuracy: 0.7152\n",
      "Epoch 64/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8210 - accuracy: 0.7167\n",
      "Epoch 65/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8193 - accuracy: 0.7174\n",
      "Epoch 66/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8130 - accuracy: 0.7199\n",
      "Epoch 67/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8099 - accuracy: 0.7219\n",
      "Epoch 68/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.8077 - accuracy: 0.7221\n",
      "Epoch 69/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8034 - accuracy: 0.7236\n",
      "Epoch 70/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.8006 - accuracy: 0.7252\n",
      "Epoch 71/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7971 - accuracy: 0.7254\n",
      "Epoch 72/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7950 - accuracy: 0.7262\n",
      "Epoch 73/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7919 - accuracy: 0.7270\n",
      "Epoch 74/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7875 - accuracy: 0.7282\n",
      "Epoch 75/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7854 - accuracy: 0.7299\n",
      "Epoch 76/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7815 - accuracy: 0.7311\n",
      "Epoch 77/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7805 - accuracy: 0.7314\n",
      "Epoch 78/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7779 - accuracy: 0.7323\n",
      "Epoch 79/100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7755 - accuracy: 0.7335\n",
      "Epoch 80/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7730 - accuracy: 0.7346\n",
      "Epoch 81/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7700 - accuracy: 0.7353\n",
      "Epoch 82/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7668 - accuracy: 0.7366\n",
      "Epoch 83/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7648 - accuracy: 0.7366\n",
      "Epoch 84/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7610 - accuracy: 0.7381\n",
      "Epoch 85/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7580 - accuracy: 0.7401\n",
      "Epoch 86/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7587 - accuracy: 0.7389\n",
      "Epoch 87/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7554 - accuracy: 0.7399\n",
      "Epoch 88/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7528 - accuracy: 0.7412\n",
      "Epoch 89/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7505 - accuracy: 0.7418\n",
      "Epoch 90/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7468 - accuracy: 0.7430\n",
      "Epoch 91/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7444 - accuracy: 0.7447\n",
      "Epoch 92/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7426 - accuracy: 0.7454\n",
      "Epoch 93/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7441 - accuracy: 0.7444\n",
      "Epoch 94/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7377 - accuracy: 0.7468\n",
      "Epoch 95/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7354 - accuracy: 0.7461\n",
      "Epoch 96/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7334 - accuracy: 0.7478\n",
      "Epoch 97/100\n",
      "4056/4056 [==============================] - 16s 4ms/step - loss: 0.7321 - accuracy: 0.7490\n",
      "Epoch 98/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7317 - accuracy: 0.7484\n",
      "Epoch 99/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7281 - accuracy: 0.7499\n",
      "Epoch 100/100\n",
      "4056/4056 [==============================] - 15s 4ms/step - loss: 0.7238 - accuracy: 0.7515\n"
     ]
    }
   ],
   "source": [
    "# Train the policy network for 100 epochs\n",
    "fast_model.fit(X, y, epochs=100, verbose=1)\n",
    "fast_model.save('files/policy_conn.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14805aac",
   "metadata": {},
   "source": [
    "# 3. Mixed MCTS in Connect Four\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b136b48",
   "metadata": {},
   "source": [
    "In the local module ch11util, we define a mix_mcts_conn() function as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2368a805",
   "metadata": {},
   "source": [
    "```python\n",
    "def mix_mcts_conn(env,model,num_rollouts=100,temperature=1.4):\n",
    "    # if there is only one valid move left, take it\n",
    "    if len(env.validinputs)==1:\n",
    "        return env.validinputs[0]\n",
    "    # create three dictionaries counts, wins, losses\n",
    "    counts={}\n",
    "    wins={}\n",
    "    losses={}\n",
    "    for move in env.validinputs:\n",
    "        counts[move]=0\n",
    "        wins[move]=0\n",
    "        losses[move]=0\n",
    "    # priors from the policy network\n",
    "    state = env.state.reshape(-1,7,6,1)\n",
    "    if env.turn==\"red\":\n",
    "        action_probs= model(state)\n",
    "    else:\n",
    "        action_probs= model(-state)     \n",
    "    ps={}\n",
    "    for a in sorted(env.validinputs):\n",
    "        ps[a]=np.squeeze(action_probs)[a-1]    \n",
    "    # roll out games\n",
    "    for _ in range(num_rollouts):\n",
    "        # selection\n",
    "        move=mix_select(env,ps,counts,wins,losses,temperature)\n",
    "        # expansion\n",
    "        env_copy, done, reward=expand(env,move)\n",
    "        # simulation\n",
    "        reward=simulate(env_copy,done,reward)      \n",
    "        # backpropagate\n",
    "        counts,wins,losses=backpropagate(\\\n",
    "            env,move,reward,counts,wins,losses)\n",
    "    # make the move\n",
    "    return next_move(ps,counts,wins,losses)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb0e0986",
   "metadata": {},
   "source": [
    "# 4. The Effectiveness of the Mixed MCTS\n",
    "\n",
    "\n",
    "## 4.1 Manually Plaly against the Mixed MCTS Agent\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4c8e35e5",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Player red has chosen 4\n",
      "Current state is \n",
      "[[0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 0]\n",
      " [0 0 0 0 0 0 0]\n",
      " [0 0 0 1 0 0 0]]\n",
      "What's your move?4\n",
      "Player yellow has chosen 4\n",
      "Current state is \n",
      "[[ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0  0  1  0  0  0]]\n",
      "Player red has chosen 5\n",
      "Current state is \n",
      "[[ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0  0  1  1  0  0]]\n",
      "What's your move?4\n",
      "Player yellow has chosen 4\n",
      "Current state is \n",
      "[[ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0  0  1  1  0  0]]\n",
      "Player red has chosen 6\n",
      "Current state is \n",
      "[[ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0  0  1  1  1  0]]\n",
      "What's your move?3\n",
      "Player yellow has chosen 3\n",
      "Current state is \n",
      "[[ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0 -1  1  1  1  0]]\n",
      "Player red has chosen 7\n",
      "Current state is \n",
      "[[ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0  0 -1  0  0  0]\n",
      " [ 0  0 -1  1  1  1  1]]\n",
      "Player red has won!\n"
     ]
    }
   ],
   "source": [
    "from utils.conn_simple_env import conn\n",
    "from utils.ch11util import mix_mcts_conn\n",
    "from tensorflow.keras.models import load_model\n",
    "model=load_model(\"files/policy_conn.h5\")\n",
    "\n",
    "env=conn()\n",
    "state=env.reset() \n",
    "while True:\n",
    "    action=mix_mcts_conn(env,model,num_rollouts=10000)\n",
    "    print(f\"Player {env.turn} has chosen {action}\")    \n",
    "    state, reward, done, info = env.step(action)\n",
    "    print(f\"Current state is \\n{state.T[::-1]}\")\n",
    "    if done:\n",
    "        if reward==1:\n",
    "            print(f\"Player {env.turn} has won!\") \n",
    "        else:\n",
    "            print(\"Game over, it's a tie!\")    \n",
    "        break  \n",
    "    action=int(input(\"What's your move?\"))\n",
    "    print(f\"Player {env.turn} has chosen {action}\")    \n",
    "    state, reward, done, info = env.step(action)\n",
    "    print(f\"Current state is \\n{state.T[::-1]}\")\n",
    "    if done:\n",
    "        print(f\"Player {env.turn} has won!\")  \n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b21e83ce",
   "metadata": {},
   "source": [
    "The agent created a double attack and won."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3febe543",
   "metadata": {},
   "source": [
    "## 4.2 Mixed MCTS vs UCT MCTS in Connect Four\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3fdd3664",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_rollouts=100\n",
    "results=[]\n",
    "for i in range(100):\n",
    "    print(i)\n",
    "    state=env.reset() \n",
    "    if i%2==0:\n",
    "        action=mcts(env,num_rollouts=num_rollouts)\n",
    "        state, reward, done, info = env.step(action)\n",
    "    while True:\n",
    "        action=mix_mcts_conn(env,model,num_rollouts=num_rollouts)  \n",
    "        state, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            # result is 1 if mixed MCTS wins\n",
    "            if reward!=0:\n",
    "                results.append(1) \n",
    "            else:\n",
    "                results.append(0)    \n",
    "            break  \n",
    "        action=mcts(env,num_rollouts=num_rollouts)   \n",
    "        state, reward, done, info = env.step(action)\n",
    "        if done:\n",
    "            # result is -1 if mixed MCTS loses\n",
    "            if reward!=0:\n",
    "                results.append(-1) \n",
    "            else:\n",
    "                results.append(0)    \n",
    "            break "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8ee606e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mixed MCTS won 66 games\n",
      "mixed MCTS lost 32 games\n",
      "the game was tied 2 times\n"
     ]
    }
   ],
   "source": [
    "# count how many times mixed MCTS won\n",
    "wins=results.count(1)\n",
    "print(f\"mixed MCTS won {wins} games\")\n",
    "# count how many times mix MCTS lost\n",
    "losses=results.count(-1)\n",
    "print(f\"mixed MCTS lost {losses} games\")  \n",
    "# count how many tie games\n",
    "ties=results.count(0)\n",
    "print(f\"the game was tied {ties} times\")      "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}