price_analytics_gh.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pricing A/B Testing - Donald Lee-Brown\n",
    "\n",
    "Originally a mock data challenge completed as part of the 2018B NYC [Insight Data Science Fellows Program](https://www.insightdatascience.com). \n",
    "\n",
    "The goal of this project is to generate insights from an A/B test that presented a random subset (the test group) of users with a higher price for a software product. The responses of these customers, as well as those of a control group shown the normal software price, were recorded along with some other information over a period of three months. \n",
    "\n",
    "Key questions addressed by this project:\n",
    "* What is the impact on conversion rate due to increasing the price of the software?\n",
    "* What is the impact on overall revenue due to increasing the price of the software?\n",
    "* Can A/B testing be made more efficient in the future by estimating how much data is needed to detect test/control differences at a significant level?\n",
    "* What are some other insights from the data that can be used to increase customer conversion rate?\n",
    "\n",
    "The project is organized into several sections:\n",
    "* Data cleaning and exploration \n",
    "* The impact of software price on conversion rate and revenue\n",
    "* Other insights\n",
    "* Conclusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports used in the notebook\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import missingno as msno\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "import math\n",
    "from scipy import stats as spst\n",
    "from scipy.stats import chisquare\n",
    "sns.set_style(\"darkgrid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data cleaning and exploration\n",
    "\n",
    "There are two data tables - one (test_results) contains information regarding customer info, which software price they were shown, and whether or not they opted to purchase the software. The second table (user_table) contains location information for users. \n",
    "\n",
    "### Reading in, joining, and looking for missing values\n",
    "\n",
    "First read in the data. We know that we can expect test_results and user_table to be joinable on user_id, but we need to check to make sure this can be done uniquely. This means we should check to make sure there are no duplicates in user_id (especially since it's not impossible that the same user would buy the software twice):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Empty DataFrame\n",
      "Columns: [Unnamed: 0, user_id, timestamp, source, device, operative_system, test, price, converted]\n",
      "Index: []\n",
      "Empty DataFrame\n",
      "Columns: [user_id, city, country, lat, long]\n",
      "Index: []\n"
     ]
    }
   ],
   "source": [
    "test_results = pd.read_csv('test_results.csv')\n",
    "user_table = pd.read_csv('user_table.csv')\n",
    "\n",
    "print(test_results[test_results.duplicated(subset=['user_id'])])\n",
    "print(user_table[user_table.duplicated(subset=['user_id'])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great - both tables have unique values for user_id (user_id is not duplicated), so we can straightforwardly join them. \n",
    "\n",
    "Once we do that, we can take a look at the joined data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>source</th>\n",
       "      <th>device</th>\n",
       "      <th>operative_system</th>\n",
       "      <th>test</th>\n",
       "      <th>price</th>\n",
       "      <th>converted</th>\n",
       "      <th>city</th>\n",
       "      <th>country</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>604839</th>\n",
       "      <td>2015-05-08 03:38:34</td>\n",
       "      <td>ads_facebook</td>\n",
       "      <td>mobile</td>\n",
       "      <td>iOS</td>\n",
       "      <td>0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Buffalo</td>\n",
       "      <td>USA</td>\n",
       "      <td>42.89</td>\n",
       "      <td>-78.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>624057</th>\n",
       "      <td>2015-05-10 21:08:46</td>\n",
       "      <td>seo-google</td>\n",
       "      <td>mobile</td>\n",
       "      <td>android</td>\n",
       "      <td>0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Lakeville</td>\n",
       "      <td>USA</td>\n",
       "      <td>44.68</td>\n",
       "      <td>-93.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317970</th>\n",
       "      <td>2015-04-04 15:01:23</td>\n",
       "      <td>ads-bing</td>\n",
       "      <td>mobile</td>\n",
       "      <td>android</td>\n",
       "      <td>0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Parma</td>\n",
       "      <td>USA</td>\n",
       "      <td>41.38</td>\n",
       "      <td>-81.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685636</th>\n",
       "      <td>2015-05-07 07:26:01</td>\n",
       "      <td>direct_traffic</td>\n",
       "      <td>mobile</td>\n",
       "      <td>iOS</td>\n",
       "      <td>1</td>\n",
       "      <td>59.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Fayetteville</td>\n",
       "      <td>USA</td>\n",
       "      <td>35.07</td>\n",
       "      <td>-78.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>820854</th>\n",
       "      <td>2015-05-24 11:04:40</td>\n",
       "      <td>ads_facebook</td>\n",
       "      <td>web</td>\n",
       "      <td>mac</td>\n",
       "      <td>0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Fishers</td>\n",
       "      <td>USA</td>\n",
       "      <td>39.95</td>\n",
       "      <td>-86.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   timestamp          source  device operative_system  test  \\\n",
       "user_id                                                                       \n",
       "604839   2015-05-08 03:38:34    ads_facebook  mobile              iOS     0   \n",
       "624057   2015-05-10 21:08:46      seo-google  mobile          android     0   \n",
       "317970   2015-04-04 15:01:23        ads-bing  mobile          android     0   \n",
       "685636   2015-05-07 07:26:01  direct_traffic  mobile              iOS     1   \n",
       "820854   2015-05-24 11:04:40    ads_facebook     web              mac     0   \n",
       "\n",
       "         price  converted          city country    lat   long  \n",
       "user_id                                                        \n",
       "604839    39.0          0       Buffalo     USA  42.89 -78.86  \n",
       "624057    39.0          0     Lakeville     USA  44.68 -93.24  \n",
       "317970    39.0          0         Parma     USA  41.38 -81.73  \n",
       "685636    59.0          0  Fayetteville     USA  35.07 -78.90  \n",
       "820854    39.0          0       Fishers     USA  39.95 -86.02  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_results.set_index('user_id', inplace=True)\n",
    "user_table.set_index('user_id', inplace=True)\n",
    "joined_test_users = test_results.join(user_table)\n",
    "joined_test_users.drop(columns = joined_test_users.columns[0], inplace=True)\n",
    "joined_test_users.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above table, we can see the various columns described in the data documentation, but it is also apparent that there are some issues with the data - for example the source column mixes underscores and dashes, which was not indicated in the documentation.\n",
    "\n",
    "We will make sure the data is clean later - for now let's check for missing values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x10411add8>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABdcAAAKgCAYAAAB5kkSnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XeYXVX1xvHvpEESeq+G6kKK5Id0RHpAuiC9SkdKQi+RIh1EiAiCCAiI9CpVuo0iLYiUV7ogTWmCtLTfH2tfchgmkJDJnDs37+d58mTmNnc8nHP3WXvttdrGjBmDmZmZmZmZmZmZmZmNvx51D8DMzMzMzMzMzMzMrLtxcN3MzMzMzMzMzMzMbAI5uG5mZmZmZmZmZmZmNoEcXDczMzMzMzMzMzMzm0AOrpuZmZmZmZmZmZmZTSAH183MzMzMzMzMzMzMJpCD62ZmZmZmZmZmZmZmE8jBdTMzMzMzMzMzMzOzCeTgupmZmZmZmZmZmZnZBHJw3czMzMzMzMzMzMxsAjm4bmZmZt1eRPQcx+NtXT0WMzMzMzMzmzy0jRkzpu4xmJmZmX1lEdFT0qiI6AdsC3wC3CnphfJ8myRPeMzMzMzMzKxTObhuZmZm3V5E9AceBGYFpgOGA8cA10ga4wC7mZmZmZmZdTaXhTEzM7NuKSKq85hdgJeBNYFFgSmBo4CNG4F1l4gxMzMzMzOzzuTMdTMzM+t2IqKXpJER0QeYCtgJmFbS0PL8HMBt5eVHAFc5g93MzMzMzMw6kzPXzczMrFspAfKRETE1cDvwCHACMKDxGkmvAKuVX48ENoqIHg6sm5mZmZmZWWdxcN3MzMy6jUaAPCJ6A5cCfYCzgMeBtSJi78ZrJb0GrAqMLq9ZqYYhm5mZmZmZWYtycN3MzMy6DUmjI2IKYBOgF7C3pOOBVcgmprtExB6V178OrAX8HvhjDUM2MzMzMzOzFuWa62ZmZtatRMQZwNpAb2BhSf8tj88NnA/MCvxC0i86eG9PSaO6cLhmZmZmZmbWopy5bmZmZk0vItoqv54OPArMARzceFDSS8D2wGvArhFxYPvPcWDdzMzMzMzMOouD62ZmZta0IqJnu997S3oS2Au4Adg8IvZrPF8JsAMs1i4ob2ZmZmZmZtZpXBbGzMzMmlKjhEtETAUcDcwLTAX8Fris/Hwu8HXgbEk/rbx3FuA/pUZ7myRPeMzMzMzMzKxTObhuZmZmTSsi+gMPA28D/wJGAhuSzUkHA+8DvwAWAH4p6dR273eNdTMzMzMzM5skXBbGzMzMmtkhwCfAlsBmkjYDfgN8G5hd0j+B3YF/AIdHxBbVNzuwbmZmZmZmZpOKg+tmZmZWu4iYMyIWa/dYD2BB4EVJz0kaWYLn2wMHAdNGxN6lzvqBwDnA5V08dDMzMzMzM5tMObhuZmZmtYqIXsAmwMERMWPjcUmjgY+BucvrNiLrrR8m6TSy1vrQiJhN0lOSDig12nt+/n/FzMzMzMzMrHM5uG5mZma1kjQSeA/YADg9IkZExI/L0w8AM0fEBcAVZMb6TyKiDZgBeBF4q93nuRSMmZmZmZmZTXIOrpuZmVntJJ0LHE82K30d+FN56kyynvo2ZHD95BKMnxdYCXhc0iddP2IzMzMzMzOb3Dm4bmZmZrWKiD7lx/mBN4C+wDYRsVgJpK9BBtuXB26LiEuB64EpgJ3LZ7R1+cDNzMzMzMxssubgupmZmdWiURu9knm+i6QBwAnA2sABEbG4pBHA6sDPyOB7G3AVsGRpctpL0piu/xeYmZmZmZnZ5KxtzBjfi5qZmVnXKgHxkRHRH9gbmAN4QtKZ5fn9yfrqtwAnSHp8HJ/T0zXWzczMzMzMrA4OrpuZmVmXiog2SWMiYmrgfmAU0BP4HXCapFfK6/YHDgBuB24EBgInSnqznpGbmZmZmZmZjeXgupmZmXW5iOgFXANMA2wOjJT074iYlsxif6oE4IcAhwD9gLeA+ZypbmZmZmZmZs3ANdfNzMysSzRqrBd9gdmAX0h6FegfEQcCzwIPAcMjYoCkYcD3yQD8fJJGRYTnL2ZmZmZmZlY7Z66bmZlZl4mIvsBqZKmXJ4FHyWD6JsAiwMXAA8AQ4B5JW7d7v2usm5mZmZmZWVNw5peZmZlNchHRVn48AzhC0kfAgcAywP7Av4FBkrYBfg48TNZi/wwH1s3MzMzMzKxZOLhuZmZmk5ykxla5c4HFImJjSVcAi5Y/gyTdUV4zJxDAK10/UjMzMzMzM7Px47IwZmZmNkm0L+FSaq5PAZxN1lzfUdI75bktgfWAZ4A1gd7AUpJGdvnAzczMzMzMzMaDM9fNzMxskijNR/tGxLcav0v6ALiDDKDPAxAR/YGpgKWBVYGnKIH1dk1QzczMzMzMzJqGM9fNzMxskoiIXsClwEbAMODGRumXiLgZaAO+J+nD8lhfoJek9xrvd+a6mZmZmZmZNSsH183MzKzTdFAKZnkyI/1g4EPgb8DhwIrAumRz0/vbB9Ijoq1Sp93MzMzMzMys6Ti4bmZmZp2iESCPiCmB5YB+wP2S/hMRA8hSMHsBvQAB6wCnSDqotkGbmZmZmZmZfUUOrpuZmdlEi4gekkZHxNTAbcBcZFPSk4AzJH1Uee3+wALALsC7wLqS/lLDsM3MzMzMzMy+Mjc0NTMzs4lWAutTAncD/wU2A7aiBNYjok/ltScDuwPrAW8AK0GWgunqcZuZmZmZmZl9Vc5cNzMzs04RESsCvwC2kPT30qB0XWBbYHqyqelVZCPTNkmjImIwMBT4hqQ3axq6mZmZmZmZ2QTrVfcAzMzMrGVMBSwC9IqIzcmg+lpkE9MRwIWAJD1WeU8b8BHQt4vHamZmZmZmZjZRHFw3MzOzznJ3+fMwGUx/DdgcuJVsbvoIsDTwGEBpcroBGZT/uMtHa2ZmZmZmZjYRXHPdWlZE9Kx7DGZmkxNJHwJbAjuQQfOVJF1O1mCfBfgAeL3yllfJhqcrSPp3Fw/XzKwpRMSUnreamZmZdU+uuW4tKSJ6llq+/YFdgHmBp4CHJd1X7+jMzCYPEbEm2bB0LmA/YEoykD6q1oGZmTWJElS/kyyNtbykkTUPyczMzMwmgIPr1rJKYP1BoA8wBuhf/uwv6ew6x2Zm1uoi4mvAb4EVgOfLn+9KGtFYAK11gGZmTSAipgC2Bo4HHgLWc4DdzMzMrPtwWRhrKRHRo/zdBmxPliJYV9ICZImCy4GzImKn2gZpZjYZkPRPYD1gNWATYFAJrPdyYN3MLEn6GLgAGAx8E7g2IvrUOyozMzMzG1/OXLeWExFTAleTzfHekrRj5bkBwOFkoH0NSY/UM0ozs9YVEW2SPjfBcMa6mdlYZbFxZPl5C3IxcgfgSmBLZ7CbmZmZNT9nrlsrGkPW9d0AmLME2wGQ9CLwK2AUEPUMz8ys+yg7gb70sapGYL2xm6jCK/pmZkUlsH4lcAQwI/AIsDFwY0T0qnF4ZmZmZjYeHFy3bq80gvpU2V67FnAZsBKwYrvn7wM+ARboqjGamXVHJatyTET0jIhpI2KaiOhdHvvCAHvJXh9dfv5hRCzV+N3MzFJEbAR8G9iDDKqvAHwfWBS4yQF2MzMzs+bm4Lp1a40SAxHRPyKOi4gjI2JzSZ8AWwF3AxdGxGoR0bu8Z3GyZMyz9Y3czKy5levryIiYmuxXcRfwVzKbcmBHZV8q722rZK/vAZxOBorMzOyzBpC7eu6XNFrSR8DvyN5BywBXNOawZmZmZtZ8XHPdur2I6Ac8BEwD9APeAy6QdFjJrLyJzF7/PfAGsATQC1jGtSzNzMatXF//CrwLXENeY5cldwdtAlzX/jraLrC+FzAM2EnSr7ty7GZmzaySILIleZ38nqS/VJ7vDZwB7EQubq7peauZmZlZ83HmunVL7UrBrEJmoX8bWBm4FtgtIo4tAZ51yAD794AFgaslfatkZHqrrZlZO5WSL5sCbcCukk6WdBS5UAkwglzU/LS2+jgC67s6sG5mk7sOyhg2mjs/QiZ97B4Rc1eeHwH8E3gUmAWYs4uGamZmZmYTwIFF63ZKDeCRETEFMA+wHFlD/WVJIyLilPLSnUug59CI2IIMEK0MnNv4LGcAmZmN1cikrJR8+RowPfByeX5z4FRgP+Bt4NyI2FnSf74gsH5Ol/9DzMyaSOPaWn4eAswHzEWW3LqWLGV4I/BRRJwp6aGImAP4JvAH4BBJH9YzejMzMzP7Ii4LY91SqQH8RzK43hv4i6Q1K88PIIM/mwHnSBpaMoauJEvEHABcUupamplZERF9gfUkXR4R+5LXywWBdYGLgaGSji9N+C4EVpN0f+X9ewM/BXZ3YN3MbKyIuApYGnia3EH8TeBh4FBgXuAC4FXgfeAjYAFgBUlP1DJgMzMzM/tSLgtj3UZjO20pV3Ae8CZwMHAJsHpE/KzxWkkvksGdS4BDImK3kjG0JfA34AigT9f+C8zMuoVhZKAH8hraj1zM/A2ZPXl8eW4K4HXyWgxARGxc3u/AuplZRURsT/as+D6woaSVgVOAVYElJF0GLAWcBjxI9rlYxoF1MzMzs+bmzHXrVkpzve2A/wMuk3RHRMwM7Ab8GPiZpH0qr58PWB84vVECJiKmBGaS9HKX/wPMzJpcRHyHbJ63naSLSkDoaOAdMgj0ETAAOAf4N7CBpNFl4XMQ0EvSjbUM3sysZqVs4ZySnmv3+OFkYH1FSe+WOep9ZFmY84GlJQ3r6vGaTQ6qpZnMzMw6m4Pr1q2UIM955dfVJN1VHp+FDLAfCQyTtG8H7+3lGutmZqmxG6hSB7gH0CZpVEScB8xPBoI+ATYia62/Xt7+CRlkX770uuhZ3vdp3XUzs8lR2cGzA/BDSS9GRH9J/4uIk4GtJc1W6qn/HbgN2KL8OR1YUtKztQ3erIWVsnfTSHrd8xUzM+tMLgtjTa0Ee6puI2upf0g2fwJA0hvAWWS5l71KYOgzHFg3M4OIWDIiZi2NS0dFRN+IWFjS6EpW1x1AAAMlvUvWAV6k/P0bsuzWsiWw3qvxPt+ompkxAlgYODUiHgJ+Unb23AmMiogzycD6LcBukkaT/YPeJ+e3ZjZpXE3OYTxfMTOzTuXMdWtajUzziOhNZlC+WrbRTg3sCRxDZqnvV3nPLMD+wPLAd8oNi5mZARExE/AH4FJJR5cFzCuA75GltW6VdG957TVkg70lxnUt9TZrM7PPi4jdgZOAnsBOki6OiGnIBcoNgIeAlUtG+3TAGWS5rbUl/beucZu1sojYAdgH2FnSfXWPx8zMWocz160plYDNyBJIv57MWH8sInYBxgA/Aw4D9omIUxrvKxnsx5P1LEd3kPluZjY5ewvYpgTW+wJ9yealhwF7AedGxLkRMSvwK+A9YDPInUQl+7LRWBoH1s3MUrs559eAV8iGz5tHxKIlaL4zcCswA3BxKRVzObA22QjagXWzTjCOe8A/A9MBq5fXtHXpoMzMrGU5c92aTqMGXkRMRWb2vEdu41sWWImsq/4rYDQwBDgKOFXSAR19TleO3caPj41Z14qIrwFzNDK1IqIXeV1dDlhE0hsRsTiwGjAYeBe4H1gZ+L2kPWsZuJlZNxARPRo7fCLiMmBuYEtgfXK3pYAfSXo0ImYAdgJWIQN9TwI/kfRkLYM3azGN8zEi+pBznxcqz+1LJhWsIumxusZoZmatxcF1axrVgGvJJDgDWBTYpDSeWQ64FugFHEfWWB8D7F1+Hyzp57UM3r5QRCxKZsh+IunRusdj46+jhZBqEMGaX7me7kD2qdhZ0rMlM301YChZtmAVSa+WJqe9yUXLuYDNy8cMknR714/ezKz7iIgVgJPJ4N0fSoBvCPBDMsB+iKS/t3uPv1PNOkmlrOgUwL3kveLVZK+YEcAcwDXAJZJ+6vJ2ZmbWGRxct9pFxDzASEkvVx5rI0vB3CfpRxExPZmx3pvMBhoEHAKcA/yPDBpd7KalzScifgMsCSwAvA38kTyWT3oy29yqNxwl+6enJDdb64YiYkMy4PMysAJZX/1Estb6UUAbsJKk1yrvmQHYhMyu3Mrnq5nZuEXE0eTOn6fJBck3K88NZmyA/edk2Zg/SfpHHWM1a0WVjPWpyeSr/sDswMbAc8BVZJB9KFnybqESiPeOWjMzmyiuR23NYGPgiojoDxARc5OBnr7AnOU1uwLbA2dLWg94gWxIcxEwt6QLy+SoVxeP3b5ARJxBlpU4FPgumQW7GrkoMk9tA7Mv1S6wfgLwe+DeiLg6IgY2zlfrHiRdSwbXlyVLvjwuaQRwJXA4mdl1d8lobxz/tyT9UtLmkkb5+mpm1rGI6E0Gzp8hd/3MWnkcST8DTgcWIXdh/gpwQohZJynzltHlnLuavO/4CdlPZgny/NwKeAJ4B5gN2BfAgXUzM5tYDq5bM3gRmB64PSLepjTPIwOyZ0XE18ksyx9KGh4RM5L/7b4L/Ad4qfFBzlxvHhExEzmZHQbcWEpKPA/0Af4ETBkR09Y4RPsClcD6xcAWZP+Dm8gMoLuAPUpDTGtyjeAOMIC8qXyLPH7LlVIEV5EBdigB9o6C6b6+mpmlUkLrU2Wx8ipyJ9Ao4PyImErSiEqA/edkQ9NdgZD0XBcP26ylRES/xs9l3tKPDKb3Ao6T9DbwP0nPMLYHwtXAjmQi1zqlfIybm5qZ2URxcN1qJ+lK4EyydEgbcE8J+Nwn6X7g/4DXyUwfyADRK+TNyY5lMuX/lptPHzJDC0mfRMQCZID2ejKD9giyIa01qVI79tvAHsDBkg4FtgOmBd4Hpva517wawZ8S9AE4XNISZDmYOYBjImKFspDSCLCPBp6KiBkcTDcz+7x2O7sGRcS2EbE2MFfZJbQ7MC9wU0T0bxdgv1PSRZKeru9fYNb9RcTMwN4RsWTl4YHkPcZKZBlRJI0pddhHSfqHpH3JnbRDyHvPbRqv69J/gJmZtRQHRaxWlczI2YG/kUH0kyJiHkkfl+c+ImvmHRIR3wVOIxvw3VsmTG1uBNWU3ifrOw8oE9+/knX0dyJ3HcwDLFTb6Gx8zAXMBDxSyi4tDPwFuAz4M3ACuevEmkzjRjIi+kXEoRFxOFmCC0nnAqeQ192jImKZEii6BjgJuJE8R83MrKLMORuB9cuAC8jr5hXAtRExWNJ1ZIb614FbIqJfCbD7vsus87SRJUOPjIilI+Jy4BEyO/1tMvC+GuTOu4hoa5yDkoZL+hVZpnKdiOjr89PMzCaGv0SsFpWMykZm5JFklsGpwHTARRExb3nuYeBOshHUhWSNylVKXb0ezjRoHhGxaUQcFBHblod+DuwJ3EfW7N5a0vtk5vMY4MnyPm/FrNk4bipeJ8+3OSNiPjKgfiewAxl83QpYp8sGaeOlXBdHloZefyWvnfsCP4mIYwAknUMG2GcFToiIXclyBg9J2roE5nuO43/CzGyy1JhzRsSRZA+L7YAAvkE2Mj01IpYCfgfsRmaw3x8RfZ0IYtZ5JL1BlnpZEbiDzFTvJ+kG8tybCRgSEcuX148h7z2q2oBFASdqmZnZRHFw3bpcJaOyf0QcFhEXAdtLel/SWcAZZDbsRSWD/SVy695qZDbCqiUDqJcnQs2jZIycRE5ojyaP1dnk9sweZCOhKDedJ5GZ6xeDt2LWrdEEqvy8aPkzL1nG533yeD1A7jzYTtKHZMDgLTKYYE2kLDxOSR6v14FBwDLAs2St9ePK6xoB9hnJhc0tKAte5flRXTx0M7PuYilysfkPkt4lF6JXAn5GlsVbtpSI2R/oTWlwamadR9LD5D1Gf+BDskkpkq4ABpNlXw6tBtgbCT1lV+0i5aP6dPHQzcysxTi4bl2qbKdtZFQ+SNa8+zowMCJmB5B0JnlzMh1wSURsSJYS+VDSvSVw1NP1gJtHROxHZnBtS5Z6WaPUFB1FBteHAgeSjUwvJQN9g0qDIatRyXJubHE/nyz58gBZg7svuViyAvAxcIqkjyJiLjJb6C2ySa01icoukFWBqYAhkp4A/svY47V9JYP9PHIHwiBgfvewMOs87XdleZdW91S9JkbEVKWk4TzA+5I+johFgL+T2bNHkXPbwaXO+rXAUpJe6PKBm7WgDuYoe5Pn3LfIHXqLAki6FNgHWAI4KCJWLo83EnpeI5MQ1pf0ThcM3czMWljbmDFOGLWuVUoNXAVMA2wGfCLp3YiYBZgTeEnSfyJiezLjZ2Ey63lRZ1I2p4j4DTCtpPXL720lO6RxjG8lM7fmA/4HPCPp9doGbJ8TEWcA65I3KVMCH5W6sUTE98hdBv8gG15+QC6KrS7p0XpGbFVlJ8/Iyt/bkjsOVpT0dEScSDanPZo8xmsBR0s6ot3n9PR11mziVc7FRkD907IDje/IGodnX0FEnA7cKul3EXE2+Z25Cdmv4m5gR0nvlR2ZASznRBCzzlO5rvYGvgb8S9JH5blVyMWse4ADJT1WHt+ETBw5RdL+7T7Pcx4zM+sUDq5blyvlCn4PXCNpWEQsCHwf2A+Ygiwzsbuk+0s20BzAnY2MSpeCaR6lbM8LEXEdmSU7qJIF3UYullwF3CHp0BqHakVE9AdmrmbRRcSswPXAJcDPqudYRMxEZrCPJsuGLEw2H77BOw+aQ2Uxa2rgFuBgchHrF8DKwIbAb4G1JN1WudEE+IWkPWsYtlnLagRsImIq8jycm9yNdztwoaTHPJ/pfiLiMeBtSd8pJSUuI0ukXS9pg/KaGcnF6H8CP5Q0orYBm7WQxjWzzHWuAeYHRgA/Ba6U9GZErFqe+yO5a/ZNcufe/MBjDqSbdR4nCph9lrd+W5eo1LfrSda1mwtYKSJOJYOvPyaDe3uU53YFkPS4pNsazfV8I9o8IuIG4Jfl10fIUi8rNZ6XNEbSy2Q5kXk//wnW1cpW2puAH7UrTzAVsDhZeqkaWO9J1qN8FJhF0smSdpA0zIH15tBo6lyO1X7AKGBEqUO6ccno2hU4uwTW+5Clm/5ABt4H1zR0s5ZV5iz9yPJ3C5W/HyYb790XEct4PtN9VJo7HwnMFhHrSXoQGAY8DiwYESuXHUM/Jeuxn+LAulnnaNwDljnMH4F+wHnkItbJwE4RMZOkO4HvkfObK8mm7hdLGu5G7Wadp3L/0aucl2aTPQfXbZJqTGIqq5ptkv5LBntWJGv9vk7W6P6BpAuBG4Gp2tfUc7ZB84iIxYGpgdNLkPYo4DHgrIhYtnHsImJm4D2yrI/VrARzdgf2KROiWcpT/wWeAFasPNY45/4N9AJm7+rx2pcrN5tTkIsm6wNXSbqvPPevUpppdjKTHXLxcjVguKQ/+mbTrHNVFi73AD4BtpR0gKQdyaBQX2CeiJi2rjHaF2t/TazMP/9MNvneovx+Bjn/eRG4ufw8P7CypCcxs05R5ipTkI2Bnwd+IOloSauTmeqHADtXAuwrkw3abwU2qn5Olw/erMVUFrt6AfeTPUamqntcZnVzcN0mmcq26P4RcVSpy31eRKwt6Xay/vbXge9K+kNE9I6IAWTGz/PO6mpOEfETMgtvZuCBkqE+GtiRDNJeD5wSEYcDZwLLkU1MrQlIeqLUhD0GeCQiFpD0b+AKMmCwaUTMUHnLR2RmkM/H5jUPGcQbSJZi+jTAVxYzbwL2jYi7ybIU0wIHNF7nm82uFRFTRMQCHTzuZpctoJJM8DVyJ8mr8Gnd3/2BvcjmwkdHxPS1DNK+UKW83dCIGFJ5/HXgMGCTiFhf0mhJV0j6Ltk08VvAupL+VsvAzVpU+X78HbmQtRB5DQVA0rbAdcBBZAb7LJIeAjYtiVsjShDQzCZS6XswKiL6krvWZyAXt7Yuj5lNthxct0mmUm/0IbJ53txkBuUNJdA+T+nOvlVEnA+cQgZhR5J18qw5/Y7c3j4P8M3K7oQngEHADWRm7NZkht5Kkp6qZ6jWkdLL4FayZM9lETGvpOOAc8ht7odFxHIRsQwZSJiarLNuTaCDXT0ir5nXAntHxIZlZ0LjdT8rz/8HuBpYqjQE6+laibX4LnB7RCwGEBErum5l99b+nCx6AFNL+l9EbEjW5x4q6QzgG8Dm5E2pNYlKCcO2iJgf2I78PnwgIjaLiNkl3Uj2tlg7IqauZLk/JelNSe/WNHwrquejFy27r8a51Sg/ARwD3AfMRu4Q+fRYS9qOnAPtDwyJiOkkfVxe0+bGwmYTr3EulfjOg8Cp5L3Fu8Bp5O6RKesco1md3NDUJqlSU30lYFPghXJBvpgsX7AaWUpkXeBEMjv2GWDX8rpengw1j4jYF3hE0l0RsTRwEZktu3EJ7lVfO1N5bqSkD7p+tDYuEXEysLyk5SNiKTLg819gI0nPRcQJwA7ATGSG0Ojy3KO1Ddo+1bgulvqGA8iGz3+T9HZEzA2cTV5z15N0R7v39m7UAPb1tT4RMRd5/VwceIrMbN5Z0pu1Dsy+knbn5FzATJL+WhYnbyXnOcsBB5INo0dGxJ5kebw1JL1W2+DtU43dlu0emw5YgJyjDgA+BIYA3yZLrK0g6VkvjjWPGNvgewqgb0nisW6q9K74Cdmw9K5yXb2ILDm5maSno9IcOiKuA6Ygd0X7nDTrZGXB8mxyt9ZWwNPkTr0fAvsA+5J9nj6qbZBmNXFw3SaZkk1wA/CWpK3LY1sCF5IX35eBKSVdUlZAP6xsxf3cTY7VpwRh7wDuAY6SdE+Z4F5Cbs3cokxwfYPZ5CJiNbJMyJaSrirH8WLGBtifj4hFyfIi7wPPlK3wVrPGDWRETE0G7WYhmwU/TTYo/WF57FfAKpQAe/XG05pDRHyDrFPZH9hR0vm+fnY/7c7JG8jg+rzkLrxDyLrr2wPPSVq2lCYYQH53vkiWLfAxr1l1zlkSCYLcrXVHY+dd2X2wPvB94HJyEfoyYDtJn9QycOtQOc8uJReej/J3YPcVEWsAVwF/Ao5td//xNrB5BwH2xnXZ36lmnawsXN5KbpzdpTzWWNT8CTC4/LlQ0v++4KPMWo7LwtikNpJsPtOoN3oRcJikn5O11Y8tmUEfVG5sXAO4yUh6ANiJDBocHhErSLqfsdvaLy21uz2JbSLtSxWU3x+jNHgqW/ceBLYEpgKujYj5Jf1d0u8l/cWB9eahsc1Lf09mUO4DLAKcS56ft0h6pTx+F3k813VQoXntIJCdAAAgAElEQVRUShTMQWatPwoMi4ily42JSxh0A5WeBqMrN5ofAPuRO0cukvQiWZLpQmBARDxGLlJfRzaJ3qJd+SarSWX+eTlwOHkMTyPPzVXKa66VtAM57/mA7EeyFFn+zppI2ZXVD1i7/D7a19buof1xknQb8ANgQeCIdvcf0wGXRMSC1WNcfu7hexKzSWIkmXw1V/XBcv6dDrwEnERp/O1rr01OPKG3TlGyRBp1KntDTm7IzJ9vlvIwjUyuE8uFdibgX5LeqQZ/PBlqHiVoAICky8n62/OTNUiXl/RX8stzauD3ETFfPSO19soiVSOLZw7Ic1LSG2QW0KbAYiWo8CBZI78PWQt6/pqGbV9uMXJXwQnAjZKeBP4NjACuiIglyUz2weRCyr51DdTGirG9KRrfb38ha6//gOxhcWtELOsAe3OLiIUiYp5285SlyXPypBKA/ZOkm0rQ/D2yT8kmZK3g4cCvgaUr5e+8+FWTdrW5v0M2SlxP0kLkMZsfODAiVmq8TtJNZF3nAAa5xnr9yr1Ho15+7/LwYGD+iNgDfG/RHTQC4hHRq5SDAUDSVWTfmPkYm+DzVzLAPg1wZ0TMXT3Gvq6aTbyoNAKuLF6NIuetS0fEOmXn15hy/r1K1mB/DfhlRKzqa69NThxct4lWJkON5hZnkQ0S9ytPHwm8QE5yz5F0YpnwzA8MJIM/1oQi4lfA4LLdHfg0wH4o8HXgqIhYpmSQbE+WFfEXaJNoTGYi4kzgpojYqWSqI+l84E7gmIiYVtKochx3Isv8+Kakec1S/ryibBq9NZm5fgS5bfpEYC1JzwDbAKvXNlIDPldyYkhEnEdmx44ovQz2J29UbomIpUpwoaeD7M0lsqfBdcCq7Z6akzwn/1le12hw2ZOss368pD9L2lnSYEk/1diGwu57UKPKAvTPyQVnAfeW564m5zsLAIeU4HvDKEkvSXqui4ds7VQylHsBqPQVIQM8dwHfiYjevp42v5Jx3ocsQblHu/uPK8jzcSHG3n88QC5Q3we8UseYbaxyns1f+d3nXDfWmKNExFQRcRLw68ZiJXnP8RIwjEwUaViw/L0/cDPwo4jo6/8WbHLh4LpNtDIZ6ktm4g0iS4ecFBG/ITNhNyIzY9eLiN9ExC+Bq8tze4O/gJvUNMBRwNYdTHAPJwMMh0fE6pLuAZaT9Hw9Q7WGDs6lvwGPA2cC10XEkPL4hcDMwDcaL5T0F7JBm49jExjHdfHZ8vdSkfXzLwSGSjoBeBdYAZgHQNKzje3RXTFe61glsH4lGWwdQGY7z1uef5hskvgocFtEbEXeuOxUy4CtQ5JeAgZLOi8ipmjsCAKeJOfT65bXjSoZ6SOA24FlSsZ7j3af5/J3zWNmsmfFAGDaxoOVgF4jg3218riPXY0iYq6IWLax86Pssrw9Iq6PiJUior+kt4HzyB0Ig5w92W2MAJ4nk7O26eD+4zCyp8zhEbFmKV+4Sbnu9uzwE62r7AAcFBEzl99n/qIXW/Mqu59HRUR/MvljE7Ix+89LfKc3WXbrHeC8iLg1Is4ld0Yj6TrgX+XnD339tcmFb7jtK2s3iRlAZg0MIoOum5JNn84iG86sQNZbnwmYm6xPumQle8sX3SYjaTPgN+Sq9LbtJrgXkRleiwDbR0Q/uSt47arnUiPzVdKZkrYia8iOAvaJiAfJerFzATtXP8PHsTmUoEEjg3nmiJiunGcCfgucDdwG7Cnp+PK2AWQmyQvVz/L26PpFxIHkbq11gTUkrQ48HRH/FxGLSPobsCPZ5PTXwF7kTgRrAo3AuKRbStmJy4ALI2K+svvgIuDIiNi0vK6RkT4bGXx/y+dhc+hosVHS5mSN9SX5/HznCuBgMrCwe0kmsXp9l/we/EY5HnuRvUgWBM4HbohsevkXsu/B7pWAnzWRDhYdx5T7j0uBU2l3PpJltl4jj/XK7d7rRa96PU0mBZwUETcDJ1fLilj3UCnP1ANYi7ynWLH82YqM75wDvClpKfIa+wGZMHI7sHz5qFmB57xzyCYnbWPGOKZpE66xzb1MageRWXjflLReeb6NDLCfT2ap7yTpw+p7y8+9vC26eUTEN8gGpR+WbEoi4iwyG2Ef4AJJ75dtf6cB1wI3SfpXXWO21O68OoasBzuA3F57gaRHImKa8tjRZEbJHOX31SXdWc/Irb0ysR1dbigvITPRpyIDr0eQk9hfAsuQfSyuJWuxHwm0Ad/xTWZzKWUn5pO0TkTMRS5C/5i8+Xgb+IWkYyNiWuD/gOeVDTGtCbSb8yxKzm82IDO6BpM7vc4gG1weCdxClos5gqy7vp6TCOrX7nuyo/nOmeQi1z7AhZLeq7x3A+DxUnLLahARU5U56Oxk8sdi5alLJQ0u5+eu5Lm5LHAF+d05FTBE0hON79c6xm+f1bgHLDsPliG/D99RNjGtno/7Ar+W9L+yaHIAuQh9s49lc4mItckFkA/Jpt031Dwk+wrKOXkuWRbtUUm7Vp7bjIzvXEPGdz4oj09P7vyaAdgD+B65G/rJrh29WX0cXLcJVrJhx0TWWH+QvIjORGaubyLp3sprNyMnQJcDB0t6rY4x25eLiAvIye3XyfISjwC7SHqmTHB3Ius5P0VmQS9Pfmm+U9OQraguUkXEZcC3ycntdGR2+rLA98s2vcZ71iZ3lOwELCXpn10+cPucShCvD5m5PJqcwM4JrEY29FqVzNw6nGxE+wEZoH0eWFPSiGoQyepTsrZGAyeTx++PZPB8OeBG8jxdkzyua0r6T01DtXFoF1h/CrhD0g4RMZQM/PyVvJGckVzs2oo85m8ALwKrlnPSQb0aVf///5L5zllkHed9aRdgt/pExIpkWa3dJP0rIhYjS2mNALYEri73Jo17lB3JzOa1yHPzKkmb1DR8a6ddEsFd5Hx1PnJX5cNk0O6pcj7uRO6EfpEsNfoOsHYju9bX1eYREbuRC82jyR1eQ50o0P1ExDfJBJCVye/Bwe2+Q6vxnUMkvRoRywHHkcldrwPbl519ZpMNl4Wx8VbKEzRKFfQggwUvkDchPwCmBw6OiIUb75F0GdnsclvyJtSaUEScQQbMjyUDd0PJ8j2/i4iQtDvwCzJb5DRgDWBrB9brExH9yiSWkvnTIyLWIXeRbAPsLWkL4BSysd7CJXOd8p6bJA0ls2kdWG8SJYg3JVly6SNgX0knSRpMXmfvJsvB9Ja0A7A4GWDfjNyBMKJcpx1Yr0EHNV9Hl5uRs8iAwEZk4+fdJK0v6RzgenKRuneXDtbGS+WcXJVswn5mefxYsqbzMmQw4V1JPyBLi2wJbA6sVDknHQCqUSUo8GXznd3IMj8nAruURBKr3xTAC5WdkrOSDYYfJuenK8LYZu6SziXLxaxO7qD9ekQsC+7z1Aw0tnnptcB/yXvEhchShTMCV0fE/OV8HEZeUweTyQTrVxZSfF2tUQdznl8Bs5OLzJsCx0XEvF0+MJsgHZRn+hv5HXknsFdEbFY91zqK75TkymPJWuxrOrBukyNnrtuXioglgX9JerX83p+8kC5LlgS5qDw+iMzCu5lcqX6i8hmrAX9wCZjmExFzkzceZ5LbLseUydI3yBXpdyQtX167FFm3+9XGfw9Wj4g4EdgN+Kmko8pjOwEnAMtL+kdEzEfuLrmaDP7sDhwl6eXK57S5XEFzqGTcXQesR2anL1td/Chbon9N1pLdQ9In7T7DWVw1aVdy4iAye2ce8vy7AniT3DL7bmWnyQzkguUcwIaS/lvD0O1LRMRV5BbnZ8jm3W9WnvsRWTrtr8Chkp5r917vImkSEfE1suHa+Mx3LiZ3myykbI5pNWmXMdmfzI68kix7NydwAbAwsKmkP1Te1/hOnYE8Py+WdHiX/wOsQxExkCx9N0TS78tjjfPxKuA1SSuVxxcARgL/LIF5lxWtWbs5z9pkkt0fgdclfVKym39Dzn8OcSJPc2pXnml+slzoI8B/yOvrz8kSwNtKurzdex3fMatw5rp9oVI/60Jgl8rDS5JZBJuT2XYASLoVWJ9sNHRMqWfZeO6OcuF2Y5PmMw3wTeD9yhbLUcATwGHA/0XEVgCSHpD0sAPrTeFnwE3ANpE11gE+AfoAz0fEADKwfhtZsmB6sgnNItUPcWC9fo3Mn8qx2JtszjYbsGw1o0TS/WS5mBXo4DvcgfWu18iCrNxkXkmeczOT5XqOIDMsNyxB2cUi4pKI+C2Z5bUOGVxwYL157UfuGpkPWLvchAIg6Rgyg/3/gF9GxGzVNzqwXp/22XjA1GQQ9oNxzHeWiIjNy2u3AhZ3YL0pVDPNg8xI3488Pv8kd3U9AVweESsARMSUlSz2t8ig34plF4rVoIMs59nIQN77jefL+fgkuYDyrYhYs7z2WUkvlMB6Dwfz6lW5dhIRlwAXk4H0e4EtI6J/yW7ehrz3OC4iVoyIgyJiwdoGbp/ROJdKeaY7yXuPG4G/k4uWH5GJWbeQTdw3rb7f8R2zz3Jw3cYpImYqNxUbSfpxRPSNiNlKVshqZAbzdiWTEvg0wL4eucJ5VkTMU/1MT4aaR0RMEREzAv8iG88sVHmuZwnSNWo+z1jPKK0jETGFpFfIAN6DwEYRcQA5+XmL7Nb+MLmLZBdJH5e3fkQG+6xJNEq4RMSUETEoInYgF0B+CPyBDPjM124b+6vAv8nyIlaTiOgNuSjSCBqUTK3lycXnLSRtTAbXlwZmjYh+wP/IoMLXyEy8FcoWXGsCjZvExjlXbj5fILc//51sCL1sNVBUAuzXktfXN7p6zPZ5lXkMEbFQmatOT55z83fwuj+R89pZIc9ruU9Q7dp9R65MHqM9gXWBH0fE4uX83B54HLg2IvYHroqIbcpnLA9sSCYgeAG6Bu2O44oRsS5Z6qcXmbQFMKYSYH8Q6Ff+fCYRxEkE9dNn+1csSwZgg8x2HgpsEhH9SoB9K7Ksz7XkfMhlmZpEWayaggyov08uXM5H7pD9Ftnz6RNgCJnQdV5EbNfB5zi+Y4aD6zYO5UI7NCKGSnqqPHw+8ExEzC3pLjKIvjhweEQs0Xivssv7FuQE1lvAmlBEnEyuUN9PZtudARwREetIGl3JtJuJbCD0csefZF2tBHoawfJNyEnqgmTjmd3JXggzkwGEnSS9GxEzA9uRk97nu37U1pF2GSP3knVjf0luob2YbKj3ETm5XTsi5o4s07U+WXv243F8tE1iJfC6V0T8GD6TnTwXWRP2SUnvlwytI8kblYeA/ST9A1hX0orAdtUSalavEtgZGVln+4SIuBo4I7Le6MtkQO+/5I6DFdsF2A8CNmtkVtbyDzDgc1mVF5JBnXvJRcubgKMiYu1xzHdeqmPM9nntviP/Qp53jwBLkQH27wLHR8RAZdPEbch57Y/InbWXlI96hlw8OaB9KTWb9Do4jueSZUQ3JMumndzB+Tg78A+yMaLVLCJ6R8RM7R77Fnku7i7pEjJZ617yWno8GWDvL+nK8rqjgMXKHMhqEBFztN9dR8YB5iKP2fVlh8gBwEnkgvRJZDznaOABciHTzDrgmuvWoRJcP5VsBPRbsmHeHWTDoD7AqpJeKjXWriczZg+T9HAHn+UawE0kIi4lsyivJYNA15ABvEYTksFk4H0q8uZlNbKGtwPsTaQEDFYhb1JeJWv+9iEnPs+Qx/EtckLUH1gUWE1uMNNUSvbzDWQzy6FkYOdbZHZPTzLb53SyAd9b5NZ3yOaln4Rr5teiZDcfTi4kn0Yen9PJG8jDJU1dbmCeIEszbQGsRQYSlpU0vHyOj1+TiLH1mfuRWZOjyAXJaYGBZN3RIWQN0lvIjMtdgT9XS7/4mDaPiDiNTAQ5gNxRMCcZfD0NWJPc/fVH8jvS850mVL4jbyITwg4hG0PPLOkvEbEGeS7+DjhG0kPlPUsAw8tCV29lU2GflzUax3GcjjwvzyDPx/2Av5Hn44+Aj4FVfA9Zr7JY/DC5m/L4xo6eiNiITBwISa+VnZc7kjsujyZL/vyI7M/2hs/BekXENORi1ZnASZXjuA1Z2m76khTSp7EIGRGnkouWC5dj+HXgGZ+TZh1zcN3GKbLe+m+BVcnV6OXJ1c3TyMnRKpUA+3VkCYpjS01ga0IRsQEZINhc0j3lsRmBNchSFHORGc7vkxPfUWSdYAdkm0hELExu4TtE0qXlsVnICe2SwJ/Jc/cHZPmJZ8nmbU/XM2Ibl4gIcqHrYEnXlcd6kDcl15Dn4tZkRvsq5edrS2C9t6QR9YzcSvD8TPK4TE0GYHuRiyV3kbXUbyYbz75dalWeCnxbkneQNJFKYL0HuYV9L7J511MRMQewGXAicIakfSJiLvI4z0POhR6pa+zWsXJ+3kA2sTyl3XPLkru91gDeJa+zI/F8p+lExEKMne9cXh5rA9pK8PxMYOfymhMk3Vt5r5sJN4mOjmPluW+TCT7LkrueXweeA9YsCyNO0qpZRKxHzkmPB34h6dWyM+8mshTs1GTT4KMkHRcR3yWP9ytkgsHOwCgH1+sVEeuQ9xyN+cyrZUfsPcBgSWeW1/Up9xlfB54C1iqlfxuf43PSrAPetmrjVOqtT01uge5BBmRvITNiRwN3lRIxN5FlCtYlm5ZY85qRzFJ/LCIWiYiDgRfIFeuhZK3RNchyFHuSQSDfaDafnmTpl48gM4IkvUFmirxBNiBeSdJgSZtJOtSB9aY1GzAv8B58Wpd0NFme4CgyYLswmQ30NHmMl4msu+/Aeo1K1s975K6Dl8jdBI+QN6BbktfWwSWwPgMZbH8F9z1oOhpbN/9P5JbnNxol8ZT9Lc4isy33jogNSmbzhmTGrOvlN6ee5OLHpwGAiOhZArPTlOd3Bw4kM9g932lO/ci56ac1fUuArldE/ICs87w7uUNh4+obHVhvKp87jhHRVhY05ysPbUTeT25Kfp+OqMyJrEaSrifLMA0F9oiIOcp9xWYlWeBI4BZJx5W3zMfYpphHSxrpwHr9JN1IzkUPJY9jIwHrHmD7sihCCaz3AL5J3o+80O5zfE6adcDBdfsyW5Nfpo+R9WWHSLqZsQH2O0uA/WZgBfLm05rXY2Qjr5vJrbTHkrWdB5HbodcCPpR0jqTfSfpXbSO1L/I2GVhfBqDcgPQpAfZDyfIwh0bEEfBpJrQ1p5eAEeT5R6lL2miw9wgwJTBTCeatSWZYXs3YBmBWr2OB75EB1j0iYjdJe5ILljORTfV+RR6z9YAdJb1T22htnEog7jhyJ8IiJWO98dyHwGVkmZ8lymMvSNpW2aSvZ0efabX6iMxKX67yHTi6BHiGk7syX5f0K893mtprwIdkAg/w6U6TT8gSPysDt5LfiQfXMUAbL587jvBpkO5BsuzoO5Juk/TXsiuhh9wosWkoe6qtRd5n/DAiZqqUg52FkjhQkgmWYWwm+3N1jNc6VjLQG8dxL/K+4iDyGP44IvYox3AVshTeP8hyo2b2JRxwsS8k6cVSw3BP8qZyz0qAfW+yg/TjETGLpHtLYKhXnWO2cZP0ALnlvTdZK39LSbtK+jO5Q+EF8phaEyuB1p8CB0Xp2q6xTbrmIrMQLiWbEDvDoImVm45hwCGVY9nItpubPJYvl8dfJjO7RN6oWs2UbmXsd+SBJcC+E3nj8jy5oPk3spazs5ybWMnqWpUsy7RpaWzaeO5lsk/JLB28zxmyTUbSm2S9343J7PRGxjPkrszHyKBCo8yINaGyc+RgYNuy27J6HGckSxZML+lh34M0r/E4jk+Q9yHV93ju2mTaBWaHlBJpkAHYFSPiAuAcMjFvmKSP6xmpfZHKcTwE+LGypO9m5G7MU8l+MxeWl68rN2o3Gy+uuW7jLSIGkPW6g6z/ex2Z5dUH+L5vLruPUlLi48rfXyMnvYPIrdEO3DW5klXwc7JR4mFkKYopySDfrMDWpbSTNbmImIk8lo26zn8gGwrvT26hXkmfbZjYy9lczafyHfkNsjfJBeRi1z8lvV/n2GzCRMSa5AL0AcAlykZeiwKXA2dLGlbrAG28RERfsjn0AWRpn8vIXZfbkzuBlpOblza90mT4cPI78TfkzssxZFblSGBlB2Kbn49j64iIQeTxO678mYdczFwE+DcwRNLfaxugjZfKcTyB/K7sT5aqXJRMuLun7M7zfYfZeHBw3SZICR78jCwBMw2Z+bNMY1u0A+zdR8k2uI/M3PqIzMZbW9LwWgdm460EZfcF9iO32/6PbEK7vo9j91IaC+9N3mROSdbmfhpYp5T98fW1GyjfkcPIxmx9yPNxEUn/rnVgNsFKgP1mMqPyGbI/Qh9yzuN+B91ERPQne1YcWx5qNDDdzDXWu48SmN2K7DvSj2x6+Sywntz0stvwcWwdlcDs8cAx5L1kb2AKSe/VOTYbf+0WSk5vn2Dn+w+z8efgejulxuaTwBHOTOpYCcquSd5onlipEewLbzcSEVOSDdlWIVenr5DkmmrdUEQsRGaLvAc87tqx3VdEzAvMSQaAHi9bMZ0x0o1ExNxk8GAAcKZLwXRfEbEKcAdZO/ZiSaeVx3s7wN69RMQ85M7L94FnJL1e74jsq4iI2cl+Fp8AT/s7snvycWwNJTB7M7lb7yRJr9Y8JPsKImINMsB+Blkm5s2ah2TWLTm4XlFqa95ONuHYx8H18ePJkJnZpOEsru7Lx641RMRqjN02/VM3pDVrHr7OtgYfx+4rItYBfgss6F163VdErEfWYF+h0g/BzCaAg+tF2cp9NbBEecjBdTMzM7PJXMnOu4msqX+Cs57NzMxSRPSX9L+6x2ETJyLaJI1p/F33eMy6G3f9BSJiCFk7fHHgzpqHY2ZmZmZNQtKtwAbAD8iGmGZmZgY4sN4aHFg3mzgOrqchwIvAd8ju5WZmZmZmAEi6EZjT297NzMysFTmwbvbVObiedgUGSrqn7oGYmZmZWfNxdp6ZmZmZmbXXq+4BNANJv697DGZmZmZmZmZmZmbWfTi4/hWsvPLK3i7TAoYNy361Q4YMqXkkNjF8HFuDj2Pr8LFsDT6OrcHHsXX4WLYGH8fW4OPYOnwsW8fdd9/dVvcYJoFJGnscMWIEgwYNmpT/E53urrvumpi3T7L/RlwWxszMzMzMzMzMzGwyMWaM84Y7i4PrZmZmZmZmZmZmZpOJtrZWTPavh8vCmJmZmZmZNbGBAwcCcPfdd9c7EOsUPo6twcexdfhY2uTIwfXO4+C6mZmZmZlZExs+fDjgusDdnes7twYfx9bhY9k6vEAy4UaNGlX3EFqGy8KYmZmZmZmZmZmZTSacud55HFw3MzMzMzMzMzMzM5tADq6bmZmZmZmZmZmZTSZ69HBIuLO45no7ks4Hzq95GGZmZmZmZmZmZmadbvTo0XUPoWV4mcLMzMzMzMzMzMzMbAI5uG5mZmZmZmZmZmY2mXBD087j4LqZmZmZmZmZmZnZZMI11zuPa66bmZmZmZk1sYEDBwJw99131zsQ6xQ+jq3Bx7F1+Fia2cRwcN3MzMzMzKyJDR8+HIAhQ4bUPBKbGMOGDQN8HLs7H8fW4WPZOrxAYnXyHgAzMzMzMzMzMzMzswnkzHUzMzMzM7Mm5rIwrcXHsTX4OLYOH0szmxgOrpuZmZmZmTUxl4VpDS5B0Rp8HFuHj2Xr8ALJhBszZkzdQ2gZLgtjZmZmZmZmZmZmZjaBHFw3MzMzMzMzMzMzM5tADq6bmZmZmZmZmZmZTSZGjx5d9xBahoPrZmZmZmZmZmZmZmYTyMF1MzMzMzMzMzMzs8lEW1tb3UNoGQ6um5mZmZmZmZmZmU0mxowZU/cQWkavugdgZmZmZmZm4zZw4EAA7r777noHYp3Cx7E1+Di2Dh9LM5sYDq6bmVlTGDhwoCe2LWD48OF1D8HMrOU0rq1DhgypeSQ2MYYNGwb4OHZ3Po6tw8eydfg+csL16OFiJp3FwXUzM2sKw4cP98S2BTRuUszMzMzMzKw5jR49uu4htAwvU5iZmZmZmZmZmZlNJnr27Fn3EFqGg+tmZmZmZmZmZmZmk4mRI0fWPYSW4bIwZmZmZmZmTcwNTVuLj2Nr8HFsHT6WNjlqa2urewgtw8F1MzMzMzOzJuaGpq3BzRNbg49j6/CxbB1eILE6ObhuZmZmZmbWxJy53lp8HFuDj2Pr8LE0s4nh4LqZmZmZmVkT+3/27jdW1vU8C/s1M+uc45a4LSFJWxSnQcR9sUnFGxGO6ogGuzj8KbQUmhaBkMKHiiiVCqNKddSI1CVOQXJomKC2IdAWqDCNSoVJUdQ2aXoGheD2CIeXKLE7tXEC7YdEUf5JAR97r5nph7WWtX2y986Zs16f51338/tJ1rLWHq99ey7pnL2v95n7cXK9Bqdka5BjHbKswwMSWnKhKQAAAABAJ87nc+sRylCuAwAAAADAhZTrAAAAAACd2Gw2rUcoQ7kOAAAAANAJa2Hmo1wHAAAAAOiEcn0+V60HAAAA4OnGcUyS7Pf7toMwCznWIMc6ZEmPVqtV6xHKUK4DAAAs2DRNSZLtdtt4Eu5jt9slkeNDJ8c6ZFmHBySXW68tM5mLdxIAAAAAoBPH47H1CGUo1wEAAAAAOnF1ZZnJXJTrAAAAAACdcHJ9Psp1AAAAAAC4kHIdAAAAAKAT5/O59QhlKNcBAAAAADqhXJ+P7fUAAAALNo5jkmS/37cdhFnIsQY51iFLerRarVqPUIZyHYBFGMfRH2wLmKap9QgA5dz9s3W73TaehPvY7XZJ5PjQybEOWdbh75G0pFwHYBGmafIH2wLu/pICAAAA1dm5DgAAAADQCWth5qNcBwAAAADohAtN56NcBwAAAADoxPF4bD1CGcp1AAAAAIBOPP/8861HKEO5DgAAAAAAF7pqPQAAAABPN45jkmS/37cdhFnIsQY51iFL4D6cXAcAAAAA6MSjR49aj1CGk+sAAAALNk1TkmS73TaehPvY7XZJ5PjQybEOWdbh0weXW6+dt56LdxIAAAAAAC6kXAcAAAAAgAtZC7VgBnIAACAASURBVAPAIozj6ON8BdytLgAAAGCZVqtV6xHKUK4DsAjTNNl3WMDd7koA5jOOYxI7ZauQYw1yrEOW9MjO9fko1wEAABbMhaY1uDyxBjnWIcs6PCChJY8pAAAAAAA6cX193XqEMpxcBwAAWDBrYWqRYw1yrEOW9MjO9fko1wEAABbMWpgarKCoQY51yLIOD0gut9lsWo9QhrUwAAAAAACdOJ/PrUcoQ7kOAAAAANAJa2Hmo1wHAAAAAOiEk+vzsXMdAABgwVxoWosca5BjHbIE7kO5DgAAsGAuNK3B5Yk1yLEOWdbhAcnlnFyfj7UwAAAAAACdOJ1OrUcoQ7kOAAAAANCJzWbTeoQylOsAAAAAAJ1YrVatRyhDuQ4AAAAA0AlrYeajXAcAAAAAgAsp1wEAAAAAOnE+n1uPUIZyHQAAAACgEy40nY9yHQAAAACgE3auz0e5DgAAAAAAF1KuAwAAAADAhZTrAAAAAACdOB6PrUcoQ7kOAAAAANAJF5rOR7kOAAAAAAAXUq4DAAAAAHRivVYJz8U7CQAAAADQCTvX56NcBwAAAADoxGq1aj1CGVetBwAAAODpxnFMkuz3+7aDMAs51iDHOmQJ3IdyHQAAYMGmaUqSbLfbxpNwH7vdLokcHzo51iHLOjwguZyT6/OxFgYAAAAAoBOn06n1CGUo1wEAAAAA4ELKdQAAAAAAuJByHQAAAAAALqRcBwAAAADohAtN53PVegAAAACebhzHJMl+v287CLOQYw1yrEOW9Gi9dt56Lsp1AACABZumKUmy3W4bT8J97Ha7JHJ86ORYhyzr8IDkcsfjsfUIZXhMAQAAAADQic1m03qEMpxcBwAAWDBrYWqRYw1yrEOW9MjJ9fko1wEAABbMWpgarKCoQY51yLIOD0gu5+T6fKyFAQAAAADohAtN5+OdBAAAAADoxOl0aj1CGcp1AAAAAAAWaclrbOxcBwAAWDAXmtYixxrkWIcsYfmWfAGrch0AAGDBXGhag8sTa5BjHbKswwMSWrIWBgAAAACgE3auz0e5DgAAAADQifP53HqEMqyFAWARxnH0cb4C7lYXAAAAsExOrs9HuQ7AIkzTZN9hAXe7KwEAAFim9doyk7l4JwEAAAAAOqFcn493EgAAAACgE9bCzEe5DgAAAADQCSfX5+OdBAAAAADoxGq1aj1CGcp1AAAAAAC4kHIdAAAAAAAupFwHAAAAAOjE8XhsPUIZynUAAAAAgE5sNpvWI5ShXAcAAAAAgAsp1wEAAAAAOmEtzHyU6wAAAAAAnVitVq1HKOOq9QAAAAA83TiOSZL9ft92EGYhxxrkWIcsgftQrgMAACzYNE1Jku1223gS7mO32yWR40MnxzpkWYcHJJdbry0zmYt3EgAAAACgE6fTqfUIZSjXAQAAAAA6cT6fW49QhnIdAAAAAAAupFwHAAAAAOiEnevzcaEpAIswjqOLaAq4u3QPAACAZVqtVq1HKEO5DsAiTNOU7XbbegzuabfbtR4BAACAZ3Ch6Xx8BgAAAAAAoBNOrs/HyXUAAIAFG8cxSaxPK0KONcixDlnSo+Px2HqEMpTrAAAAC3Z3n4X1aQ/b3eo0OT5scqxDlnV4QHI5J9fnYy0MAAAAAABcSLkOAAAAANAJJ9fno1wHAAAAAIALKdcBAAAAADpxPp9bj1CGch0AAAAAoBPrtUp4Lt5JAAAAAIBOnE6n1iOUoVwHAAAAAOiEcn0+ynUAAAAAgE5YCzMf7yQAAAAAAFxIuQ4AAAAA0InNZtN6hDKU6wAAAAAAnTgej61HKEO5DgAAAADQCTvX53PVegAAAACebhzHJMl+v287CLOQYw1yrEOWwH0o1wEAABZsmqYkyXa7bTwJ97Hb7ZLI8aGTYx2yrMMDksudz+fWI5ThMwAAAAAAAHAh5ToAAAAAQCdOp1PrEcpQrgMAAAAAdGKz2bQeoQzlOgAAAABAJ+xcn48LTQEAABZsHMckLmyrQo41yLEOWdKj4/HYeoQylOsAAAALNk1TkmS73TaehPvY7XZJ5PjQybEOWdbhAcnlrIWZj3IdAABgwZxcr0WONcixDlnSo9Vq1XqEMpTrAAAAC+bkeg1OydYgxzpkWYcHJJc7nU6tRyjDhaYAAAAAAJ2wFmY+Tq4DAAAsmLUwtcixBjnWIUt65ELT+SjXAQAAFsxamBqsoKhBjnXIsg4PSC5n5/p8rIUBAAAAAOjE+XxuPUIZTq4DAAAsmLUwtcixBjnWIUvgPpTrAAAAC2YtTA1WUNQgxzpkWYcHJJezc30+1sIAAAAAAHRis9m0HqEMJ9cBAAAWzFqYWuRYgxzrkCU9Wq+dt56Lch2ARRjH0R9sC7hbXQDAfKyFqcEKihrkWIcs6/D3yMudTqfWI5ShXAdgEaZp8gfbAu7+kgIAAMAyObk+H+8kAAAAAEAnHj161HqEMpTrAAAAAACdeP7551uPUIZyHQAAAACgE+fzufUIZSjXAQAAAAA6sVqtWo9QhnIdAAAAAKATTq7PR7kOAAAAAAAXUq4DAAAAAMCFlOsAAAAAAJ04nU6tRyhDuQ4AAAAA0AkXms7nqvUAAJAk4zhmv9+3HoN7mqap9QgAAAA8w3rtvPVclOsAAAALNo5jkngIXYQca5BjHbKkR+fzufUIZSjXAQAAFuzuU0Hb7bbxJNzHbrdLIseHTo51yLIOD0guZy3MfJTrACzCNE3+YFvA3V9SAAAAoDoLdgAAAAAAOnE8HluPUIZyHQAAAAAALqRcBwAAAADoxHqtEp6LdxIAAAAAoBMuNJ2Pch0AAAAAoBPn87n1CGUo1wEAAAAAOnE6nVqPUIZyHQAAAAAALqRcBwAAAADohLUw81GuAwAAAAB0wlqY+SjXAQAAAAA68dxzz7UeoYyr1gMAAADwdOM4Jkn2+33bQZiFHGuQYx2ypEer1ar1CGU4uQ4AAAAAABdych0AAGDBpmlKkmy328aTcB+73S6JHB86OdYhyzp8+uByx+Ox9QhlOLkOAAAAAAAXUq4DAAAAAHTidDq1HqEMa2EAAAAWzIWmtcixBjnWIUt69Nxzz7UeoQzlOgAAwILZuV6D/c41yLEOWdbhAcnlVqtV6xHKsBYGAAAAAKAT5/O59QhlKNcBAAAAAOBCynUAAAAAgE4cj8fWI5Rh5zoAizCOo115BdztBQYAAGCZ1mvnrefinQQAAAAA6ISd6/Nxch2ARZimKdvttvUY3NNut2s9AgAAAM9gLcx8nFwHAAAAAOjEarVqPUIZTq4DAAAs2DiOSeJukiLkWIMc65AlPbJzfT7KdQAAgAW7uyza+rSH7W51mhwfNjnWIcs6PCChJY8pAAAAAAA64ULT+SjXAQAAAAA68dDK9RdeeKH1CE+lXAcAAAAA6MTV1cPaFP6pT32q9QhPpVwHAAAAAOjE8XhsPUIZynUAAAAAgE6sVqvWI5ShXAcAAAAA6IST6/NRrgMAAAAAdGK9VgnPxTsJAAAAANCJ8/nceoQylOsAAAAAAJ1wcn0+3kkAAAAAgE4o1+fjnQQAAAAAgAsp1wEAAAAAOnF9fd16hDKU6wAAAAAAcCHlOgAAAABAJzabTesRyrhqPQAAAABPN45jkmS/37cdhFnIsQY51iFLenQ6nVqPUIZyHQAAYMGmaUqSbLfbxpNwH7vdLokcHzo51iHLOjwgoSVrYQAAAAAAOrFeq4Tn4p0EAAAAAIALKdcBAAAAADpxPB5bj1CGnesAAAAL5kLTWuRYgxzrkCU92mw2rUcow8l1AAAAAIBOnM/n1iOU4eQ6AADAgk3TlCTZbreNJ+E+drtdEjk+dHKsQ5Z1+PTB5ZTr83FyHQAAAAAALqRcBwAAAACACynXAQAAAAA6sV6rhOfinQQAAAAA6MTpdGo9QhnKdQAAAAAAuJByHQAAAACgE06uz0e5DgAAAADQCTvX5+OdBAAAAADoxGazaT1CGcp1AAAAAAC40FXrAQAAAHi6cRyTJPv9vu0gzEKONcixDlnSo+vr69YjlKFcB2ARxnH0B9sCpmlqPQJAOXf/bN1ut40n4T52u10SOT50cqxDlnX4e+Tlrq5UwnPxTgKwCNM0+YNtAXd/SQEAAIDq7FwHAAAAAOjE8XhsPcJFPu/zPq/1CE+lXAcAAAAA6MR6/bAq4V/8xV9sPcJTPax3EgAAAACA1+2hnVxfMuU6AAAAAEAnVqtV6xHKcKEpAIswjqNb3guYpqn1CAAAADyDcn0+ynUAFmGapmy329ZjcE+73a71CAAAADyDcn0+1sIAAAAAAHTidDq1HqEM5ToAAAAAQCecXJ+Pch0AAAAAoBPn87n1CGUo1wEAAAAAOrFeq4Tn4p0EAAAAAOjEo0ePWo9QhnIdAAAAAKATV1dXrUcoQ7kOAAAAANAJa2Hm450EAAAAAOjE6XRqPUIZynUAAAAAgE44uT4f7yQAAAAAQCeOx2PrEcpQrgMAAAAAwIVcDQvAIozjmP1+33oM7mmaptYjAAAA8Aybzab1CGUo1wFYhGmast1uW4/BPe12u9YjAAAAwBvCWhgAAAAAgE6sVqvWI5ShXAcAAAAA6IQLTedjLQwAAMCCjeOYJO4mKUKONcixDlnSo/P53HqEMpTrACyCC01rcKEpwPzu/tnqbpKH7e5eEjk+bHKsQ5Z1+HskLSnXAVgEF5rW4EJTAAAAemHnOgAAAAAAXEi5DgAAAAAAF7IWBoBFsHO9BjvXAQAAlm2z2bQeoQzlOgCLYOd6DXauAwAA0AtrYQAAAAAAOnE6nVqPUIaT6wAAAAs2jmOSWJ9WhBxrkGMdsqRH67Xz1nNRrgMAACzY3X0W1qc9bHer0+T4sMmxDlnW4QHJ5a6vr1uPUIbHFAAAAAAAnXByfT5OrgMAACyYtTC1yLEGOdYhS3q0Wq1aj1CGch0AAGDBrIWpwQqKGuRYhyzr8IDkci40nY/PAAAAAAAAdOJ8PrceoQwn1wFYhHEcnTgo4O50JQAAAMt0daUSnot3EoBFmKbJRzILuPt4LQAAAMvk5Pp8rIUBAAAAAOjE9fV16xHKUK4DAAAAAHRis9m0HqEM5ToAAAAAQCdWq1XrEcpQrgMAAAAAdMLO9fko1wEAAAAAOrFeq4TnctV6AABIknEcs9/vW4/BPU3T1HoEAAAAnuF0OrUeoQzlOgCLME1Ttttt6zG4p91u13oEAAAAnsFamPn4DAAAAAAAQCeshZmPdxIAAAAAoBPWwsxHuQ4AAAAA0Inj8dh6hDKU6wAAAAAAndhsNq1HKEO5DgAAAAAAF1KuAwAAAAB04nw+tx6hjKvWAwBAkozjmP1+33oM7mmaptYjAAAA8AzrtfPWc1GuA7AI0zRlu922HoN72u12rUcAAADgGexcn4/HFAAAAAAAnTidTq1HKEO5DgAAAADQCeX6fJTrAAAAAACdcKHpfOxcBwAAWLBxHJPExd9FyLEGOdYhS3q0Wq1aj1CGk+sAAAAAAJ2wFmY+Tq4DAAAs2DRNSZLtdtt4Eu5jt9slkeNDJ8c6ZFmHTx9cbrPZtB6hDCfXAQAAAAA6Yef6fJxcfx08Eavh7gQQAAAAAPRCuT4f5frr8M53vrP1CMzg7iNgAAAAANCLqyuV8FyshQEAAAAA6IQLTeejXAcAAAAA6IS1MPNRrgMAAAAAwIWU6wAAAAAAnVitVq1HKEO5DgAAAAAAF1KuAwAAAADAha5aDwAAAMDTjeOYJNnv920HYRZyrEGOdciSHrnQdD7KdQAAgAWbpilJst1uG0/Cfex2uyRyfOjkWIcs6/CAhJaU6wAAAAvm5HotcqxBjnXIkh6t1zaFz0W5DgAAsGBOrtfglGwNcqxDlnV4QHK50+nUeoQyPKYAAAAAAOjE9fV16xHKUK4DAAAAAHTi6soyk7ko1wEAAAAA4ELKdQAAAACATpzP59YjlKFcBwAAAADohLUw81GuAwAAAAB04ng8th6hDI8pAFiEcRyz3+9bj8E9TdPUegQAAACeYbPZtB6hDOU6AIswTVO2223rMbin3W7XegQAAACe4XQ6tR6hDGthAAAAAAA6sV6rhOfinQQAAAAAgAtZCwMAALBg4zgmibtJipBjDXKsQ5b0yFqY+SjXAVgEF5rW4EJTgPnd/bPV3SQP2929JHJ82ORYhyzr8PfIyynX56NcB2ARXGhagwtNAebn5HotcqxBjnXIErgP5ToAAMCCObleg1OyNcixDlnW4QEJLbnQFAAAAAAALqRcBwAAAADoxGq1aj1CGdbCvA4+blKDS/cAAHgI7FyvRY41yLEOWQL3oVx/Hd75zne2HoEZuHQPAICHwM71Gux3rkGOdciyDg9IaEm5DgAAsGBOrtcixxrkWIcs6dH5fG49QhnKdQAAgAVzcr0Gp2RrkGMdsqzDA5LLXV2phOfiQlMAAAAAgE4cj8fWI5ShXAcAAAAA6MRms2k9Qhk+AwAAALBgdq7XIsca5FiHLOmRk+vzUa4DAAAsmJ3rNdjvXIMc65BlHR6QXG69tsxkLsp1ABZhHEd/KCrgrgACAACA6pTrACzCNE1OjRRwdwIIAACAZVqtVq1HKMNnAAAAAAAAOmHn+nyU6wAAAAAAndhsNq1HKEO5DgAAAADQiUePHrUeoQzlOgAAAABAJ9ZrlfBcvJMAAAAAAJ1woel8lOsAAAAAAJ04n8+tRyhDuQ4AAAAAABdSrgMAAAAAdMLJ9fko1wEAAAAAOnF1ddV6hDK8kwAAAAs2jmOSZL/ftx2EWcixBjnWIUt6dDweW49QhnIdAABgwaZpSpJst9vGk3Afu90uiRwfOjnWIcs6PCC53Gq1aj1CGdbCAAAAAAB0ws71+SjXAQAAAADgQsp1AAAAAIBObDab1iOUoVwHAAAAAOiEC03no1wHAAAAAIALKdcBAAAAAOBCynUAAAAAgE7YuT4f5ToAAAAAAFxIuQ4AAAAA0Inr6+vWI5ShXAcAAAAA6MR6rRKei3cSAAAAAKATp9Op9QhlXLUeAACSZBzH7Pf71mNwT9M0tR4BAAAA3hDKdQAWYZqmbLfb1mNwT7vdrvUIAAAAPIO1MPPxTgIAAAAAdOJ8PrceoQzlOgAAAABAJ1arVesRyrAWBgAAYMHGcUwSd5MUIcca5FiHLIH7UK4DAAAs2N1l0e4medju7iWR48MmxzpkWYcHJJezFmY+1sIAAAAAAHTChabzcXIdgEUYx9GJgwLuTlcCAACwTI8ePWo9QhnKdQAWYZomH8ks4O7jtQAAACzT6XRqPUIZPgMAAAAAANCJ5557rvUIZTi5DsAiWAtTg7UwAPMbxzGJC9uqkGMNcqxDlvTIyfX5KNcBWARrYWqwFgYAAGDZnFyfj3IdAABgwe4+FeQh9MN29wBajg+bHOuQZR0+fXC54/HYeoQy7FwHAAAAAOjEeq0Snot3EgAAAACgE3auz0e5DgAAAADQCSfX5+OdBAAAAADohJ3r81GuAwAAAAB0YrVatR6hDOU6AAAAAABc6Kr1AACQJOM4Zr/ftx6De5qmqfUIAOWM45gk/j1ZhBxrkGMdsgTuQ7kOwCJM05Ttdtt6DO5pt9u1HgGgnLsHl/49+bDd/TtSjg+bHOuQZR0ekFzOhabzUa4DAAAsmJPrtcixBjnWIUt65ELT+SjXAVgEa2FqsBYGYH5OrtfglGwNcqxDlnX4eyQtKdcBWARrYWqwFgYAAGDZVqtV6xHKUK4DsAhOrtfg5DoAAAC9sL0eAAAAAAAu5OQ6AItgLUwN1sIAAAAs22azaT1CGU6uAwAAAAB04nw+tx6hDOU6AAAAAEAnTqdT6xHKUK4DAAAAAHTCyfX52LkOwCKM45j9ft96DO5pmqbWIwAAAPAMq9Wq9QhlKNcBWAQXmtbgQlMAAIBlc6HpfKyFAQAAAADohJ3r81GuAwAAAAB0ws71+SjXAQAAAADgQsp1AAAAAIBOuNB0Psp1AAAAAIBO2Lk+H+U6AAAAAEAn1muV8Fy8kwAAAAAAnbAWZj5XrQcAAADg6cZxTJLs9/u2gzALOdYgxzpkSY+shZmPch0AAGDBpmlKkmy328aTcB+73S6JHB86OdYhyzo8ILmck+vzsRYGAAAAAAAupFwHAAAAAOiEtTDzUa4DAAAAAHRivVYJz8U7CQAAAADQCTvX56NcBwAAAADohHJ9Psp1AAAAAIBOHI/H1iOUoVwHAAAAAOjEZrNpPUIZynUAAAAAgE6cz+fWI5Rx1XoAAAAAnm4cxyTJfr9vOwizkGMNcqxDlvTo0aNHrUcoQ7kOAACwYNM0JUm2223jSbiP3W6XRI4PnRzrkGUdHpBc7upKJTwXa2EAAAAAADqxXquE5+KdBAAAAACACynXAQAAAAA6cTweW49QhnIdAAAAAAAupFwHAAAAAIALKdcBAAAAAOBCynUAAAAAALiQch0AAAAAoBObzab1CGUo1wEAAAAAOnE+n1uPUIZyHQAAAACgE6vVqvUIZSjXAQAAAAA6cTweW49QhnIdAAAAAKATTq7P56r1AAAAADzdOI5Jkv1+33YQZiHHGuRYhyzp0XrtvPVcvJMAAAAAAJ04nU6tRyjDyXUAAIAFm6YpSbLdbhtPwn3sdrskcnzo5FiHLOvw6YPL2bk+H+U6AIswjqM/FBVwVwABAACwTFdXKuG5eCcBWIRpmpwaKeDuBBAAAADLZC3MfOxcBwAAAADoxGq1aj1CGcp1AAAAAAC4kHIdAAAAAAAupFwHAAAAAOjE+XxuPUIZynUAAAAAgE4o1+dz1XoAAEiScRyz3+9bj8E9TdPUegQAAACewYWm81GuA7AI0zRlu922HoN72u12rUcAAADgGa6uVMJzsRYGAAAAAKAT19fXrUcoQ7kOAAAAANCJ9VolPBfvJAAAAABAJ+xcn48FOwAsggtNa3ChKQAAwLKdTqfWI5ShXAdgEVxoWoMLTQEAAOiFch0AAGDBxnFMEp/wKkKONcixDlnSo/P53HqEMuxcBwAAAADohJ3r83FyHQAAYMHu7rOwPu1hu1udJseHTY51yLIOnz6gJSfXAQAAAAA64eT6fJxcBwAAWDA712uRYw1yrEOW9Oh0OrUeoQzlOgAAwIJZC1ODFRQ1yLEOWdbhAQktWQsDAAAAANAJJ9fno1wHAAAAAOjEeq0Snot3EgAAAAAALqRcBwAAAADoxGq1aj1CGcp1AAAAAIBOnM/n1iOUoVwHAAAAAIALKdcBAAAAADrhQtP5eCcBAAAAADphLcx8lOsAAAAAAJ1Qrs9HuQ4AAAAAABdSrgMAAAAAdOJ4PLYeoYyr1gMAAADwdOM4Jkn2+33bQZiFHGuQYx2ypEer1ar1CGUo1wEAABZsmqYkyXa7bTwJ97Hb7ZLI8aGTYx2yrMMDkssp1+djLQwAAAAAQCc2m03rEcpQrgMAAAAAdOJ8PrceoQzlOgAAAABAJ06nU+sRylCuAwAAAAB0ws71+SjXAQAAAAA6Yef6fJTrAAAAAACduL6+bj1CGcp1AAAAAIBOWAszH+U6AAAAAABc6Kr1AACQJOM4Zr/ftx6De5qmqfUIAAAA8IZQrgOwCNM0Zbvdth6De9rtdq1HAAAAgDeEtTAAAAAAAHAh5ToAAAAAQCdcaDof5ToAAAAAQCfO53PrEcpQrgMAAAAAwIWU6wAAAAAAnVivVcJz8U4CAAAAAMCFlOsAAAAAAHChq9YDAECSjOOY/X7fegzuaZqm1iMAAADwDKvVqvUIZSjXAViEaZqy3W5bj8E97Xa71iMAAADwDMfjsfUIZSjXAQAAAADeYMMw/MEkH3jVt7/ncDh8Tn9fJ9fno1wHYBGshanBWhgAAIDX7O1JPpjkP3jse68k+bnP5W+6XruGcy7KdQAWwVqYGqyFAQAAeM3enuRHDofDT76Rv6m1MPPxmAIAAAAA4I339iSf2x0wT6Bcn4+T6wAAAAs2jmOSWJ9WhBxrkGMdsqSVYRieT/Jrk/zuYRi+JTeHoP9akvfeZ+f68XjMyy+/nI997GN561vfmhdffDGbzeazXvP888/fY3Iep1wHAAAAAHhjvTU33ew/TvK1uSnavyPJm1/vDzwej3nPe96Tj370o/nkJz+Z559/Pl/8xV+cr/u6r/tMwb5arXI6nWYYn0S5DgAAsGh3l0W7m+Rhu7uXRI4PmxzrkGUdD/XTB4fD4ceGYfiCw+HwM7ff+vvDMKyS/A/X19e5urq8tn355Zc/U6wnyac//el84hOfyHvf+975BuezKNcBAAAAAN5gjxXrdz6a5Lmf/dmfzRd90Rdd/PM+9rGP5ZVXXvkl33/LW96St7zlLTmfzzmfz3n06FE+/OEPv76h+SzKdQAAgAWzc70WOdYgxzpkSSvDMPy+JN+Z5C2Hw+HTt9/+iiQ//4Vf+IX/3Ov5mW9961vzpje96TMn15PkTW96U77hG74h73jHOz7zvePxmHe/+92vf3g+Q7kOAACwYNbC1GAFRQ1yrEOWdTzgByR/K8kqyZ8fhuFP5mYH+7cl+bbVavWfv54f+OKLL+Ztb3tbPvKRj+RTn/pUXnjhhbz97W/Piy+++FmvO5/P952dW8p1AAAAAIA30OFw+JlhGH57km9P8sNJfiHJn0vyp5K8rnJ9s9nk/e9/f15++eV8/OMfz5d92ZflxRdf/MxlpndcaDof5ToAAAAAwBvscDj8vSTvmvNnbjabvOMd7/isNTCvtlqt5vwtu7ZuPQAAAAAAAG8M5fp8lOsAAAAAAJ1Qrs9HuQ4AAAAAABdSrgMAAAAAdMKFpvNRrgMAAAAAdOJ4PLYeoYyr1gMAQJKM45j9ft96DO5pmqbWIwCUM45jkvj3ZBFyrEGOdciSHj333HOtRyhDuQ7AIkzTlO1223oM7mm327UeAaCcuweX/j35sN39O1KOD5sc65BlHR6Q0JK1MAAAAAAAnbAWZj7KdQAAAAAAuJByHQAAAAAALqRcI6FaSQAAC7VJREFUBwAAAADoxPl8bj1CGS40BQAAWLBxHJO4sK0KOdYgxzpkCdyHch0AAGDBpmlKkmy328aTcB+73S6JHB86OdYhyzo8ILncem2ZyVyU6wAAAAvm5HotcqxBjnXIkh5ZCzMf5ToAAMCCObleg1OyNcixDlnW4QHJ5Zxcn493EgAAAACgE06uz0e5DgAAAADQic1m03qEMpTrAAAAAACduL6+bj1CGXauA7AI4zjalVfA3V5gAObjQtNa5FiDHOuQJXAfynUAFmGaJpcJFXB3MRQA83GhaQ0uT6xBjnXIsg4PSC7nQtP5eCcBAAAAADrhQtP5KNcBAAAAADqhXJ+Pch0AAAAAoBPWwszHOwkAAAAA0InT6dR6hDJcaAoAALBg4zgmcWFbFXKsQY51yBK4D+U6AADAgk3TlCTZbreNJ+E+drtdEjk+dHKsQ5Z1eEByudVq1XqEMqyFAQAAAACACzm5DgAAsGDWwtQixxrkWIcs6dH5fG49QhlOrgMAAAAAwIWcXAcAAFgwO9drsN+5BjnWIcs6fPqAlpxcBwAAAACACynXAQAAAAA6sVqtWo9QhnIdAAAAAKATLjSdj53rAAAACzaOYxI7ZauQYw1yrEOW9Gi9dt56Lsp1AACABXOhaQ0uT6xBjnXIsg4PSC7n5Pp8lOsAAAAL5uR6LXKsQY51yJIeXV9ftx6hDJ8BAAAAAADoxHPPPdd6hDKU6wAAAAAAcCFrYQAAABbMzvUa7HeuQY51yLIOq30uZy3MfJxcBwAAAADoxHqtEp6LdxIAAAAAoBPn87n1CGUo1wEAAAAAOnF1ZVP4XJTrAAAAAACdOJ1OrUcoQ7kOAAAAANAJa2Hmo1wHAAAAAIALKdcBAAAAADqxWq1aj1CGch0AAAAAoBPWwsxHuQ4AAAAA0AkXms5HuQ4AAAAA0In1WiU8F+8kAAAAAEAnnFyfj3IdAAAAAKATTq7P56r1AAAAADzdOI5Jkv1+33YQZiHHGuRYhyzpkXJ9Psp1ABZhHEd/sC1gmqbWIwCUc/fP1u1223gS7mO32yWR40MnxzpkWYe/R9KSch2ARZimyR9sC7j7SwoAAADLtFqtWo9Qxmsu14dh+FVJ3pvkdyX51Ul+PMlfTPJnDofD9VP+N78iyUeSfPBwODyxMRmG4cXbn/tVSVZJfiTJtx4Oh+97wmv//ST/YZIhyStJfjDJHz8cDn//Ca99R5L3JfmNSc5JfiDJNx4Oh0884bVvT/Inb2d4IcmHknzT4XD44ae9HwAAAAAA9Os1LdgZhuHNSf52bortH0vyXyb5hSTvT/LBYRh+yeOOYRiuknwgyZc84+f+ztuf+1VJvjvJX03y5Un+12EYfs+rXvutSf5Cks9P8l1J/uckvz3J3xmG4Te+6rVfnWR/+7P+UpK/keTfTPLyMAxf+qrXvi3JDyV5V5L/KclfSfKOJD80DMNvetb7AgAAAADwkByPx9YjNDUMwwvDMPzoMAzvfsKvfcEwDD/96g75aV7ryfX/JMmvS/LHDofDn33sN/urSf5Akn8jyfc+9v3Pz01Z/jXP+D/x5iT/XZKfSfKbD4fDP7j9/rfl5vT6n0nyPbff++eTfGOSn0jyFYfD4edvv/9XkvxvSf50bsrx3Bb9fz7JP0nylYfD4f+7/f4Hknz/7Wu/9rFRviPJ5yX5TYfDYbp97Xcm+b+S/NdJFOwAAAAAQAk9l+vDMLwpNwe8f/0Tfu3zk/zNJF/wWn/ea70a9kuT/L+5KZsf9923X9/x2BB/IMlHc1Osf/8zfubXJvkXknzzXbGeJIfD4ceT/Ge5Ob3+5ttvf0VuHgR88K5Yv33t9yX5h0n+1cd+7rtzszbmv70r1m9f+wO38/zbtytuMgzDW2/n/J67Yv32tT+amxPsXzkMw/iM/w8AAAAAAA/Gev1aK+FableD/59Jfu0Tfu2rk3w4yT99yc98TSfXD4fDH3zKL/26268/9dj3vj7JJ3OzhuUX8/TT678zN7vQP/iE3++/eNW3fub267/0+DeHYfinkvzKJD/92Le/+vbrS0/4PV9K8tuS/ObcnIr/5V77R5L8liTTE34dAAAAAOBBeYgXmn7oQx/Kiy++mM1mc58f868l+b7cHOz+x6/6ta9J8p256Yz/79f6A1/zhaZ3bteufGFuTp7/iST/KDenvO98S5K/czgcXhmG4Z3P+FFfnuQnk1wPw/Bnb3/er0zyw7m5pPTxwvvv3v7n9w7D8MeS/OUk/0ySb7/9+t7HXnv35OEf5Jf6iduv//LreC0AAAAAAG+w973vfXnb296W97///a+7YD8cDt9199+HYXj1r33z7fe/7JKf+Xo+A/AtuTmp/l/l5lLT33Y4HH7usUH+j8Ph8Mpr+Dm/OsmjJD+Y5Pfm5gT7X8vNCpjvG4bhdz/2M89JfkdunhzskvxcbtbB/L4kf/RwOOwe+7m/6vbrz+eX+oXbr//s63gtAAAAAABvsE9+8pP5yEc+kpdffrn1KJ/l4pPruSm1/3RuTn3/niQ/OAzD7zgcDj984c/5Fbkprn80yW84HA4/myTDMHxHkg8l+QvDMHzp4XD41O3r/2huLk79aG6O739+bsr1PzEMw+F2/3qSPHf79e5/97i7773pdbz2M/b7/cP77ARPtd/vW4/ADORYgxzrkGUNcqxBjnXIsgY51iDHOmTJQn1Ou8fnn38+L730pC3Zy/Gud73rm3OzvuUzh8NfeeWV0zd90ze996WXXvrWZoO9ysXl+uFw+G/u/vswDL8rNzeo/vfDMPwrtyfMX6vT7dc/fles3/78Dw/D8IEkfzg3O9G/fxiGP5TkP03yN5L8/sPh8Onb3/9bcrOE/q8Pw/BrDofDT+dm33uSPP+E3/OF2693O3UueS0AAAAAAJ9jL7300vuSvK/1HL+ce10NezgcvjfJDyT59XnCLau/jLu1Kx9+wq/dXSB69zP/8O3X/+iuWL/9/T+e5P25OQX/795++25FzZPWudx97xdex2sBAAAAACDJazi5PgzDVZJ3JlkdDofvf8JL/uHt1y9I8vELfu+P5eZi1CedGr9b1/JPbr++JcmnDofDjz/htT92+/VLbr/+P7dff81j/z2PfS9JDk947au9+rUAAAAAAJDktZ9c/5tJPjAMw5OuYv0NSc5JnlR8P8sP3n7915/wa195+/VHbr/+VJIXhmH4kie89q23X3/y9uvfvv36W57w2nfmZh3Ny6/xtcnN/ncAAAAAAPiMX7ZcPxwO10n+em5Omf/Hj//aMAzfkJsi/HsPh8NPXfh7/8Ukj5J88zAM/+JjP/Orknxtkr93OBzu1sP8j7dfv+32JP3da784yXuSfPp2xiT5W0n+UZKvH4bhSx977W9N8jVJPni7mz2Hw+ETSX4oyb8zDMNXPvbaL0/yh5L83ddxUSsAAAAAAAt1OBxWh8Phf3/C9z9++2s/8Vp+zmu90PQ9ublc9E8Nw/Cu3Jwo/4okvzU3J9a//jX+nMcHPQzD8I1Jvj3JjwzD8N1J3pzk9+fmotE/8tjLvyvJv5Xk30vy5cMw/C//f3t3qKJFFIZx/L/d4lVMMm5RtCpossk2Ma2IRu9Co2D1BgwGmxhlN+tbBU3egSyfYb4gX9qxiPL7pcMwnDOnPgPPW12t7rd2o5/OzNf9vhfLspxWb6uz/XDUK9VJ9aODHwTV0+pj9WFZljfVRWuwflQ93novAAAAAAD+f5eqhZmZb9Vx9bq6Vj1rrWN5WR3PzPc/OXxmXrSG5l+qh/v1++r6zJz99t7P6m71fP/oSesA0/Pq9sy8Otj3XXWn+lw9qu61VtvcOOxtn5nz6mZrRcxJ9aC1CubWzHwKAAAAAAAOHO12u7/9DQAAAAAA8E+57EBTAAAAAABgT7gOAAAAAAAbCdcBAAAAAGAj4ToAAAAAAGwkXAcAAAAAgI2E6wAAAAAAsJFwHQAAAAAANhKuAwAAAADARsJ1AAAAAADYSLgOAAAAAAAb/QISIw+gXAwncwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1060bf240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "msno.matrix(joined_test_users)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above diagram, the horizontal lines indicate missing values. There are quite a few missing values, mostly in the user table. It also looks like there are a few missing timestamps - we can just remove these as they represent a small fraction of the total data. Operative system is also missing quite a few values (5% of data in fact) - we don't want to throw this away, and will deal with it in the steps below. \n",
    "\n",
    "Drop the missing timestamps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined_test_users = joined_test_users.dropna(subset=['timestamp'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data cleaning and converting categorical variables\n",
    "\n",
    "Now we need to make sure the joined set is clean. A few quick diagnostics (not shown here) reveal that there are several issues to be fixed before proceeding:\n",
    "* The timestamp column is generally in accordance with datetime format, but some years are labeled as 2815 instead of 2015. Additionally, some of the times go to 60 mins/60 seconds - I changed these to 59 instead. \n",
    "* In the source column there isn't standardization between dash and underscore instances: change them all to underscores for consistency. \n",
    "* There are some typos in the operative_system column - I can just fix these via replacement. \n",
    "* Linux and Other are missing from operative_system, and there are many missing values as noted above - replace the missing values with \"Other\" as a catch-all category (some of these in Other may actually be windows/mac, etc. but set them to Other for now). \n",
    "* Some price information appears to be incorrect - values way higher than 59 dollars. As I don't know what the price should have been, I will exclude these rows. \n",
    "\n",
    "For now, because of all the missing data, we'll exclude the customer demographics information from the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/site-packages/pandas/core/internals.py:3462: FutureWarning: Passing in 'datetime64' dtype with no frequency is deprecated and will raise in a future version. Please pass in 'datetime64[ns]' instead.\n",
      "  return self.apply('astype', dtype=dtype, **kwargs)\n"
     ]
    }
   ],
   "source": [
    "# fix timestamp issue\n",
    "joined_test_users['timestamp'] = joined_test_users['timestamp'].str.replace('2815', '2015')\n",
    "joined_test_users['timestamp'] = joined_test_users['timestamp'].str.replace('60', '59')\n",
    "joined_test_users['timestamp'] = joined_test_users['timestamp'].astype('datetime64')\n",
    "\n",
    "# fix - and _ issue\n",
    "joined_test_users['source'] = joined_test_users['source'].str.replace('-', '_')\n",
    "\n",
    "# replace typos in operative_system column\n",
    "joined_test_users['operative_system'] = joined_test_users['operative_system'].str.replace('i0S', 'iOS')\n",
    "joined_test_users['operative_system'] = joined_test_users['operative_system'].str.replace('wnddows', 'windows')\n",
    "joined_test_users['operative_system'] = joined_test_users['operative_system'].str.replace('anroid', 'android')\n",
    "joined_test_users['operative_system'] = joined_test_users['operative_system'].str.replace('apple', 'mac')\n",
    "joined_test_users['operative_system'] = joined_test_users['operative_system'].str.replace('apple', 'mac')\n",
    "joined_test_users['operative_system'] = joined_test_users['operative_system'].fillna('Other')\n",
    "\n",
    "# select only 39/59 prices\n",
    "joined_test_users = joined_test_users[joined_test_users['price'] <= 60.0]  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now with cleaned data, it will be helpful to map the categorical variables (source, operating system, and device) to numerical values. This can be done by assigning a numerical value to each category in the aforementioned columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>source</th>\n",
       "      <th>device</th>\n",
       "      <th>operative_system</th>\n",
       "      <th>test</th>\n",
       "      <th>price</th>\n",
       "      <th>converted</th>\n",
       "      <th>city</th>\n",
       "      <th>country</th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>604839</th>\n",
       "      <td>2015-05-08 03:38:34</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Buffalo</td>\n",
       "      <td>USA</td>\n",
       "      <td>42.89</td>\n",
       "      <td>-78.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>624057</th>\n",
       "      <td>2015-05-10 21:08:46</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Lakeville</td>\n",
       "      <td>USA</td>\n",
       "      <td>44.68</td>\n",
       "      <td>-93.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317970</th>\n",
       "      <td>2015-04-04 15:01:23</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Parma</td>\n",
       "      <td>USA</td>\n",
       "      <td>41.38</td>\n",
       "      <td>-81.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685636</th>\n",
       "      <td>2015-05-07 07:26:01</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>59.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Fayetteville</td>\n",
       "      <td>USA</td>\n",
       "      <td>35.07</td>\n",
       "      <td>-78.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>820854</th>\n",
       "      <td>2015-05-24 11:04:40</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>Fishers</td>\n",
       "      <td>USA</td>\n",
       "      <td>39.95</td>\n",
       "      <td>-86.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  timestamp  source  device  operative_system  test  price  \\\n",
       "user_id                                                                      \n",
       "604839  2015-05-08 03:38:34     1.0       0                 2     0   39.0   \n",
       "624057  2015-05-10 21:08:46     5.0       0                 3     0   39.0   \n",
       "317970  2015-04-04 15:01:23     2.0       0                 3     0   39.0   \n",
       "685636  2015-05-07 07:26:01    11.0       0                 2     1   59.0   \n",
       "820854  2015-05-24 11:04:40     1.0       1                 0     0   39.0   \n",
       "\n",
       "         converted          city country    lat   long  \n",
       "user_id                                                 \n",
       "604839           0       Buffalo     USA  42.89 -78.86  \n",
       "624057           0     Lakeville     USA  44.68 -93.24  \n",
       "317970           0         Parma     USA  41.38 -81.73  \n",
       "685636           0  Fayetteville     USA  35.07 -78.90  \n",
       "820854           0       Fishers     USA  39.95 -86.02  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# encode everything\n",
    "opsys_cats = {'mac': 0, 'windows': 1, 'iOS': 2, 'android': 3, 'Other': 4}\n",
    "source_cats = {'ads_google': 0, 'ads_facebook': 1, 'ads_bing': 2, 'ads_yahoo': 3, 'ads_other': 4, 'seo_google': 5,\n",
    "              'seo_facebook': 6, 'seo_bing': 7, 'seo_yahoo': 8, 'seo_other': 9, 'friend_referral': 10,\n",
    "               'direct_traffic': 11}\n",
    "device_cats = {'mobile': 0, 'web': 1}\n",
    "\n",
    "joined_test_users['operative_system'].replace(opsys_cats, inplace=True)\n",
    "joined_test_users['source'].replace(source_cats, inplace=True)\n",
    "joined_test_users['device'].replace(device_cats, inplace=True)\n",
    "\n",
    "joined_test_users.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining the control/test sets and ensuring they are not biased\n",
    "\n",
    "With our data cleaned and categorical variables mapped to numerical values, we can now split into the A/B groups - those that saw the 39 dollar price (control) and those that saw the 59 dollar price (test)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "control_group = joined_test_users[joined_test_users['price'] == 39.0]\n",
    "test_group = joined_test_users[joined_test_users['price'] == 59.0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we dive into the analysis, it's critical to make sure that the testing and training samples are similar to one another in order to control for variables like time distributions, device choice, etc. (even with large samples like this). Key questions we need to answer when comparing the control/test groups:\n",
    "\n",
    "* Do time distributions look the same?\n",
    "* Are operating systems the same?\n",
    "* Are devices the same?\n",
    "* Are sources the same?\n",
    "\n",
    "This is very important to confirm since one could imagine a scenario in which the test set is dominated by mobile users who are less likely to buy the software. \n",
    "\n",
    "The statistical test for this, since we are testing two categorical datasets, is Chi-squared, but it can be informative to just give the distributions a quick look to see if they are the same. Later on, if the significance of our eventual metrics is near our adopted threshold, we may want to be more rigorous with testing these distributions.\n",
    "\n",
    "As an example of this quick analysis, we can plot the time distribution of the test and control groups to reveal that the distributions are very similar, so we don't need to immediately test for significance. There is a small increase in users in May but this is reflected in both test and control samples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl8AAAD8CAYAAABJqMF0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGaBJREFUeJzt3XmYHXWd7/F3LyGQdEJQOmyJLC7fZxCMIuQmioAwhEWUOINXEUWCIIxgxACizgUBWYUQdlkEZEZEYJBFL4u7DgFEBkRE+A4Rg0QWA0xCQjBLp+8f5zT3ELqTQ6f7152T9+t58vSp+tWp+hZ0PfXpX1X9qqmzsxNJkiSV0TzQBUiSJK1NDF+SJEkFGb4kSZIKMnxJkiQVZPiSJEkqyPAlSZJUUGupDUVEM3AxMA5YDBySmbNq2s8H3g8sqM7aNzPnl6pPkiSphGLhC5gMrJuZEyNiAjAd2LemfTtgj8x8vmBNkiRJRZW87LgjcAdAZt4LbN/VUO0VeztwWUTMjIiDC9YlSZJUTMmer5FA7WXEjohozcxlwHDgAuAcoAX4RUTcn5m/r13BK68s6WxtbSlWcF9paWmio8M3CUhvlMeO1DseO4PDkCEtTd3NLxm+XgJG1Ew3V4MXwCLgvMxcBBARP6dyb9hrwtfChYtL1NnnRo0axrx5iwa6DGmN47Ej9Y7HzuDQ3j6i2/klLzvOBPYGqN7z9XBN2zuAuyKiJSKGULlE+UDB2iRJkooo2fN1E7B7RNwNNAFTImIaMCszb42Ia4B7gaXAv2XmIwVrkyRJKqKps3PNuSY8d+6CNafYGnb/Sr3jsSP1jsfO4NDePqLbe74cZFWSJKkgw5ckSVJBhi9JkqSCDF+SJEkFGb4kSZIKMnxJkiQVZPiSJEkqyPAlSZJUkOGrAS1dupSbb76x19+/7bYf8tGP7t2HFUmSpC4lXy9U3A7Tf110e789eqei2+vJT396J1dffQWTJ//zQJciSZJWYM9XA1qTXhklSdLapqF7vtYETz/9V8499ywefPAB2tra+OhH9+PAAw/mb397jgsumMH9999Hc3MTu+02iSOOOIqhQ4dy220/5Ic/vJnx4ydw443X0dLSwq67TuILX/gSv/vdA5x22kkA7Ljj9txww62ceuqJbLXVW/nNb+5h8eLFfOc717JkyeIe1y9JkvqP4WsALVmyhGnTjmTLLd/KpZdeydy5c/n6179Ke/torr76SsaMGcMFF1zKSy/N54wzvkFnJxx99HEAPProI4wePZqLLvo2Dz/8EN/85qmMHz+B7bcfz9SpR3PNNVdz5ZXfZdSoDYDKfVzTp1/I0KHrMHz4cA4//OCVrl+SJPUPw9cAuv/++3j++bl8+9v/TltbG1tt9TamTTuOjo4O5s59jssuu4qRI9cHYNq04zjuuC9x2GFHANDR0cGxx/4rbW1tbL75FvzgB9fz2GN/ZOLE99PW1kZzczNvfvOGr25rwoT3MW7cuwG4665frXL9kiSpf3jP1wCaPfsJNttsLG1tba/OmzRpL1544XnGjBn7ajAC2Hbbd9HR0cGcOX8BYP31R73me8OGDWfZsmU9bmvjjTet2e6fV7l+SZLUPwxfA6i1dUi389dZ5/X3XXV0LH/NzyFDXv/dld1ov84667yh9UuSpP5h+BpAY8eO5emn5/DyywtfnXfFFZdyzTVXM2fOU7z00vxX5z/yyO9paWlhzJgxq1xvU1PTStu32GLL1Vq/JEnqPcPXABo/fiKjR2/EmWeeyuzZf+aee2Zyww3fZ+rUaYwduznf+MYJzJr1OA88cD/nnns2u+02ifXXH7XK9a633nosXLiQv/zlyW4vRW6//fjVWr8kSeq9hr7hfrAMetqTlpYWTj99OueccyYHH/wpNthgA6ZMOYTddpvE1ltvw4wZ3+Swww5ivfWGMWnSnhx22JF1rXe77XZg88234KCD9ufii694XXtzczOnn352r9cvSZJ6r2lNGpBz7twFa06xNUaNGsa8eYsGugxpjeOxI/WOx87g0N4+otv7gLzsKEmSVJDhS5IkqSDDlyRJUkGGL0mSpIIMX5IkSQUZviRJkgoyfEmSJBVk+JIkSSrI8CVJklSQ4WuAPf548tBDD67WOjo7O7n55v9g+fLlfVSVJEnqLw39bsf2i8YU3d7cI+a84e987WvHcuCBBzNu3Ht6vd3f/e4Bzj77DPbZZzLNzeZpSZIGM8/UA6wv3q25Jr2fU5KktZ3hawAdeeTnePbZZzjzzFM49dQTeeKJPzF16uHsuuv7+fjHJ3Pttd99NVi9/PJCTjjhq+y1165MmrQzX/vasbzwwvM888zTTJ16OAC77DKBBx64fyB3SZIkrYLhawCddtpZjB69EUceeRRf/OIxHHPMVN75zm25+uprOeqoY7nhhmu58cbrALj88kt45pmnufDCy7j00qv4n/95kfPPP4fRozfi1FO/CcBNN93GttuOG8hdkiRJq9DQ93wNdiNHrk9zczPDh7fxy1/+lBEjRnLYYUcAMHbsWzj00H/hqqsuZ7/9PsGzzz7NsGHD2GSTTRk2bBjHH38yCxYsoKWlhREjRgKwwQZvorXV/6WSJA1mnqkHidmzZzN79hPsvvsHXp23fPlyli5dytKlS/nUpw7imGO+yD777M52272XnXb6IHvu+aEBrFiSJPVGsfAVEc3AxcA4YDFwSGbO6maZ/wvckpmXlKptMOjo6ODd734vxx771de1tbS0sM027+LGG3/IzJn/yT33zOTCC8/lJz+5gwsuuHQAqpUkSb1V8p6vycC6mTkR+AowvZtlTgHeVLCmAdfU1ATAW96yOU899SQbb7wJY8aMZcyYsTz+eHLNNVfT3NzM9dd/jz/+8Q9MmrQXX//6KZx11nk8+OB/8eKLL7y6DkmSNPiVvOy4I3AHQGbeGxHb1zZGxH7AcuD2nlbQ1jaU1taWfi1ydYwaNazb+S0tzT22tbUN59ln53DAAQdw1VWXM2PGGRx88Gd57rnnmDHjLCZPnsyoUcOYP/8Fzj//Jk4++RQ23HBDfvWrn7DJJpuw+eab8sorLwHwzDOzecc7gqFDh/bTHkplrezYkdQzj53BrWT4GgnMr5nuiIjWzFwWEdsAnwT2A07oaQULFy5+Y1vsxaCnq2Xeom5njxo1jHk9tO27735cdNF5zJ79JGeffR7nn38OH/vYfowYMYI99/wQBx54KPPmLeLTnz6UF1+czxe+cCSLFi1i663fyRlnzGDBgsWMHj2G8eMncuCBn+bEE09j550/2J97KRWzsmNHUs88dgaH9vYR3c5vKjVAZ0ScA9ybmddXp+dk5pjq528COwOvAFsAS4CpmXlH7Trmzl2wRo4m6kEg9Y7HjtQ7HjuDQ3v7iG7vCyrZ8zUT+DBwfURMAB7uasjML3d9jogTgWdXDF6SJEmNoGT4ugnYPSLuBpqAKRExDZiVmbcWrEOSJGnAFLvs2Be87CitXTx2pN7x2Bkcerrs6OuFJEmSCjJ8SZIkFWT4kiRJKsjwJUmSVJDhS5IkqSDDlyRJUkGGL0mSpIIMX5IkSQUZviRJkgoyfEmSJBVk+JIkSSqo5Iu1JUlSg2u/aMxAl7BKc4+YM6DbN3xJkjSAdpj+64EuYZV+e/ROA11CQ/GyoyRJUkGGL0mSpIIMX5IkSQUZviRJkgoyfEmSJBVk+JIkSSrI8CVJklSQ4UuSJKkgw5ckSVJBhi9JkqSCDF+SJEkFGb4kSZIKMnxJkiQVZPiSJEkqyPAlSZJUUOtAFyCpMeww/dcDXcIq/fbonQa6BEkyfDW69ovGDHQJqzT3iDkDXYIkScUYvlbgX++SwD9cJPUf7/mSJEkqyPAlSZJUkOFLkiSpIMOXJElSQcVuuI+IZuBiYBywGDgkM2fVtB8BHAR0Aidn5o9K1SZJklRKyZ6vycC6mTkR+AowvashIjYEPg+8D9gN+FZENBWsTZIkqYiS4WtH4A6AzLwX2L6rITOfB8Zl5lJgY2BeZnYWrE2SJKmIkuN8jQTm10x3RERrZi4DyMxlEXEkcBJwfncraGsbSmtrS/9XOsiNGjVsoEvoU422Pxq8Gu13rdH2R4NXo/2uDfT+lAxfLwEjaqabu4JXl8y8MCIuA26PiA9m5i9q2xcuXFygzMFv3rxFdS/b3o919JU3sj/S6vDYkXrHY6d32ttHdDu/rvAVEVsCpwE7AEOA19yPlZlvqWM1M4EPA9dHxATg4Zr1B3A68M/AUio35C+vpzZJkqQ1Sb09X1dQuRfrPCo9WL1xE7B7RNxNJbxNiYhpwKzMvDUiHgLuofK04+2Z+atebkeSJGnQqjd87QDsnJkP9HZDmbkcOHyF2Y/VtJ9E5X4vSZKkhlXv045PULlhXpIkSauh3p6vs4HLI2IGlSC2pLYxM3/e14VJkiQ1onrD19XVnxd209YJOP6DJElSHeoKX5npOyAlSZL6QN3jfEXEcOBA4B+o9HQ9BlxbHZ1ekiRJdairRysi3gU8TuWdjJtV/30ZeDQitu6/8iRJkhpLvT1f5wN3Aod2jUofEa3A5VTG/tq9f8qTJElqLPXeyzUeOKP2dUDVz2cAE/ujMEmSpEZUb/h6GnhbN/PfwWtfli1JkqSVqPey46XAtyPi68B91Xn/CzgRuKQf6pIkSWpIb2SQ1eFUXq79puq8Z4CzgBn9UJckSVJDqnecr04qvVwnRsRo4O+Z2dsXbEuSJK21egxfEXEwcE1mLq5+XrH91c+ZeWX/lCdJktRYVtbzdTxwC7C4+rknnYDhS5IkqQ49hq/M3LK7zyuKiPa+LkqSJKlR1TvCfUd3ISsitgD+3NdFSZIkNaqV3fP1GeCz1ckm4NaIWLrCYptQGQNMkiRJdVjZPV//AWxOJXjtCNwFLKxp76xO39hv1UmSJDWYld3z9TJwMkBEzAa+n5mLu9ojYlRmzuvvAiVJkhpJva8XupPKZcdTauY9FhE/iogN+6EuSZKkhlRv+PpW9ecVNfM+AAwBLujTiiRJkhpYveFrV+ALmfnqk42Z+ThwFLBnfxQmSZLUiOoNXwuA7sb62gxY0nflSJIkNbZ6X6x9BXBFRBwPPFCd9x7gJOA7/VCXJElSQ6o3fJ1EpZfsDKBrsNW5wHnAmf1QlyRJUkOqK3xl5nIq73c8vvp045LMfKlfK5MkSWpA9fZ8ERHbA+8EWqrTTcBQ4D2ZeWj/lCdJktRY6gpfEXESlZ6vZ4GNgL9Wf7biCPeSJEl1q/dpx0OBwzNzU+ApYBcq4esnwOx+qUySJKkB1Ru+3gzcUf38IPC+6quF/hX43/1RmCRJUiOqN3zNAbaqfn4U2K76eQHg64UkSZLqVO8N95cB10XEFOBm4GcR8RywG/C7/ipOkiSp0dTV85WZZwLTgJcz8z4qrxXaD1gOfLb/ypMkSWos9T7t+G3gjMycBZCZV/Dal2xLkiSpDvXe8/VPwLL+LESSJGltUO89X+cAl0bEucCTwN9rGzPzib4uTJIkqRH1GL4iYg/gl5m5GDi5Onv3bhbtpDrq/cpERDNwMTAOWAwc0nUZs9r+JeAT1cnbMvOkuvZAkiRpDbKyy443AqOrn58EdgC27ObfVt1++/UmA+tm5kTgK8D0roaI2Ao4AHgfMBGYFBHvqn83JEmS1gwru+z4N+CyiLgfeAvwSeDlHpY9oY5t7Uh1oNbMvLf6rsguTwF7ZmYHQEQMYYVLmwBtbUNpbV1lJ1vDGzVq2ECX0KcabX80eDXa71qj7Y8Gr0b7XRvo/VlZ+PoocCLw/ur0BGBJN8t11rmtkcD8mumOiGjNzGWZuRR4vvqy7rOABzPzv1dcwcKFi+vcVGObN29R3cu292MdfeWN7I+0Ojx2pN7x2Omd9vYR3c7vMXxl5kNUAhgR8WfgI5n5wmrU8BJQW0VzZr76BGVErAtcSWXU/M+vxnYkSZIGrbqedszMLftgWzOBDwPXR8QE4OGuhmqP1y3Az6sDukqSJDWkeoea6As3AbtHxN1AEzAlIqYBs6g8LbkzMDQi9qou/9XMvKdgfZIkSf2uWPjKzOXA4SvMfqzm87qlapEkSRoo9Y5wL0mSpD5g+JIkSSrI8CVJklSQ4UuSJKkgw5ckSVJBhi9JkqSCDF+SJEkFGb4kSZIKMnxJkiQVZPiSJEkqyPAlSZJUkOFLkiSpIMOXJElSQYYvSZKkggxfkiRJBRm+JEmSCjJ8SZIkFWT4kiRJKsjwJUmSVJDhS5IkqSDDlyRJUkGGL0mSpIIMX5IkSQUZviRJkgoyfEmSJBVk+JIkSSrI8CVJklSQ4UuSJKkgw5ckSVJBhi9JkqSCDF+SJEkFGb4kSZIKMnxJkiQVZPiSJEkqyPAlSZJUUGupDUVEM3AxMA5YDBySmbNWWKYduBvYNjP/Xqo2SZKkUkr2fE0G1s3MicBXgOm1jRGxB/BjYKOCNUmSJBVVMnztCNwBkJn3Atuv0L4c+EfgxYI1SZIkFVXssiMwEphfM90REa2ZuQwgM38CEBE9rqCtbSitrS39WuSaYNSoYQNdQp9qtP3R4NVov2uNtj8avBrtd22g96dk+HoJGFEz3dwVvOq1cOHivq1oDTVv3qK6l23vxzr6yhvZH2l1eOxIveOx0zvt7SO6nV/ysuNMYG+AiJgAPFxw25IkSYNCyZ6vm4DdI+JuoAmYEhHTgFmZeWvBOiRJkgZMsfCVmcuBw1eY/Vg3y21RpCBJkqQB4CCrkiRJBRm+JEmSCjJ8SZIkFWT4kiRJKsjwJUmSVJDhS5IkqSDDlyRJUkGGL0mSpIIMX5IkSQUZviRJkgoyfEmSJBVk+JIkSSrI8CVJklSQ4UuSJKkgw5ckSVJBhi9JkqSCDF+SJEkFGb4kSZIKMnxJkiQVZPiSJEkqyPAlSZJUkOFLkiSpIMOXJElSQYYvSZKkggxfkiRJBRm+JEmSCjJ8SZIkFWT4kiRJKsjwJUmSVJDhS5IkqSDDlyRJUkGGL0mSpIIMX5IkSQUZviRJkgoyfEmSJBVk+JIkSSqotdSGIqIZuBgYBywGDsnMWTXthwKHAcuAUzLzR6VqkyRJKqVkz9dkYN3MnAh8BZje1RARGwNTgfcDewCnR8TQgrVJkiQV0dTZ2VlkQxFxDnBfZn6/Ov3XzNys+vkjwN6ZeXh1+ibgtMz8bZHiJEmSCinZ8zUSmF8z3RERrT20LQDWL1WYJElSKSXD10vAiNptZ+ayHtpGAPNKFSZJklRKyfA1E9gbICImAA/XtN0HfCAi1o2I9YF/AP5QsDZJkqQiSt7z1fW047uAJmAKlTA2KzNvrT7t+DkqgfC0zLyxSGGSJEkFFQtfa5qI6PoP89bMfGKFtsOBbwGnZub/6ePtbgH8GXh77VAcUiOJiP2B7wHHZOb0VS2/kvUcRGVomjERsQvwC2BIzS0NUkMYqHOS+oeDrK7cUuDD3cyfDJhapd7bH5gFfGY113Md8J7VL0daI3hOahCGr5X7NfCR2hkRMRJ4H/DggFQkreEiYgMq4/mdCGwbEb0OT5n5SmbO7avapEHOc1KDKDbC/RrqFmB6RKyfmV1DYewN/CcwvHbBiDiOygj9Y4AXgMsz84Rq2y+pPECwJ7AelVH+RwAXADtTGWbj4sw8rWaVH4mIzwObAT8DPpOZL/THTkqF7UflLRfXAScAB1E9cVSPlV8BHwR2AP4L+Fxm/rHa3gmcAhwOPAR8l+plx6J7IA2M1T4nVR94mwlslpnPVpcN4BFg08z8W5ldWbvZ87VyjwKzgb1q5u0L3Fy7UER8CjgGOBR4B3AScHxEjK9ZbAqVk8y+VMYx+zGVLuSJwGeBL0fEASss/0lgFyqXVb7aN7skDbhPArdX78u6BfhkRAypaT8OuAnYDpgD3B4R69a07wvsCHyxUL3SYLHa56TMvLe6jv1qvvJx4GcGr3IMX6t2C9Vr7NUTxB7ArSss81dgSmb+LDNnZ+YlwLPAO2uWuT0z78rM+4F/BDYFDsrMP2TmncARwMs1yx+Xmfdl5m+A66n0lklrtIjYFNiJ/3+y+AGwIdVhaKruzMwZmfkolZPHm6j0Gne5LCseKVGzNMj0xTnpWuBjNct/vDpPhRi+Vu0WYK/qaPy7Ao9k5nO1C2TmL4C5EXF6RNwcEU8CGwMtNYvNrvm8NZUhNubXrOOazKz96+VPNZ/nA7V/+Utrqk8AHcBt1enfAE/z2hvv7+76kJkLgP+mMvZfl9n9W6I0qPXFOel7wI4RsUlEbAO8lUpvswoxfK3a3cAyKpc59qWbX9CI+CyV+7LWo/KX/G5ULpfU+nvN5yV1bLdjhemmOuuVBrP9gSHACxGxjMql902AfSJiw+oyKw4T0QIsr5n+O9Laa7XPSdV7KB8G/olKD9httZ0B6n+Gr1XIzOXAj6g8YfJhVri2XvUvVMZXOSoz/w14HtiIngPT48Bbq0+pABARJ0XEd/qydmkwiYi3A9sDXwLeXfNvHyqBbP/qou+u+c76wNuA3xctVhqk+vCcdG31+x/CS47F+bRjfW4B/h14YsXB7apeAHaLiB8AbcBpVE4mQ3tY353AU8DlEXEisAUwlcoTXFKj2p/KO1svycza3qs/RMTdVC49LgQ+HhE/o3JJ8htU/mL/aelipUGsL85J11IZ7mUJlTCnguz5qs+PqVz66O4vDKg8dTWMyuPyN1Hpzr2RHgZ/zMwOKt3FbwIeAC4BTs7M6/q2bGlQ2R/43grBq8u3gPcC21C5H+UQKsNMtAF7ZObSYlVKg99qn5My8y9Uzj+3ZuYr/VqtXsfXC0kaNKrjfN3lK1Kk/hcRfwKOzMzbB7qWtY2XHSVJWotExN5UbsJvodKLpsIMX5IkrV2+BGwLHFC9DUaFedlRkiSpIG+4lyRJKsjwJUmSVJDhS5IkqSDDlyRJUkGGL0mSpIL+H9kOPGDgusqhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1060bf588>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "grouped_by_month_control = control_group['timestamp'].groupby(control_group['timestamp'].dt.month).count()\n",
    "grouped_by_month_control = grouped_by_month_control/sum(grouped_by_month_control)\n",
    "\n",
    "grouped_by_month_test = test_group['timestamp'].groupby(test_group['timestamp'].dt.month).count()\n",
    "grouped_by_month_test = grouped_by_month_test/sum(grouped_by_month_test)\n",
    "\n",
    "x1 = [1, 5, 9]\n",
    "x2 = [2, 6, 10]\n",
    "fig, ax = plt.subplots(figsize=(10,4))\n",
    "plt.bar(x1, grouped_by_month_control, label='control')\n",
    "plt.bar(x2, grouped_by_month_test, label='test')\n",
    "plt.xticks(x1, ('March', 'April', 'May'), fontsize=14)\n",
    "plt.ylabel('fraction', fontsize=14)\n",
    "plt.legend(loc=2, fontsize=14)\n",
    "plt.ylim([0.0, 0.5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do this for the other variables (operating system, source, device) to confirm that the distributions of control and test samples in the various categories are consistent - just operating system results are shown below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:4: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead\n",
      "  after removing the cwd from sys.path.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmAAAAD5CAYAAACAnYlyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmYXGWZ9/Fvuhuy0CHtaAeUMCA6cw8oBtlMNOwEDaLGbcaFV4EXCBBECCCCC6CyG3ZwEETDgAuIoCigMPMqIYCyG1RuCBARQScRQhIiIdv7x6kOZdOd7iZdp6o738915UpVnTqn7q6nll89z3POGbJq1SokSZJUnqZ6FyBJkrSuMYBJkiSVzAAmSZJUMgOYJElSyQxgkiRJJTOASZIklayl3gX0xbx5ixr6mBmtrUNZvHhpvctQH9luA5dtNzDZbgOT7dZ37e0jh3S3zB6wftTS0lzvEvQq2G4Dl203MNluA5Pt1r8MYJIkSSUzgEmSJJXMACZJklQyA5gkSVLJDGCSJEklM4BJkiSVzAAmSZJUMgOYJElSyQbUkfClgaT9ojF1edx5U5+qy+NKknrPHjBJkqSSGcAkSZJKZgAbhJYtW8b111/7qte/8cYb+OAH9+7HiiRJUrVBPwdsh+m3lfp4dx+9c6mP15Vbb/05M2Z8i8mTP1zvUiRJUhfsARuEVq1aVe8SJEnSGgz6HrBG9/TTf+bcc8/i/vvvo7W1lQ9+8CN86lMH8L//+1cuuOAc7rnnNzQ1DWGPPfZi6tQjGTp0KDfeeAM33HA9O+44jmuv/QHNzc3svvtefOYzR/HAA/dx6qknAzBhwvZcc81POOWUk9hiizfx61/fydKlS/nOd77HSy8t7Xb7kiSptgxgdfTSSy8xbdrhvPGNb+KSSy5n3rx5nHji8bS3j2bGjMsZM2YMF1xwCQsXPs/pp3+VVavg6KOPA+APf/gdo0eP5qKLLmP27Ac588xT2HHHcWy//Y4cccTRXHXVDC6//Era2l4DFPO6pk+/kKFD12eDDTbgkEMOWOP2JUlS7RjA6uiee37D/PnzuOyy/6K1tZUttngz06Ydx4oVK5g3769885vfZsMNRwEwbdpxHHfcUUyZMhWAFStWcOyxX6C1tZXNNtucH/3oah5++PeMH/8uWltbaWpq4rWvfd3qxxo37p2MHbsNALff/qsety9JkmrHAFZHc+c+ziabbEpra+vq2/baaxJXXvkdxozZdHU4Ath667exYsUKnnrqSQBGjWr7h/VGjNiA5cuXd/tYG2/8hqrHfaLH7Q9EZe9wAY2x04UkaeBxEn4dtbSs1+Xt66//ynlYK1as/If/11vvleuuafL9+uuv36ftS5Kk2jGA1dGmm27K008/xQsvLF5927e+dQlXXTWDp576EwsXPr/69t/97rc0NzczZkzPp7cZMmTIGpdvvvkb12r7kiRp7RjA6mjHHcczevRGnHHGKcyd+wR33jmLa675PkccMY1NN92Mr371y8yZ8yj33XcP5577dfbYYy9GjWrrcbvDhw9n8eLFPPnkH7scltx++x3XavuSJGntDPo5YGXO0WlrG8GCBUt6ff/m5mZOO206Z599BgccsC+vec1r2H//A9ljj73Yaqu3cs45ZzJlyn4MHz6CvfZ6D1OmHN6r7W677Q5sttnm7Lffx7n44m+9YnlTUxOnnfb1V719SZK0doYMpIN2zpu3qKGL7WsAU/9qtEn47RfVZzh33tSn6vK49eB7bmCy3QYm263v2ttHdjsnqCY9YBHRBFwMjAWWAgdm5pyq5VOB/YBVwFcy86cRMRy4EhgNLAI+nZnzalGfpMbRaMFZkspQqzlgk4FhmTke+DwwvWNBRLwOOAx4J7AH8I2IGAIcCszOzJ2AK4Av1qg2SZKkuqpVAJsA3AyQmXcB23csyMz5wNjMXAZsDCzIzFXV6wA3AXvWqDZJkqS6qtUk/A2B56uur4iIlsxcDpCZyyPicOBk4Pwu1lkEjKKT1tahtLQ016jktdfc3ERb24h6l6ESNWJ7N2JNjcbnqL78rByYbLf+VasAthAYWXW9qSN8dcjMCyPim8BNEbFbp3VGAgs6b3Tx4qU1Krd/OEFx3bOm9m4vsY5qvgZ75nNUX35WDky2W9+1t4/sdlmthiBnAXsDRMQ4YHbHgij8qDLvaxnFJP2V1esAk4CZNapNkiSprmrVA3YdMDEi7gCGAPtHxDRgTmb+JCIeBO6k2Avypsz8VUTcDcyIiNuBl4BP1Kg2SVqjehxCZF06fIikGgWwzFwJHNLp5oerlp9MMf+rep0lwEdrUY8kSVIj8VREkiRJJTOA1dmjjyYPPnj/Wm1j1apVXH/9D1m5cmU/VSVJkmpp0J8LsvS5HH2cx3HCCcfyqU8dwNixb3/VD/nAA/fx9a+fzj77TKapyUwtSVKj89u6zvrjXJwD6XyekiTJAFZXhx9+MH/5yzOcccbXOOWUk3j88cc44ohD2H33d/Ef/zGZ733vytXh6oUXFvPlLx/PpEm7s9deu3DCCcfyt7/N55lnnuaII4r9HXbddRz33XdPPf8kSZLUCwawOjr11LMYPXojDj/8SD772WM45pgjeMtbtmbGjO9x5JHHcs013+Paa38AwKWX/ifPPPM0F174TS655Ns899yznH/+2YwevRGnnHImANdddyNbbz22nn+SJEnqhUE/B6yRbbjhKJqamthgg1Z++ctbGTlyQ6ZMmQrAppv+MwcddCjf/valfOQjH+Mvf3maESNG8PrXv4ERI0bwpS99hUWLFtHc3MzIkRsC8JrX/BMtLTapJEmNzm/rBjF37lzmzn2ciRN3Wn3bypUrWbZsGcuWLWPffffjmGM+yz77TGTbbbdj55134z3veW8dK5YkSa+WAaxBrFixgm222Y5jjz3+Fcuam5t561vfxrXX3sCsWTO5885ZXHjhudxyy81ccMEldahWkiStDeeA1dmQIUMA+Od/3ow//emPbLzx6xkzZlPGjNmURx9NrrpqBk1NTVx99Xf5/e8fYq+9JnHiiV/jrLPO4/777+XZZ/+2ehuSJGlgMIDV2fDhw/njH+cyfvy7WLZsGWec8TXmzn2Cu+++i7PPPnP1/K6//vWvnHPOWcye/SB//vNT3HLLTWy00caMGtXG8OEjAHjkkYdZunRpPf8cSZLUC4N+CLLME9y2tY2ABUv6tM6HPvTvXHTReTzzzJ+ZPv18zj//bA44YF9GjhzJpEn7cPDBhwFw0EGHsmTJCxx//DEsWbKErbZ6C2eeeS7Nzc286U1vZscdxzN16kGcdNKp7LLLbrX48yRJUj8Z9AGs0U2e/GEmT/7w6usXXvjNLu83bNgwjjvuixx33BdfsWy99dbj7LMvqFmNkiSpfzkEKUmSVDIDmCRJUskMYJIkSSUzgEmSJJXMACZJklQyA5gkSVLJDGCSJEklM4BJkiSVzAAmSZJUMgOYJElSyWpyKqKIaAIuBsYCS4EDM3NO1fKjgI9Vrt6YmSdHxBDgKeDRyu13ZubxtahPkiSpnmp1LsjJwLDMHB8R44DpwAcAImIL4JPAO4BVwMyIuA5YAtyXme+rUU2SJEkNoVZDkBOAmwEy8y5g+6plfwLek5krMnMlsB7wIrAdsElE/L+IuDEioka1SZIk1VWtesA2BJ6vur4iIloyc3lmLgPmV4YczwLuz8xHImJj4LTMvCYiJgBXAjtUb7S1dSgtLc01KnntNTc30dY2ot5lqESN2N6NWFOjacTnqBFrqhU/Kwcm261/1SqALQRGVl1vyszlHVciYhhwObAIOKxy8z3AcoDMvD0iNomIIZm5qmO9xYuX1qjc/tHWNoIFC5bUuwyVaE3t3V5iHdV8Dfasp+eoHm23LrWbn5UDk+3Wd+3tI7tdVqshyFnA3gCVOWCzOxZUer5+DDyYmVMyc0Vl0YnAkZX7jAWerA5fkiRJg0WtesCuAyZGxB3AEGD/iJgGzAGagV2AoRExqXL/44HTgSsj4r0UPWH71ag2SZKkuqpJAKtMrj+k080PV10e1s2q761FPZIkSY3EA7FKkiSVzAAmSZJUMgOYJElSyWo1CV+SNIjtMP22ujzu3UfvXJfHlfqbPWCSJEklswdMkjQotF80pvTHnDf1qdIfU4ODPWCSJEklM4BJkiSVzAAmSZJUMgOYJElSyQxgkiRJJTOASZIklcwAJkmSVDIDmCRJUskMYJIkSSUzgEmSJJXMACZJklQyA5gkSVLJDGCSJEklM4BJkiSVzAAmSZJUMgOYJElSyQxgkiRJJTOASZIklaylFhuNiCbgYmAssBQ4MDPnVC0/CvhY5eqNmXlyRAwHrgRGA4uAT2fmvFrUJ0mSVE+16gGbDAzLzPHA54HpHQsiYgvgk8A7gfHAXhHxNuBQYHZm7gRcAXyxRrVJkiTVVa0C2ATgZoDMvAvYvmrZn4D3ZOaKzFwJrAe8WL0OcBOwZ41qkyRJqquaDEECGwLPV11fEREtmbk8M5cB8yNiCHAWcH9mPhIR1essAkZ13mhr61BaWpprVPLaa25uoq1tRL3LUIkasb0bsaZG04jPUSPW1Iga7XlqtHpqye+4/lWrALYQGFl1vSkzl3dciYhhwOUUQeuwLtYZCSzovNHFi5fWpNj+0tY2ggULltS7DJVoTe3dXmId1XwN9qyn56gebWe79U6jvefWpXbzO67v2ttHdrusVkOQs4C9ASJiHDC7Y0Gl5+vHwIOZOSUzV3ReB5gEzKxRbZIkSXVVqx6w64CJEXEHMATYPyKmAXOAZmAXYGhETKrc/3jgG8CMiLgdeAn4RI1qkyRJqqseA1hEbAp8HBjWcVtmfmVN61Qm1x/S6eaHqy4Po2sf7akeSZKkga43Q5DXUEyq/2vVP0mSJL1KvRmCXJSZHpNLkiSpn/QmgD0UER8D7gdWAWTmIzWtSpIkaRDrTQDbpvKvwypg99qUI0mSNPj1GMAyc7eIeC3wJuDxzJxf+7IkSZIGrx4n4UfER4E7gBOAuyJi35pXJUmSNIj1Zi/IacB2mTkZeDvw2dqWJEmSNLj1JoCtzMzFAJm5iOLE2ZIkSXqVejMJ/7GImA7cBuwMPFbbkiRJkga33vSAHQA8DkykCF8H1bQiSZKkQa7bABYR21cu7g48CvyE4lyOu5VQlyRJ0qC1piHIPYB7KM4DWW0V8IuaVSRJkjTIdRvAMvOMysVZmXlZx+0RcUTNq5IkSRrEug1gEfFx4P3AbhHRceT7ZuCtwPkl1CZJkjQorWkI8mbgGeC1wH8CQ4CVuBekJEnSWul2En5mPpeZvwQOAzbJzF8B7wXWL6k2SZKkQak3h6GYQdETBnAj8K3alSNJkjT49SaAUekJIzNv6+06kiRJ6lpvjoS/ICIOBu4EdgQW1bYkSZKkwa03vVn7AVsBZ1b+P6CWBUmSJA12PfaAZea8iDgDWI9iT8h/AebXujBJkqTBqscAFhGXA+OADYDhFOeFHFfjuiRJkgat3gxB/hvwFuDnFEOQL9a0IkmSpEGuN5PwF2XmqojYIDPnR0SPxwGLiCbgYmAssBQ4MDPndLpPO3AHsHVmvhgRQ4CnKE78DXBnZh7flz9GkiRpIOhNALs3Io4Bno6I71Ocjqgnk4FhmTk+IsYB04EPdCyMiHcDpwMbVa3zJuC+zHxfr6uXJEkagHoTwGYATwN/ByYBv+nFOhMoTmVEZt4VEdt3Wr4S2BO4t+q27YBNIuL/VR7rqMzMXjyWJEnSgNKbAPatzJxQuXxDL7e7IfB81fUVEdGSmcsBMvMWgIioXucZ4LTMvCYiJgBXAjv08vEkSZIGjG4DWESMyszngRci4hwgKXquyMxv9rDdhcDIqutNHeFrDe4BOgLa7RGxSUQMycxVHXdobR1KS0tvRkDro7m5iba2EfUuQyVqxPZuxJoaTSM+R41YUyNqtOep0eqpJb/j+teaesBuAHYGngCeA0b3YbuzgPcBV1fmgM3uxTonAn8DzoyIscCT1eELYPHipX0ooXxtbSNYsGBJvctQidbU3u0l1lHN12DPenqO6tF2tlvvNNp7bl1qN7/j+q69fWS3y9YUwP4eEXdTHHj1D1W3rwK+0sNjXgdMjIg7KA7eun9ETAPmZOZPulnndODKiHgvRU/Yfj08hiRJ0oC0pgA2CXgDcAlwWF82mpkrgUM63fxwF/fbvOryc8B7+/I4kiRJA1G3AawSop7CUCRJktSvenMkfEmSJPUjA5gkSVLJDGCSJEklM4BJkiSVzAAmSZJUMgOYJElSyQxgkiRJJTOASZIklcwAJkmSVDIDmCRJUskMYJIkSSUzgEmSJJXMACZJklQyA5gkSVLJDGCSJEklM4BJkiSVzAAmSZJUMgOYJElSyQxgkiRJJTOASZIklcwAJkmSVDIDmCRJUskMYJIkSSVrqcVGI6IJuBgYCywFDszMOZ3u0w7cAWydmS9GxHDgSmA0sAj4dGbOq0V9kiRJ9VSrHrDJwLDMHA98HphevTAi3g38Atio6uZDgdmZuRNwBfDFGtUmSZJUV7UKYBOAmwEy8y5g+07LVwJ7As92tQ5wU2W5JEnSoFOTIUhgQ+D5qusrIqIlM5cDZOYtABHR3TqLgFGdN9raOpSWluaaFNwfmpubaGsbUe8yVKJGbO9GrKnRNOJz1Ig1NaJGe54arZ5a8juuf9UqgC0ERlZdb+oIX71cZySwoPMdFi9e2j/V1Uhb2wgWLFhS7zJUojW1d3uJdVTzNdiznp6jerSd7dY7jfaeW5faze+4vmtvH9ntsloNQc4C9gaIiHHA7L6sA0wCZtamNEmSpPqqVQ/YdcDEiLgDGALsHxHTgDmZ+ZNu1vkGMCMibgdeAj5Ro9okSZLqqiYBLDNXAod0uvnhLu63edXlJcBHa1GPJElSI/FArJIkSSUzgEmSJJXMACZJklQyA5gkSVLJDGCSJEklM4BJkiSVzAAmSZJUslodiHXA2mH6bXV53LuP3rkujytJkspnD5gkSVLJDGCSJEklcwhyAGi/aExdHnfe1Kfq8riSJA129oBJkiSVzAAmSZJUMgOYJElSyQxgkiRJJTOASZIklcy9ICVJUt2sq3v62wMmSZJUMgOYJElSyRyClCRpHVKPcx57vuNXsgdMkiSpZAYwSZKkkhnAJEmSSmYAkyRJKllNJuFHRBNwMTAWWAocmJlzqpYfBEwBlgNfy8yfRsQ/AY8AD1Xudl1mnleL+iRJkuqpVntBTgaGZeb4iBgHTAc+ABARGwNHANsDw4DbI+IWYFvge5n5mRrVJEmS1BBqNQQ5AbgZIDPvoghbHXYEZmXm0sx8HpgDvA3YDtg2In4VEddExOtrVJskSVJd1aoHbEPg+arrKyKiJTOXd7FsETAKeBi4NzNvjYhPAhcAH6neaGvrUFpammtUcn21tY2odwmv0Ig1NZpGfI4asaZG04jPUSPW1Iga7XlqtHoaVSM+T/WuqVYBbCEwsup6UyV8dbVsJLAA+DWwpHLbdcBXOm908eKl/V9pg1iwYEm3y9pLrKPammpSwXYbmHp6jurRdrZb7zTae852651Gazcop+3a20d2u6xWQ5CzgL0BKnPAZlct+w2wU0QMi4hRwJYUE+8vAz5cuc8ewL01qk2SJKmuatUDdh0wMSLuAIYA+0fENGBOZv4kIs4HZlIEwC9k5osR8Xng8og4DHgBOLBGtUmSJNVVTQJYZq4EDul088NVyy8FLu20zhPAbrWoR5IkqZF4IFZJkqSSGcAkSZJKZgCTJEkqmQFMkiSpZAYwSZKkkhnAJEmSSmYAkyRJKpkBTJIkqWQGMEmSpJIZwCRJkkpmAJMkSSqZAUySJKlkBjBJkqSSGcAkSZJKZgCTJEkqmQFMkiSpZAYwSZKkkhnAJEmSSmYAkyRJKpkBTJIkqWQGMEmSpJIZwCRJkkpmAJMkSSpZSy02GhFNwMXAWGApcGBmzqlafhAwBVgOfC0zfxoRrwO+CwwHngb2z8wltahPkiSpnmrVAzYZGJaZ44HPA9M7FkTExsARwLuAdwOnRcRQ4MvAdzNzJ+B+ioAmSZI06NQqgE0AbgbIzLuA7auW7QjMysylmfk8MAd4W/U6wE3AnjWqTZIkqa6GrFq1qt83GhGXAddm5k2V608CW2Tm8ojYF9g6M4+rLLsCuAL4z8rtf4+ILYArMnNCvxcnSZJUZ7XqAVsIjKx+nMxc3s2ykcCCTrd33CZJkjTo1CqAzQL2BoiIccDsqmW/AXaKiGERMQrYEnioeh1gEjCzRrVJkiTVVa2GIDv2gnwbMATYnyJczcnMn1T2gjyYIgCempnXRsRGwAyK3q/5wCcy84V+L06SJKnOahLApDJFxCpgYmbe+irW/RowITN37ffC9KpExObAE8C/ZOaciBgJnAD8B/AG4BngRxQ/3v5WtV4LxV7X+wGbUvyQ+xnwxcz83zL/Bv2jStssA3bLzF++ivVPAvbsbl5wRNwO3JqZJ61FmeuEiBgNnAS8H3gt8CTFIaDOzMy/V+6zGzAvMx+KiP0oDhc1pj4VD14eiFWDweuB2+pdhPrNnyja9ImIaKVo20nAZ4AADqTYm/quymFtOpwGfBw4FPhX4GPA1sBNETGkvPJVA1+nCAxaCxGxCcU0oC2BfSv/Hwt8BPhVRIyo3PV/gI273Ij6TU0OxCqVKTP/Uu8a1H8ycwXwF4CI+CqwAbBtZi6u3OWPlR6PO4FzKEIXwAHAlMy8pep+nwQeA94B3FXSn6B+VtX2WjsXUvzAmVi1Y9zciLgN+B3wJeD4ehW3rjGArUHVUMgHgPOB1wGXAN+p/Ps34L8pvgBeAk6tXN6I4mj+p2fmNyrbGgGcSTGM0gz8FJiamYvK+nsGmoh4ELg8M8+rXL8eeGNmjq1c/zBwOvBmKkOQETGX4tfyJ4BtKHbwmJqZd1fW2Qr4JrAtxY4fczo95njgLODtwDzgrMy8KCI+CFwGtGfmyojYhuKAwR/KzOsq686u1HMDcCkwkeI9ditwmEGxd6red/9GEapO6PwFnJlLI+I04LsRMTUznwVWAbtHxHWVEEdmPh4Rb6lsT31UeT+cCWxH8fzOBP4vxWv7QOAXFD2Ty4EfANMyc2Vl3S8Dh1PMAz6+03bnAldT9MIsoJgv/C8UgfqdwGKK9+lXKu+3k6gagqy8H88ANqF4Xzqa04PK0OP7gfdXhS8AMnNBRJwLHFs5VBTALRFxMjC3sv6XKA6ivh7wLeCYzFxVWXYwxfD/aOAB4Kiqz9y5dGrrzo+/rvJF2zufp3jhHgJMA34IfI5iWGQXii+J4yr3+QjFMMl3gAsi4g2VbVwC7AF8CNgNeCtwdml/wcD0c4rnisoQ0k7AWyp7z0JxsN6buljvRIovjXHAixS/+qicceFnFF/G2wLXAQd1rBQRW1J0vd9GEcBOBM6MiI9SBO0NKb4ooGj3VRRndOg4w8OWFF9IXwU2r9xnHMWH0jlr8Tysq9aneM5/083ymRQBd7vK9fMohh//GBGXRsTHIqItM3/fMbdFvVeZe/czih8QbwH2ArYAvlC5y47AVhTvyy9QBLF3V9Y9GDiS4rNxYuX/zv4P8B6KH0ttFO35NEVv5aHAVODoLuraiuIL/RsUbT8MGL+Wf+66YDuK7/w1vZ/aKb6jAP6d4scsFEH3rRQHTJ8CHAW8FyAi3kfxmXcUxefmTcD/RMTrq7a9uq0NXy8zgPXO1zLzt5l5JfAs8P3M/O/MvA34JcUv9Ycoznl5V2Y+TtEb1gxEJTB8DPhMZs7MzAcpwtwj9fhjBpCfAztX9qrdmuK5f4wi1EDxwX5zF+tdkZnXZ+ZvKT5AOs7EsCfFB8yhmflwZl4MXF+13kHAbzPzhMx8JDNnABcAn8vMhRRDWLtW7rsLxQfNu6q2fV9mzqMIX4uBJzLz9xQfPme9+qdhnfVPlf+f62Z5x+2vA8jMr1K8zx6j2PP6e8AzEXFsLYscxDag+Bz7SmY+kZmzgGspwhgU4XdK5b10OfAgsENl2UHA+Zn508x8gGKv986uqnyuPkARwl6sbO8PmfljiuGwz3Wx3v4UZ1M5JzMfpuhls3e5Z719P3WMjD1X1fO8HDgoCz+gaOuxlWWfoxjt+XFmPpqZpwD3UPSQdqhua1UYwHrn8arLfwf+2On60My8HhgWEdMj4mdUum0pQti/Uryo7+1YKTN/k5l+Ka/Z7cBQil6nXSh+od0FTIiIN1L8KvtlF+s9VnV5IdAUEc0Uv9Yf6zScdU/V5S2BX3fa1h0UARuKQLhrVW/cWcC2ETGcIgx29MadTtHDNi8ibqQIZ7/r5d+slz1b+b+7ycAdvcur94TMzB9k5i4UoeyjFK+hMyPiAzWrcpCqDJl/BzgqIq6IiHuAYyg+0wDmV04n12EhxfAUFO+1B6u29RBFwKo2t+rylhQ/YJZV3XYH8LqIeF2n9Tpve1n1dXWrz++nKvMrP0I7PE/R8whF250aEYs7/lH0lP1r1f3nvrqSBzcDWO907jJd2fkOlcMZfLdy3//i5V4aKOaHqY8ycylFwNqVlwPYTIpep4nAbZm5pItVu3q+h3T6v0P1B37nLwgovmw6fhH+HNiZojduSWV3+nkUQzF78vL5T+8ENqP4Bfgcxcnou+qp05r9neLLYPtulu9A8X67LyLeFhHndSzIzAWZ+UOKYbN7KF4v6oPKHnOzKV7b91IMMU2vusua3medL8MrP0df7OZyh+ZO/3f3OPCP72N17W6KNljT+2k+/9jh0GFFF7d1tEELxVDxNlX/Ovau7NBV+67zDGD95xDgiMw8LjO/T9F9D8WL9HGKF/DbO+4cEXtGxCOV4TV1r2Me2AReDmDvAPah6/lfa/IQ8OaIeE3VbW+vuvyHyrarjQeycvleinY8nJfP1DCTou2HUZlbERFHAu/IzKsy85MUByHetXKwYfXNpcDnKvORVouI9YAvAj/KzPkUXwJHVM68sVplkvDzFEFZffNBYGFm7p2Z52XmTIo5YL05pMdDvDwcSUS8GWhdw/3/QNGbvF7VbeMpem06t13nbTfz8txMdaPyPvkhcGLluGyrVabJTKPY6amvc7QS2DQz53T8owhku/ZD2YPvF7slAAACaklEQVSae0H2n78B+0TErymOYXR+5fahmbkoIr4NnFc5C8BSikni/92xx5C69XOKob5nK29sImIRRQDran7ImtxKMXx8eUR8gaKX8iO8fHiCi4EjI+JUiqGXcRQTgT8LUNkb61bg0xR7A0ExYf8i4OqOPe8oDgJ6SEQcQDE35ZMUBzuc38d6VUzu3Q24LSJOoBjKfSPFgSQ34OW2uS8ifgr8KCKOp2iX11KEiG0oDs6qvvkbsElETKQY1v8o8GGKvX97ciFwcUTcRxGuLqCLkYMq3wVOBi6JiLMo9og8GfhG5X1Xfd/LgM9W9rL8AXAY4EFCe+coih+Nv6js4fgExZy+0ygOT3Fy5X6LKXZ4ursX2zyb4jP1YYoh/30pdrq4pJ9rH3Tsfek/B1AMTf0OuAK4huKLvaOH5SiKHpKbgVsohkWOKb/MgSUzk2LPqOpzg84EnqxMwO3LtpZR9EZtSNGbdTBF6OpY/hTFnj3vphh6+RJwdGZeVrWZn1PsnddRz20UPQLVQ4xfqiy/nuL1sCWwT1VAUy9Vhph3B35MsZdjUpyy7H5gh06H9vh3ih6z44HfUwTutwI7V9pWfXM1xXSKqyneL3tQfI4FMHxNK2bmfwFfpmizmRR7U3Z7yJ3KvMz3AG+iaNuLKut+uYv7Pgq8j6K9H6CY7+cQfy9U3i/jKZ63GRTvp3Mp9gjfuWpKxzkUc1lP7MU2f0BxpIATKT7vPghMdsJ9zzwVkSRJUsnsAZMkSSqZAUySJKlkBjBJkqSSGcAkSZJKZgCTJEkqmQFMkiSpZAYwSZKkkhnAJEmSSmYAkyRJKtn/B6aYP+/gcefkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a0f8a85f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def onehot_encode_function(categorical_column):\n",
    "    onehot_encoder = OneHotEncoder(sparse=False)\n",
    "    encoded = categorical_column.dropna()\n",
    "    encoded = encoded.reshape(len(encoded), 1)\n",
    "    onehot_encoded = onehot_encoder.fit_transform(encoded)\n",
    "    return(onehot_encoded)\n",
    "\n",
    "onehot_opsys_control = onehot_encode_function(control_group['operative_system'])\n",
    "onehot_opsys_test = onehot_encode_function(test_group['operative_system'])\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10,4))\n",
    "plt.bar(np.arange(0, 15, 3), np.sum(onehot_opsys_control, axis=0)/np.sum(onehot_opsys_control), label='control')\n",
    "plt.bar(np.arange(1, 16, 3), np.sum(onehot_opsys_test, axis=0)/np.sum(onehot_opsys_test), label='test')\n",
    "plt.xticks(np.arange(0, 15, 3),list(opsys_cats.keys()), fontsize=14)\n",
    "plt.legend(loc=2, fontsize=14)\n",
    "plt.ylabel('fraction')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we have our data loaded in and cleaned, and we've mapped our categorical variables we'll use later to numerical values. We've also confirmed that our control and test sets do not appear to be systematically biased when compared with one another. \n",
    "\n",
    "# 2. The impact of software price on conversion rate and revenue\n",
    "\n",
    "### Testing conversion rate differences\n",
    "\n",
    "We can now start our analysis. The first question we are interested in answering is whether or not the conversion rate went down for the higher-price group relative to the control. Intuition tells us that as price increases conversion rate should decrease, but we can assess the statistical significance of any difference.\n",
    "\n",
    "We can frame the problem as a hypothesis testing problem: our null hypothesis is that there is no difference in conversion rate, and we can use a z-test to accept or reject this null hypothesis. We use a z-test here because our sample sizes are very large. The underlying distribution is binomial, so we use the appropriate z-test. \n",
    "\n",
    "First, we calculate the conversion probabilities in the control/test groups:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conversion rate (control): 0.019875838340522076\n",
      "conversion rate (test): 0.015578063109764203\n"
     ]
    }
   ],
   "source": [
    "# calculate probabilities based on the samples\n",
    "num_control = control_group['converted'].count()\n",
    "num_test = test_group['converted'].count()\n",
    "\n",
    "prob_buy_control = control_group[control_group['converted']==1]['converted'].count()/num_control\n",
    "prob_buy_test = test_group[test_group['converted']==1]['converted'].count()/num_test\n",
    "\n",
    "print('conversion rate (control):', prob_buy_control)\n",
    "print('conversion rate (test):', prob_buy_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks like a very small difference in conversion rate, but since the overall conversion rate is small it's actually a difference of 20%. But how significant is the difference?\n",
    "\n",
    "We can use those calculated conversion rates to calculate the p-value associated with our null hypothesis: how likely is it that the difference in conversion rates is due to random chance?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the p-value is 0.0\n"
     ]
    }
   ],
   "source": [
    "# compute the significance of this result, leveraging what we just calculated - null is that p1=p2\n",
    "pjoint = (num_control*prob_buy_control+num_test*prob_buy_test)/(num_control+num_test)\n",
    "\n",
    "zstat = (prob_buy_control - prob_buy_test)/math.sqrt((pjoint*(1-pjoint)*(1/num_control + 1/num_test)))\n",
    "pval = 1.0 - spst.norm.cdf(abs(zstat))\n",
    "print('the p-value is', pval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our p-value is essentially zero - thus we can reject our null hypothesis with a high degree of confidence. Another way of putting it is that at the greater than 99.99% confidence level, we can rule out the conversion rates being the same. \n",
    "\n",
    "### Testing revenue differences\n",
    "\n",
    "So it looks like our conversion rate decreases by 22% when the higher price is implemented. But overall we are interested in how revenue is affected - we may have lost some customers, but those that remain may make up the lost revenue with higher purchase prices.\n",
    "\n",
    "The test for this is the same as that used above, except now we are testing the difference in \"revenue per customer\". That is, the conversion rate times the price of the software. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "revenue per customer (control): 0.7751576952803609 revenue per customer (test): 0.919105723476088\n",
      "the p-value is 0.0\n"
     ]
    }
   ],
   "source": [
    "expected_revenue_control = 39.0*prob_buy_control\n",
    "expected_revenue_test = 59.0*prob_buy_test\n",
    "\n",
    "print('revenue per customer (control):', expected_revenue_control, 'revenue per customer (test):',expected_revenue_test)\n",
    "\n",
    "pjoint = (num_control*39.0*prob_buy_control+num_test*59.0*prob_buy_test)/(num_control+num_test)\n",
    "zstat = (39.0*prob_buy_control - 59.0*prob_buy_test)/math.sqrt((pjoint*(1-pjoint)*(1/num_control + 1/num_test)))\n",
    "pval = 1-spst.norm.cdf(abs(zstat))\n",
    "print('the p-value is', pval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we know that revenue increases by 19% under the higher prices - this means that the drop in customers is far outweighed by the higher price that the remaining customers are paying. \n",
    "\n",
    "# 3. Other insights\n",
    "\n",
    "### Testing efficiency\n",
    "\n",
    "We achieved very high statistical significance with this A/B test - so high in fact that we are in the realm of diminishing returns, where collecting more data only marginally impacts the overall significance because our significance is so high. \n",
    "\n",
    "Thus, it is worth seeing how much shorter we could have run the test and still generated the same insights. If the testing period could be made shorter, then insights could be generated more rapidly and important business decisions made earlier.\n",
    "\n",
    "The way we can see how much shorter we could have made the test is by setting a target significance, then reducing the sample size until that target significance is reached. As large changes in pricing are an important business decision and inappropriate pricing can have lasting business impact, we adopt a 3-sigma threshold (>99.7% confidence)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sigma level: 2.9826519948572128\n",
      "the p-value is 0.0014288135836746374\n"
     ]
    }
   ],
   "source": [
    "# calculate probabilities based on the samples\n",
    "num_control = control_group['converted'].count()\n",
    "num_test = test_group['converted'].count()\n",
    "\n",
    "prob_buy_control = control_group[control_group['converted']==1]['converted'].count()/num_control\n",
    "prob_buy_test = test_group[test_group['converted']==1]['converted'].count()/num_test\n",
    "\n",
    "# define a factor that lowers the number of samples in each group\n",
    "factor = 0.12\n",
    "# just re-run the test from before, except with a smaller sample size\n",
    "pjoint = (factor*num_control*prob_buy_control+factor*num_test*prob_buy_test)/(factor*num_control+factor*num_test)\n",
    "\n",
    "zstat = (prob_buy_control - prob_buy_test)/math.sqrt(pjoint*(1-pjoint)*(1/(factor*num_control) + 1/(factor*num_test)))\n",
    "print('sigma level:', zstat)\n",
    "pval = 1.0 - spst.norm.cdf(abs(zstat))\n",
    "print('the p-value is', pval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we can see from the above test is that we could have reached our target significance with only 12% of the data we collected. This translates to about 12 days of data collection, though we may want to make it a flat 2 weeks in order to evenly sample weekends, which may have lower sales volumes. \n",
    "\n",
    "There is some danger in shortening the collection period - if a major holiday falls during the collection period this may change the overall conversion rates. Longer collection periods tend to average out these sorts of short-term effects. \n",
    "\n",
    "In the future, this sort of analysis should be done before running an A/B test - once a level of significance is chosen, the number of samples required to statistically confirm/deny a given effect size can be derived and an appropriate-length test can be run. \n",
    "\n",
    "### Customer insights\n",
    "\n",
    "We'd like to leverage this data to derive some actionable insights about customer behavior that could increase the overall conversion rate. There are several possible avenues to explore here:\n",
    "\n",
    "* Do users who convert do something differently than users who don't? Is there something we can do to convert the users who don't buy the software?\n",
    "* Can we understand the demographics of users who tend to convert, in order to better target those kinds of users in the future? \n",
    "* Which customers do we lose by switching to a higher-priced product? Can we identify those customers and target them with promotions, etc. to increase their conversion rate?\n",
    "\n",
    "Answers to any one of the above questions would increase the overall conversion rate. Here, we focus on the third question: can we identify users we lose by increasing the software price in order to target them with customer retention efforts?\n",
    "\n",
    "We can look for users we've lost by selecting only the users that convert in the test and control groups, then looking at how the distributions of those users' associated features change. For starters, let's see if there are significant differences in operating systems between the control and test conversions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:4: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead\n",
      "  after removing the cwd from sys.path.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmAAAAD5CAYAAACAnYlyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmYXGWZ//93uhuy0IF2pANKGBGcuQcUg2wmGnaCgqjgMuPCT4EfexAhgAiKLMpu2MFBEA0DoiCCqIDCzCgQQHYMIjcEiBhBJwghCZGQ7fvHqYayTSedpM+p6s77dV25UlVnqbvq6VP1qec855xBixcvRpIkSdVpaXQBkiRJqxoDmCRJUsUMYJIkSRUzgEmSJFXMACZJklQxA5gkSVLF2hpdwPKYMWN2U58zo719MHPmzGt0GVpOtlv/Zdv1T7Zb/2S7Lb/OzuGDeppmD1gfamtrbXQJWgG2W/9l2/VPtlv/ZLv1LQOYJElSxQxgkiRJFTOASZIkVcwAJkmSVDEDmCRJUsUMYJIkSRUzgEmSJFXMACZJklSxfnUmfEmqQudFIyt/zhnjp1f+nJIaxx4wSZKkihnAJEmSKmYAG4Dmz5/PDTdct8LL33TTT9lzz936sCJJklRvwI8B22ri7ZU+331Hblvp8y3Jbbf9gkmTvsMee3y80aVIkqQlsAdsAFq8eHGjS5AkSUsx4HvAmt1zz/2Jc889i4ceepD29nb23PMTfO5z+/J///cXLrjgHO6//15aWgax0067MH784QwePJibbvopP/3pDWy99Wiuu+6HtLa2suOOu/CFLxzBww8/yKmnngTA2LFbcu21N3LKKSey4YYb8Zvf3M28efP43veu5rXX5vW4fkmSVC4DWAO99tprTJhwKG9/+0ZccsnlzJgxgxNOOJbOzhFMmnQ5I0eO5IILLmHWrJc5/fSvs3gxHHnkMQD8/ve/Y8SIEVx00WVMmfIIZ555CltvPZott9yaww47kquumsTll19JR8ebgGJc18SJFzJ48OqsscYaHHTQvktdvyRJKo8BrIHuv/9eXnhhBpdd9l+0t7ez4YbvYMKEY1i4cCEzZvyFb3/7u6y55loATJhwDMcccwQHHjgegIULF3L00V+hvb2dt71tA37842t4/PHHGDPm/bS3t9PS0sKb37z26881evT7GDVqMwDuvPPXy1y/JEkqjwGsgaZNe5r11luf9vb21x/bZZddufLK7zFy5PqvhyOATTd9NwsXLmT69GcBWGutjr9bbtiwNViwYEGPz7Xuum+te95nlrl+qSpVHygDzXGwjKRVm4PwG6itbbUlPr766v84DmvhwkV/9/9qq/3jsksbfL/66qsv1/olSVJ5DGANtP766/Pcc9N55ZU5rz/2ne9cwlVXTWL69D8ya9bLrz/+u9/9ltbWVkaOXPYlUgYNGrTU6Rts8PaVWr8kSVo5BrAG2nrrMYwYsQ5nnHEK06Y9w913T+baa3/AYYdNYP3138bXv/41pk59kgcfvJ9zz/0mO+20C2ut1bHM9Q4dOpQ5c+bw7LN/WOJuyS233Hql1i9JklZOKWPAIqIFuBgYBcwD9svMqXXTxwN7A4uBkzPzZxExCJgOPFmb7e7MPHZla6lyrEdHxzBmzpzb6/lbW1s57bSJnH32Gey771686U1vYp999mOnnXZhk03exTnnnMmBB+7N0KHD2GWXD3LggYf2ar2bb74Vb3vbBuy996e5+OLv/MP0lpYWTjvtmyu8fkmStHIGlXHSzoj4GPCRzNw7IkYDx2bmR2vT1gZ+DWwGDAEeA/4Z2Ag4JzM/3NN6Z8yY3dRnGF3eAKbmYLs1VjMOwu+8qPpd8TPGT6/8ORvFba5/st2WX2fn8B7HBJW1C3IscAtAZt4DbNk1ITNfAEZl5nxgXWBmZi4GtgDWi4j/jYibIiJKqk2SJKmhyjoNxZrAy3X3F0ZEW2YuAMjMBRFxKHAScH5tnueB0zLz2ogYC1wJbFW/0vb2wbS1tZZU8sprbW2ho2NYo8vQcrLdVj3N2N7NWFNZ3Ob6J9utb5UVwGYBw+vut3SFry6ZeWFEfBu4OSJ2AH4DdAW0OyNivYgYVOsdA2DOnHkllds37J7tn2y3Vc+y2ruzojrqrUp/g25z/ZPttvw6O4f3OK2sXZCTgd0AamPApnRNiMKPa4Pu51MM0l8EnAAcXptnFPBsffiSJEkaKMrqAbseGBcRdwGDgH0iYgIwNTNvjIhHgLspjoK8OTN/HRG/Ba6MiA9R9ITtXVJtkiRJDVVKAMvMRcBB3R5+vG76SRTjv+qXeQn4UBn1SJIkNRNPxCpJklQxA5gkSVLFDGCSJEkVM4A12JNPJo888tBKrWPx4sXccMOPWLRoUR9VJUmSylTWUZBNo/JLiizn5USOO+5oPve5fRk16j0r/JQPP/wg3/zm6ey++x60tJipJUlqdn5bN1hfXIuzjOt5SpKk8hjAGujQQw/gz39+njPO+AannHIiTz/9FIcddhA77vh+/uM/9uDqq698PVy98socvva1Y9l11x3ZZZftOO64o/nrX1/g+eef47DDijN+bL/9aB588P5GviRJktQLBrAGOvXUsxgxYh0OPfRwvvjFozjqqMN45zs3ZdKkqzn88KO59tqrue66HwJw6aX/yfPPP8eFF36bSy75Li+99CLnn382I0aswymnnAnA9dffxKabjmrkS5IkSb0w4MeANbM111yLlpYW1lijnV/96jaGD1+TAw8cD8D66/8z++9/MN/97qV84hOf4s9/fo5hw4bxlre8lWHDhnH88Scze/ZsWltbGT58TQDe9KZ/oq3NJpUkqdn5bd0kpk2bxrRpTzNu3DavP7Zo0SLmz5/P/Pnz2WuvvTnqqC+y++7j2HzzLdh22x344Ae9cIAkSf2RAaxJLFy4kM0224Kjjz72H6a1trbyrne9m+uu+ymTJ9/B3XdP5sILz+XWW2/hggsuaUC1kiRpZTgGrMEGDRoEwD//89v44x//wLrrvoWRI9dn5Mj1efLJ5KqrJtHS0sI113yfxx57lF122ZUTTvgGZ511Hg899AAvvvjX19chSZL6BwNYgw0dOpQ//GEaY8a8n/nz53PGGd9g2rRnuO++ezj77DNfH9/1l7/8hXPOOYspUx7hT3+azq233sw666zLWmt1MHToMACeeOJx5s2b18iXI0mSemHA74KcsZwnRl0ZHR3DYObc5VrmYx/7dy666Dyef/5PTJx4Pueffzb77rsXw4cPZ9ddd+eAAw4BYP/9D2bu3Fc49tijmDt3Lpts8k7OPPNcWltb2Wijd7D11mMYP35/TjzxVLbbbocyXp4kSeojg/rTSTxnzJjd1MV2dAxj5nIGMDWe7dZYW028vfLnvO/IbZc6vfIraFDtj8VGc5vrn2y35dfZObzHMULugpQkSaqYAUySJKliBjBJkqSKGcAkSZIqVspRkBHRAlwMjALmAftl5tS66eOBvYHFwMmZ+bOIGApcCYwAZgOfz8wZZdQnSZLUSGX1gO0BDMnMMcCXgYldEyJibeAQ4H3ATsC3ImIQcDAwJTO3Aa4AvlpSbZIkSQ1VVgAbC9wCkJn3AFt2TcjMF4BRmTkfWBeYmZmL65cBbgZ2Lqk2SZKkhirrRKxrAi/X3V8YEW2ZuQAgMxdExKHAScD5S1hmNrBW95W2tw+mra21pJJXXmtrS3EyVvUrttuqpxnbuxlrKovbXP9ku/WtsgLYLGB43f2WrvDVJTMvjIhvAzdHxA7dlhkOzOy+0jlzmvsyO56krn+y3VY9y2rvzorqqLcq/Q26zfVPttvy6+wc3uO0snZBTgZ2A4iI0cCUrglR+HFt3Nd8ikH6i+qXAXYF7iipNkmSpIYqqwfsemBcRNwFDAL2iYgJwNTMvDEiHgHupjgK8ubM/HVE3AdMiog7gdeAz5RUmwaoZrukTSMuZwOr1iVtJKm/KiWAZeYi4KBuDz9eN/0kivFf9cvMBT5ZRj2SJEnNxBOxSpIkVcwAJkmSVDEDmCRJUsUMYJIkSRUzgEmSJFXMACZJklQxA5gkSVLFDGCSJEkVM4BJkiRVzAAmSZJUMQOYJElSxQxgkiRJFTOASZIkVcwAJkmSVDEDmCRJUsUMYJIkSRUzgEmSJFXMACZJklQxA5gkSVLF2spYaUS0ABcDo4B5wH6ZObVu+hHAp2p3b8rMkyJiEDAdeLL2+N2ZeWwZ9UmSJDVSKQEM2AMYkpljImI0MBH4KEBEbAh8FngvsBi4IyKuB+YCD2bmh0uqSZIkqSmUtQtyLHALQGbeA2xZN+2PwAczc2FmLgJWA14FtgDWi4j/jYibIiJKqk2SJKmhyuoBWxN4ue7+wohoy8wFmTkfeKG2y/Es4KHMfCIi1gVOy8xrI2IscCWwVf1K29sH09bWWlLJK6+1tYWOjmGNLkMVasb2bsaamk0zvkfNWFNZ/Kzsn2y3vlVWAJsFDK+735KZC7ruRMQQ4HJgNnBI7eH7gQUAmXlnRKwXEYMyc3HXcnPmzCup3L7R0TGMmTPnNroMVWhp7d1ZYR31/BtctmW9R41ou1Wp3fys7J9st+XX2Tm8x2ll7YKcDOwGUBsDNqVrQq3n6yfAI5l5YGYurE06ATi8Ns8o4Nn68CVJkjRQlNUDdj0wLiLuAgYB+0TEBGAq0ApsBwyOiF1r8x8LnA5cGREfougJ27uk2iRJkhqqlABWG1x/ULeHH6+7PaSHRT9URj2SJEnNxBOxSpIkVcwAJkmSVDEDmCRJUsUMYJIkSRUzgEmSJFXMACZJklQxA5gkSVLFDGCSJEkVM4BJkiRVzAAmSZJUsWVeiigi1gc+Td3lgzLz5DKLkiRJGsh60wN2LbAm8Je6f5IkSVpBvbkY9+zM/GrplUiSJK0iehPAHo2ITwEPAYsBMvOJUquSJEkawHoTwDar/euyGNixnHIkSZIGvmUGsMzcISLeDGwEPJ2ZL5RfliRJ0sC1zEH4EfFJ4C7gOOCeiNir9KokSZIGsN4cBTkB2CIz9wDeA3yx3JIkSZIGtt4EsEWZOQcgM2cDr5ZbkiRJ0sDWm0H4T0XEROB2YFvgqXJLkiRJGth6E8D2BQ4ExgGPAV9e1gIR0QJcDIwC5gH7ZebUuulHAJ+q3b0pM0+KiKHAlcAIYDbw+cycsRyvRZIkqV/ocRdkRGxZu7kj8CRwIzAV2KEX690DGJKZYygC28S69W4IfBZ4HzAG2CUi3g0cDEzJzG2AKwBP/ipJkgakpfWA7QTcT3EdyHqLgV8uY71jgVsAMvOeujAH8Efgg5m5ECAiVqMYVzYWOLM2z83A8b15AZIkSf1NjwEsM8+o3ZycmZd1PR4Rh/VivWsCL9fdXxgRbZm5IDPnAy9ExCDgLOChzHwiIuqXmQ2s1X2l7e2DaWtr7cXTN0ZrawsdHcMaXYYq1Izt3Yw1NZtmfI+asaay+FnZP9lufavHABYRnwY+AuwQEV1nvm8F3gWcv4z1zgKG191vycwFdeseAlxOEbQOWcIyw4GZ3Vc6Z868ZTxtY3V0DGPmzLmNLkMVWlp7d1ZYRz3/BpdtWe9RI9puVWo3Pyv7J9tt+XV2Du9x2tJ2Qd4CPA+8GfhPYBCwiN4dBTkZ+DBwTUSMBqZ0Taj1fP0E+J+6XrauZXYD7gV2Be7oxfNIkhpgq4m3N+R57zty24Y8r9TXlrYL8iXgVxExHdgqM6+OiNMpwtiyXA+Mi4i7KILbPhExgWIQfyuwHTA4InatzX8s8C1gUkTcCbwGfGZFX5QkSVIz681pKCYBX6ndvgn4DsUA/R5l5iLgoG4PP153e0gPi36yF/VIkiT1a705Ez6Z+ava/7f3dhlJkiQtWW96wGZGxAHA3cDWFAPnJUmStIJ6E8D2ptgFuSfFmfD3LbMgSZJWROdFIyt/zhnjp1f+nBoYlhnAMnNGRJwBrEYxoP5fgBfKLkySJGmgWmYAi4jLgdHAGsBQ4OnafUmSJK2A3gyo/zfgncAvgE0oLhskSZKkFdSbADY7MxcDa2TmC8DqJdckSZI0oPUmgD0QEUcBz0XEDyhOpCpJkqQV1NsTsT4H/I3iEkH3llqRJEnSANebAPadzBxbu/3TMouRJElaFfQYwCJircx8GXglIs4BkuJi3GTmtyuqT5IkacBZ2hiwrt6uZ4CXgBHAurV/kiRJWkFL2wX5t4i4j+LEq7+ve3wxcHKpVUmSJA1gSwtguwJvBS4BDqmmHEmSpIGvxwCWmYuA6cCHqitHkiRp4OvNecAkSZLUhwxgkiRJFTOASZIkVcwAJkmSVLHenAl/uUVEC3AxMAqYB+yXmVO7zdMJ3AVsmpmvRsQgikH/T9ZmuTszjy2jPkmSpEYqJYABewBDMnNMRIwGJgIf7ZoYER8ATgfWqVtmI+DBzPxwSTVJkiQ1hbJ2QY4FbgHIzHuALbtNXwTsDLxY99gWwHoR8b8RcVNEREm1SZIkNVRZPWBrAi/X3V8YEW2ZuQAgM28F6JaxngdOy8xrI2IscCWwVf0M7e2DaWtrLankldfa2kJHx7BGl6EKNWN7N2NNzaYZ36NmrKkZNdv71Gz1lMnvuL5VVgCbBQyvu9/SFb6W4n6gK6DdGRHrRcSgzFzcNcOcOfP6vtI+1NExjJkz5za6DFVoae3dWWEd9fwbXLZlvUeNaDvbrXeabZtbldrN77jl19k5vMdpZe2CnAzsBlAbAzalF8ucABxeW2YU8Gx9+JIkSRooyuoBux4YFxF3AYOAfSJiAjA1M2/sYZnTgSsj4kMUPWF7l1SbJElSQ5USwGrXkTyo28OPL2G+Depuv4TXnZQkSasAT8QqSZJUMQOYJElSxQxgkiRJFTOASZIkVcwAJkmSVDEDmCRJUsUMYJIkSRUzgEmSJFXMACZJklQxA5gkSVLFDGCSJEkVM4BJkiRVzAAmSZJUMQOYJElSxQxgkiRJFTOASZIkVcwAJkmSVDEDmCRJUsUMYJIkSRVrK2OlEdECXAyMAuYB+2Xm1G7zdAJ3AZtm5qsRMRS4EhgBzAY+n5kzyqhPkiSpkcrqAdsDGJKZY4AvAxPrJ0bEB4BfAuvUPXwwMCUztwGuAL5aUm2SJEkNVUoPGDAWuAUgM++JiC27TV8E7Aw80G2ZM2u3bwaOL6m2pdpq4u2NeFruO3LbHqd1XjSywkreMGP89IY8ryRJA11ZPWBrAi/X3V8YEa+Hvcy8NTP/upRlZgNrlVSbJElSQ5XVAzYLGF53vyUzFyzHMsOBmd1naG8fTFtba99U2GQ6OoY1uoR/0Iw1NZtmfI+asaZm04zvUTPW1Iya7X1qtnrK1Nraskq93rKVFcAmAx8GromI0cCUXi6zG3AvsCtwR/cZ5syZ15c1NpWZM+f2OK2zwjrqLa0mFWy3/mlZ71Ej2s52651m2+ZWpXbr6Bi2Sr3evtDZObzHaWUFsOuBcRFxFzAI2CciJgBTM/PGHpb5FjApIu4EXgM+U1JtkiRJDVVKAMvMRcBB3R5+fAnzbVB3ey7wyTLqkSRJaiaeiFWSJKliBjBJkqSKGcAkSZIqZgCTJEmqmAFMkiSpYgYwSZKkihnAJEmSKmYAkyRJqpgBTJIkqWIGMEmSpIoZwCRJkipmAJMkSaqYAUySJKliBjBJkqSKGcAkSZIqZgCTJEmqmAFMkiSpYgYwSZKkihnAJEmSKmYAkyRJqlhbGSuNiBbgYmAUMA/YLzOn1k3fHzgQWAB8IzN/FhH/BDwBPFqb7frMPK+M+iRJkhqplAAG7AEMycwxETEamAh8FCAi1gUOA7YEhgB3RsStwObA1Zn5hZJqkiRJagpl7YIcC9wCkJn3UIStLlsDkzNzXma+DEwF3g1sAWweEb+OiGsj4i0l1SZJktRQZfWArQm8XHd/YUS0ZeaCJUybDawFPA48kJm3RcRngQuAT9SvtL19MG1trSWV3FgdHcMaXcI/aMaamk0zvkfNWFOzacb3qBlrakbN9j41Wz1lam1tWaVeb9nKCmCzgOF191tq4WtJ04YDM4HfAHNrj10PnNx9pXPmzOv7SpvEzJlze5zWWWEd9ZZWkwq2W/+0rPeoEW1nu/VOs21zq1K7dXQMW6Veb1/o7Bze47SydkFOBnYDqI0Bm1I37V5gm4gYEhFrARtTDLy/DPh4bZ6dgAdKqk2SJKmhyuoBux4YFxF3AYOAfSJiAjA1M2+MiPOBOygC4Fcy89WI+DJweUQcArwC7FdSbZIkrbK2mnh75c9535HbVv6cza6UAJaZi4CDuj38eN30S4FLuy3zDLBDGfVIkiQ1E0/EKkmSVDEDmCRJUsUMYJIkSRUraxC+JEnSMnVeNLIhzztj/PSGPG8Xe8AkSZIqZgCTJEmqmAFMkiSpYgYwSZKkihnAJEmSKmYAkyRJqpgBTJIkqWIGMEmSpIoZwCRJkipmAJMkSaqYAUySJKliBjBJkqSKGcAkSZIqZgCTJEmqmAFMkiSpYm1lrDQiWoCLgVHAPGC/zJxaN31/4EBgAfCNzPxZRKwNfB8YCjwH7JOZc8uoT5IkqZHK6gHbAxiSmWOALwMTuyZExLrAYcD7gQ8Ap0XEYOBrwPczcxvgIYqAJkmSNOCUFcDGArcAZOY9wJZ107YGJmfmvMx8GZgKvLt+GeBmYOeSapMkSWqoQYsXL+7zlUbEZcB1mXlz7f6zwIaZuSAi9gI2zcxjatOuAK4A/rP2+N8iYkPgiswc2+fFSZIkNVhZPWCzgOH1z5OZC3qYNhyY2e3xrsckSZIGnLIC2GRgN4CIGA1MqZt2L7BNRAyJiLWAjYFH65cBdgXuKKk2SZKkhiprF2TXUZDvBgYB+1CEq6mZeWPtKMgDKALgqZl5XUSsA0yi6P16AfhMZr7S58VJkiQ1WCkBTKpSRCwGxmXmbSuw7DeAsZm5fZ8XphUSERsAzwD/kplTI2I4cBzwH8BbgeeBH1P8ePtr3XJtFEdd7w2sT/FD7ufAVzPz/6p8Dfp7tbaZD+yQmb9ageVPBHbuaVxwRNwJ3JaZJ65EmauEiBgBnAh8BHgz8CzFKaDOzMy/1ebZAZiRmY9GxN4Up4sa2ZiKBy5PxKqB4C3A7Y0uQn3mjxRt+kxEtFO07a7AF4AA9qM4mvqe2mltupwGfBo4GPhX4FPApsDNETGouvJVgm9SBAathIhYj2IY0MbAXrX/jwY+Afw6IobVZv0fYN0lrkR9ppQTsUpVysw/N7oG9Z3MXAj8GSAivg6sAWyemXNqs/yh1uNxN3AORegC2Bc4MDNvrZvvs8BTwHuBeyp6CepjdW2vlXMhxQ+ccXUHxk2LiNuB3wHHA8c2qrhVjQFsKep2hXwUOB9YG7gE+F7t378B/03xBfAacGrt9joUZ/M/PTO/VVvXMOBMit0orcDPgPGZObuq19PfRMQjwOWZeV7t/g3A2zNzVO3+x4HTgXdQ2wUZEdMofi1/BtiM4gCP8Zl5X22ZTYBvA5tTHPgxtdtzjgHOAt4DzADOysyLImJP4DKgMzMXRcRmFCcM/lhmXl9bdkqtnp8ClwLjKLax24BDDIq9U7fd/RtFqDqu+xdwZs6LiNOA70fE+Mx8EVgM7BgR19dCHJn5dES8s7Y+Lafa9nAmsAXF+3sH8P9T/G3vB/ySomdyAfBDYEJmLqot+zXgUIpxwMd2W+804BqKXpiZFOOF/4UiUL8PmEOxnZ5c295OpG4XZG17PANYj2K7dG/OMtR2PX4E+Ehd+AIgM2dGxLnA0bVTRQHcGhEnAdNqyx9PcRL11YDvAEdl5uLatAModv+PAB4Gjqj7zJ1Gt7bu/vyrKv9oe+fLFH+4BwETgB8BX6LYLbIdxZfEMbV5PkGxm+R7wAUR8dbaOi4BdgI+BuwAvAs4u7JX0D/9guK9orYLaRvgnbWjZ6E4We/NS1juBIovjdHAqxS/+qhdceHnFF/GmwPXA/t3LRQRG1N0vd9OEcBOAM6MiE9SBO01Kb4ooGj3xRRXdOi6wsPGFF9IXwc2qM0zmuJD6ZyVeB9WVatTvOf39jD9DoqAu0Xt/nkUux//EBGXRsSnIqIjMx/rGtui3quNvfs5xQ+IdwK7ABsCX6nNsjWwCcV2+RWKIPaB2rIHAIdTfDaOq/3f3f8HfJDix1IHRXs+R9FbeTAwHjhyCXVtQvGF/i2Kth8CjFnJl7sq2ILiO39p21MnxXcUwL9T/JiFIui+i+KE6QcCRwAfAoiID1N85h1B8bl5M/A/EfGWunW/3taGrzcYwHrnG5n528y8EngR+EFm/ndm3g78iuKX+qMU17y8JzOfpugNawWiFhg+BXwhM+/IzEcowtwTjXgx/cgvgG1rR9VuSvHeP0URaqD4YL9lCctdkZk3ZOZvKT5Auq7EsDPFB8zBmfl4Zl4M3FC33P7AbzPzuMx8IjMnARcAX8rMWRS7sLavzbsdxQfN++vW/WBmzqAIX3OAZzLzMYoPn7NW/G1YZf1T7f+Xepje9fjaAJn5dYrt7CmKI6+vBp6PiKPLLHIAW4Pic+zkzHwmMycD11GEMSjC74G1bely4BFgq9q0/YHzM/NnmfkwxVHv3V1V+1x9mCKEvVpb3+8z8ycUu8O+tITl9qG4mso5mfk4RS+bvcvL1tvtqWvP2Et1Pc8LgP2z8EOKth5Vm/Ylir09P8nMJzPzFOB+ih7SLvVtrRoDWO88XXf7b8Afut0fnJk3AEMiYmJE/Jxaty1FCPtXij/qB7oWysx7M9Mv5aW7ExhM0eu0HcUvtHuAsRHxdopfZb9awnJP1d2eBbRERCvFr/Wnuu3Our/u9sbAb7qt6y6KgA1FINy+rjfuLGDziBhKEQa7euNOp+hhmxERN1GEs9/18jXrDS/W/u9pMHBX7/LrR0Jm5g8zczuKUPZJir+hMyPio6VVOUDVdpl/DzgiIq6IiPuBoyg+0wBeqF1Orsssit1TUGxrj9St61GKgFVvWt3tjSl+wMyve+wuYO2IWLvbct3XPb/+vnq03NtTnRdqP0K7vEzR8whF250aEXO6/lH0lP1r3fzTVqzkgc0A1jvdu0wXdZ+hdjqD79fm/S/e6KWBYnyYllNmzqMIWNvzRgC7g6LXaRxwe2bOXcKiS3q/B3X7v0v6t2TMAAAD+klEQVT9B373Lwgovmy6fhH+AtiWojdubu1w+hkUu2J25o3rn94NvI3iF+BLFBejX1JPnZbubxRfBlv2MH0riu3twYh4d0Sc1zUhM2dm5o8odpvdT/H3ouVQO2JuCsXf9gMUu5gm1s2ytO2s+234x8/RV3u43aW12/89PQ/8/XasJbuPog2Wtj29wN93OHRZuITHutqgjWJX8WZ1/7qOruyypPZd5RnA+s5BwGGZeUxm/oCi+x6KP9KnKf6A39M1c0TsHBFP1HavqWdd48DG8kYAey+wO0se/7U0jwLviIg31T32nrrbv6+tu94YIGu3H6Box0N540oNd1C0/RBqYysi4nDgvZl5VWZ+luIkxNvXTjas5XMp8KXaeKTXRcRqwFeBH2fmCxRfAofVrrzxutog4ZcpgrKWz57ArMzcLTPPy8w7KMaA9eaUHo/yxu5IIuIdQPtS5v89RW/yanWPjaHotenedt3X3cobYzPVg9p28iPghNp52V5XGyYzgeKgp+Udo5XA+pk5tesfRSDbvg/KHtA8CrLv/BXYPSJ+Q3EOo/Nrjw/OzNkR8V3gvNpVAOZRDBL/764jhtSjX1Ds6nuxtmETEbMpAtiSxocszW0Uu48vj4ivUPRSfoI3Tk9wMXB4RJxKsetlNMVA4C8C1I7Gug34PMXRQFAM2L8IuKbryDuKk4AeFBH7UoxN+SzFyQ5fWM56VQzu3QG4PSKOo9iV+3aKE0muwRtt82BE/Az4cUQcS9Eub6YIEZtRnJxVy+evwHoRMY5it/4ngY9THP27LBcCF0fEgxTh6gKWsOegzveBk4BLIuIsiiMiTwK+Vdvu6ue9DPhi7SjLHwKHAJ4ktHeOoPjR+MvaEY7PUIzpO43i9BQn1eabQ3HA0329WOfZFJ+pj1Ps8t+L4qCLS/q49gHH3pe+sy/FrqnfAVcA11J8sXf1sBxB0UNyC3ArxW6Ro6ovs3/JzKQ4Mqr+2qB3AM/WBuAuz7rmU/RGrUnRm3UARejqmj6d4sieD1DsejkeODIzL6tbzS8ojs7rqud2ih6B+l2Mx9em30Dx97AxsHtdQFMv1XYx7wj8hOIox6S4ZNlDwFbdTu3x7xQ9ZscCj1EE7ncB29baVsvnGorhFNdQbC87UXyOBTB0aQtm5n8BX6Noszsojqbs8ZQ7tXGZHwQ2omjbi2rLfm0J8z4JfJiivR+mGO/nLv5eqG0vYyjet0kU29O5FEeEb1s3pOMcirGsJ/RinT+kOFPACRSfd3sCezjgftm8FJEkSVLF7AGTJEmqmAFMkiSpYgYwSZKkihnAJEmSKmYAkyRJqpgBTJIkqWIGMEmSpIoZwCRJkipmAJMkSarY/wNWGN//5rpIZAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a0cd5fc50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# positive cases only\n",
    "converted_control = control_group[control_group['converted']==1]\n",
    "converted_test = test_group[test_group['converted']==1]\n",
    "\n",
    "onehot_opsys_control = onehot_encode_function(converted_control['operative_system'])\n",
    "onehot_opsys_test = onehot_encode_function(converted_test['operative_system'])\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10,4))\n",
    "plt.bar(np.arange(0, 15, 3), np.sum(onehot_opsys_control, axis=0)/np.sum(onehot_opsys_control), label='control')\n",
    "plt.bar(np.arange(1, 16, 3), np.sum(onehot_opsys_test, axis=0)/np.sum(onehot_opsys_test), label='test')\n",
    "plt.xticks(np.arange(0, 15, 3),list(opsys_cats.keys()), fontsize=14)\n",
    "plt.legend(loc=2, fontsize=14)\n",
    "plt.ylabel('fraction')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There doesn't seem to be much of a pattern here - maybe a higher fraction of users on iOS convert relative to Android when the price is higher, but it's not a large effect. \n",
    "\n",
    "Now let's look at the sources of converted users:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:4: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead\n",
      "  after removing the cwd from sys.path.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABP0AAAD9CAYAAADH7AJ+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xm4HUWZ+PFvcsMWkhBHAyhBEMd5FQWUTaKIskVQUfCn4wIqKAgSjLKpoCMugGwBBAICiqAgIw6DoiIKbuwKsggor0SIEEAmoAECEkKS3x/VN5xc7prcczvp+/08T57c0326q07V6a4+b1dVj1i0aBGSJEmSJEmSmmNk3RmQJEmSJEmSNLgM+kmSJEmSJEkNY9BPkiRJkiRJahiDfpIkSZIkSVLDGPSTJEmSJEmSGsagnyRJkiRJktQwo9qx04gYCZwObALMA/bOzBkt6w8E3l+9vCwzvxwRI4BZwN3V8usz87B25E+SJEmSJElqsrYE/YBdgVUzc1JEbAVMA94FEBEbALsDrwcWAVdHxCXAU8DNmblLm/IkSZIkSZIkDQvtGt67NXA5QGbeAGzesu5+YKfMXJCZC4GVgKeBzYB1IuLXEXFZRESb8iZJkiRJkiQ1WruCfuOAx1peL4iIUQCZOT8zH4mIERFxAnBLZv4FeAj4WmZuCxwNnN+mvEmSJEmSJEmN1q7hvY8DY1tej8zMZztfRMSqwDnAE8D+1eKbgGcBMvOaiFgnIkZk5qLO7f71r2cWjRrV0aYsr5g6OkawYMGivt+otrD862X518vyr591UC/Lv16Wf70s/3pZ/vWzDupl+dfL8l/+rLRSx4julrcr6HctsAtwUTWn3+2dK6oHdvwI+FVmHtuyzRHAo8BxEbEJcF9rwA9g7tx5bcruimv8+NHMmfNU3dkYtiz/eln+9bL862cd1Mvyr5flXy/Lv16Wf/2sg3pZ/vWy/Jc/EyaM7XZ5u4J+lwA7RsR1wAhgr4g4CJgBdABvBlaJiJ2r9x8GHAOcHxFvp/T427NNeZMkSZIkSZIarS1Bv+oBHft1WXxXy9+r9rDp29uRH0mSJEmSJGk4adeDPCRJkiRJkiTVxKCfJEmSJEmS1DAG/SRJkiRJkqSGMegnSZIkSZIkNYxBP0mSJEmSJKlhDPpJkiRJkiRJDTOq7gxo+TFh+sQhSWf2lFlDko4kSZIkSdJwZU8/SZIkSZIkqWEM+kmSJEmSJElLaf78+fzwhxcv9faXXfZjdtvtbYOYo8KgnyRJkiRJkrSUrrzy55x33rfqzsbzOKefJEmSJEmSltkW064a0vRuPHibIU2vJ4sWLao7C90y6CdJkiRJkqRh4cEHH+Dkk4/nlltuZsyYMey223v48Ic/yv/938OceupJ3HTT7xk5cgTbbz+ZKVM+zSqrrMJll/2YH//4h2y55VZcfPH36ejoYLvtJvPJTx7IrbfezNFHfxmArbfenB/84FKOOupLbLDBy/nd765n3rx5nHvuhTzzzLwe998uBv0kSZIkSZLUeM888wwHHXQAL3vZyznzzHOYPXs2RxxxGBMmrMl5553DxIkTOfXUM3n88cc45pivsmgRHHzwZwH485/vZM0112T69G9y++23cdxxR7Hlllux+eZbMnXqwVxwwXmcc875jB//AqDM0zdt2mmsssrKrL766uy330d73X87GPSTJEmSJElS49100+955JHZfPOb32XMmDFssMG/c9BBn2XBggXMnv0wZ531bcaNWwOAgw76LJ/97IHsu+8UABYsWMChh36eMWPGsN566/O//3sRd931JyZNeiNjxoxh5MiRvPCFL1qc1lZbvYFNNnktANdc89s+998OPshDkiRJkiRJjTdz5j2ss866jBkzZvGyyZN35tFHH2HixHUXB+QANtpoYxYsWMCsWfcBsMYa45fYbvTo1Xn22Wd7TGvttV/Sku69fe6/HQz6SZIkSZIkqfFGjVqp2+Urr/z8efUWLFi4xP8rrfT8bXt7gMfKK688oP23g0E/SZIkSZIkNd66667Lgw/O4skn5y5e9q1vnckFF5zHrFn38/jjjy1efuedf6Sjo4OJEyf2ud8RI0b0un799V+2TPtfWgb9JEmSJEmS1HhbbjmJNddci2OPPYqZM+/l+uuv5Qc/+G+mTj2Iddddj69+9YvMmHE3N998EyeffALbbz+ZNdYY3+d+V1ttNebOnct99/2t2yG/m2++5TLtf2n5IA9JkiRJkiQ1XkdHB1/72jROPPFYPvrRPXjBC17AXnvtzfbbT2bDDV/DSScdx7777slqq41m8uSd2HffA/q130033YL11lufPff8AKef/q3nrR85ciRf+9oJS73/pTWit/HHy5vZs59YcTI7RMaPH82cOU8Nyr4mTG9fl9JWs6fMGpJ0hsJglr8GzvKvl+VfP+ugXpZ/vSz/eln+9bL862cd1Mvyr5flv/yZMGFst+OLHd4rSZIkSZIkNYxBP0mSJEmSJKlhDPpJkiRJkiRJDWPQT5IkSZIkSWoYg36SJEmSJElSwxj0kyRJkiRJkhrGoJ8kSZIkSZLUMAb9JEmSJEmSpIYZVXcG9Jwtpl01JOncePA2Q5KOJEmSJEmS6mFPP0mSJEmSJDXe3Xcnt912yzLtY9GiRfzwh//DwoULBylX7WPQT5IkSZIkSY13+OGHct99f1umfdx6682ccMIxK0TQz+G9kiRJkiRJWmYTpk8c0vRmT5k1oPcvWrRomdMcjH0MFXv6SZIkSZIkqdEOOODj/P3vD3HssUdy1FFf4p57/srUqfux3XZv5H3v25ULLzx/cUDvySfn8sUvHsbOO2/H5Mlv5vDDD+XRRx/hoYceZOrU/QB4y1u24uabb6rzI/XJoJ8kSZIkSZIa7eijj2fNNdfigAM+zac+dQiHHDKVV796I84770I+/elD+cEPLuTii78PwNlnf4OHHnqQ0047izPP/Db//Oc/OOWUE1lzzbU46qjjALjkksvYaKNN6vxIfXJ4ryRJkiRJkhpt3Lg1GDlyJKuvPobf/OZKxo4dx777TgFg3XVfyj77fIJvf/ts3vOe9/P3vz/I6NGjefGLX8Lo0aP5r//6Ck888QQdHR2MHTsOgBe84N8YNWr5Dqst37mTJEmSJEmSBtHMmTOZOfMedtzxTYuXLVy4kPnz5zN//nz22GNPDjnkU7zjHTuy6aabsc0227LTTm+vMcdLpy1Bv4gYCZwObALMA/bOzBkt6w8E3l+9vCwzvxwRqwHnA2sCTwAfyczZ7cifJEmSJEmShqcFCxbw2tduxqGHHva8dR0dHbzmNRtz8cU/5tprr+b666/ltNNO5oorLufUU8+sIbdLr11z+u0KrJqZk4DPAdM6V0TEBsDuwBuAScDkiNgY+ARwe2a+CfgO8IU25U2SJEmSJEnDzIgRIwB46UvX4/77/8baa7+YiRPXZeLEdbn77uSCC85j5MiRXHTR9/jTn+5g8uSdOeKIIzn++K9zyy1/4B//eHTxPlYE7Qr6bQ1cDpCZNwCbt6y7H9gpMxdk5kJgJeDp1m2AnwE7tClvkiRJkiRJGmZWW201/va3mUya9Ebmz5/PscceycyZ93LjjTdw4onHLZ6v7+GHH+akk47n9ttv44EHZnHFFT9jrbXWZo01xrPaaqMB+Mtf7mLevHl1fpw+tWtOv3HAYy2vF0TEqMx8NjPnA49ExAjgeOCWzPxLRLRu8wSwRtedjhmzCqNGdbQpy8PH+PGjh3X6g6mjY2SjPs+KxvKvl+VfP+ugXpZ/vSz/eln+9bL862cd1Mvyr9fyVP4Dzccee+zBCScczyOPPMxZZ53Fsccey8c+tgfjxo1jt912Y+rUTzFq1CgOOeQgjjlmHp///KE8+eSTbLTRxpxxxjd44QvHMm7cRrzxjVszZco+HH/8NHbYYfnts9auoN/jwNiW1yMz89nOFxGxKnAOJbi3fzfbjAXmdN3p3LnLdwR1RTFnzlPdLp9Qc/p122LaVUOSzo0HbzMk6QwH48ePXm6/T8OB5V8/66Beln+9LP96Wf71svzrZx3Uy/KvV6/lP2XW0GZmgN+DyZN3YfLkXRa/PvnkM5ZYP3fuM8AzABx44Oc48MDPLZlcld6xx578vGV1mjBhbLfL2zW891rgbQARsRVwe+eKqoffj4DbMnPfzFzQdRtgZ+DqNuVNkiRJkiRJarR29fS7BNgxIq4DRgB7RcRBwAygA3gzsEpE7Fy9/zDgDOC8iLiGElb9YJvyJkmSJEmSJDVaW4J+1QM69uuy+K6Wv1ftYdP3tiM/kiRJkiRJ0nDSruG9kiRJkiRJkmpi0E+SJEmSJElqGIN+kiRJkiRJUsMY9JMkSZIkSZIaxqCfJEmSJEmS1DAG/SRJkiRJkqSGMegnSZIkSZIkNYxBP0mSJEmSJKlhDPpJkiRJkiRJDWPQT5IkSZIkSWqYUXVnQFIxYfrEIUln9pRZQ5KOJEmSJEmqjz39JEmSJEmSpIYx6CdJkiRJkiQ1jEE/SZIkSZIkqWEM+kmSJEmSJEkNY9BPkiRJkiRJahiDfpIkSZIkSVLDGPSTJEmSJEmSGsagnyRJkiRJktQwBv0kSZIkSZKkhjHoJ0mSJEmSJDWMQT9JkiRJkiSpYQz6SZIkSZIkSQ1j0E+SJEmSJElqGIN+kiRJkiRJUsMY9JMkSZIkSZIaxqCfJEmSJEmS1DAG/SRJkiRJkqSGMegnSZIkSZIkNYxBP0mSJEmSJKlhDPpJkiRJkiRJDWPQT5IkSZIkSWoYg36SJEmSJElSwxj0kyRJkiRJkhrGoJ8kSZIkSZLUMKPasdOIGAmcDmwCzAP2zswZXd4zAbgO2Cgzn46IEcAs4O7qLddn5mHtyJ8kSZIkSZLUZG0J+gG7Aqtm5qSI2AqYBryrc2VEvBU4BlirZZuXAzdn5i5typMkSZIkSZI0LLRreO/WwOUAmXkDsHmX9QuBHYB/tCzbDFgnIn4dEZdFRLQpb5IkSZIkSVKjtSvoNw54rOX1gohY3KswM6/IzEe7bPMQ8LXM3BY4Gji/TXmTJEmSJEmSGq1dw3sfB8a2vB6Zmc/2sc1NwLMAmXlNRKwTESMyc1HnG8aMWYVRozoGP7fDzPjxo4d1+nWr+/PXnf5g6ugY2ajPs6Kx/OtnHdTL8q+X5V8vy79eln/9rIN6Wf71svxXHO0K+l0L7AJcVM3pd3s/tjkCeBQ4LiI2Ae5rDfgBzJ07b9AzOhzNmfNUt8sn1Jz+cGH5D57x40c36vOsaCz/+lkH9bL862X518vyr5flXz/roF6Wf70s/+XPhAlju13erqDfJcCOEXEdMALYKyIOAmZk5qU9bHMMcH5EvJ3S42/PNuVNkiRJkiRJarS2BP0ycyGwX5fFd3XzvvVb/v4n8PZ25EeSJEmSJEkaTtr1IA9JkiRJkiRJNTHoJ0mSJEmSJDVMu+b0kyRJkiRJWq5NmD5xSNKZPWXWkKQjteoz6BcR6wIfAFbtXJaZX2lnpiRJkiRJkiQtvf4M7/0BMA54uOWfJEmSJEmSpOVUf4b3PpGZX2h7TiRJkiRJkiQNiv4E/e6IiPcDtwCLADLzL23NlSRJkiRJkqSl1p+g32urf50WAdu1JzuSJEmSJEmSllWfQb/M3DYiXgi8HLgnMx9pf7YkSZIkSVLTbTHtqiFJ58aDtxmSdKTlSZ8P8oiI9wLXAYcDN0TEHm3PlSRJkiRJkqSl1p+n9x4EbJaZuwKvAz7V3ixJkiRJkiRJWhb9CfotzMy5AJn5BPB0e7MkSZIkSZIkaVn050Eef42IacBVwDbAX9ubJUmSJEmSJLWbcyo2W396+n0UuAfYkRLw26etOZIkSZIkSZK0THoM+kXE5tWf2wF3A5cCM4BthyBfkiRJkiRJkpZSb8N7twduAj7QZfki4Bdty5EkSZIkSZKkZdJj0C8zj63+vDYzv9m5PCKmtj1XkiRJkiRJkpZaj0G/iPgA8E5g24jYrlrcAbwGOGUI8iZJkiRJkiRpKfQ2vPdy4CHghcA3gBHAQnx6ryRJkiRJkpbRhOkThySd2VNmDUk6y5seH+SRmf/MzN8A+wPrZOZvgbcDKw9R3iRJkiRJkiQthR6Dfi3Oo/T4A7gM+Fb7siNJkiRJkiRpWfUn6EfV44/MvKq/20iSJEmSJEmqR29z+nWaExEfB64HtgSeaG+WJEmSJEmSJC2L/vTa2xPYEDiu+v+j7cyQJEmSJEmSpGXTZ0+/zJwdEccCK1Ge4PsK4JF2Z0ySJEmSJEnS0ukz6BcR5wBbAasDqwH3VK8lSZIkSZIkLYf6M7z3lcCrgZ9Thvc+3dYcSZIkSZIkSVom/Qn6PZGZi4DVM/MRYOU250mSJEmSJEnSMuhP0O8PEXEI8GBE/DfQ0eY8SZIkSZIkSVoGfc7pB5wHPAj8C9gZ+H1bcyRJkiRJkiRpmfQn6PetzNy6+vvH7cyMJEmSJEmSpGXXY9AvItbIzMeAJyPiJCCBhQCZedYQ5U+SJEmSJEnSAPU2p19nr757gX8CawJrV/8kSZIkSZIkLad6G977r4i4EXgF8OeW5YuAr7Q1V5IkSZIkSZKWWm9Bv52BlwBnAvsPTXYkSZIkSZIkLaseg36ZuRCYBbx96LIjSZIkSZIkaVn15+m9AxYRI4HTgU2AecDemTmjy3smANcBG2Xm0xGxGnA+Ze7AJ4CPZObsduRPkiRJkiRJarLeHuSxLHYFVs3MScDngGmtKyPircAvgLVaFn8CuD0z3wR8B/hCm/ImSZIkSZIkNVq7gn5bA5cDZOYNwOZd1i8EdgD+0d02wM+q9ZIkSZIkSZIGqC3De4FxwGMtrxdExKjMfBYgM68AiIietnkCWKPrTseMWYVRozrakuHhZPz40cM6/brV/fnrTn8wdXSMbNTnWdFY/vWzDupl+dfL8q+X5V8vy79+1sGKp+76qjv9utX9+etOvy7tCvo9DoxteT2yM+DXz23GAnO6vmHu3HmDk7thbs6cp7pdPqHm9IcLy3/wjB8/ulGfZ0Vj+dfPOqiX5V8vy79eln+9LP/6WQcrHn+H1cvyb68JE8Z2u7xdw3uvBd4GEBFbAbcPZBtgZ+Dq9mRNkiRJkiRJarZ29fS7BNgxIq4DRgB7RcRBwIzMvLSHbc4AzouIa4BngA+2KW+SJEmSJElSo7Ul6JeZC4H9uiy+q5v3rd/y91PAe9uRH0mSJEmSJGk4adfwXkmSJEmSJEk1MegnSZIkSZIkNYxBP0mSJEmSJKlhDPpJkiRJkiRJDWPQT5IkSZIkSWoYg36SJEmSJElSwxj0kyRJkiRJkhrGoJ8kSZIkSZLUMAb9JEmSJEmSpIYx6CdJkiRJkiQ1jEE/SZIkSZIkqWEM+kmSJEmSJEkNY9BPkiRJkiRJahiDfpIkSZIkSVLDGPSTJEmSJEmSGsagnyRJkiRJktQwBv0kSZIkSZKkhhlVdwYkSRJsMe2qtqdx48HbtD0NSZIkScsHg36SJIkJ0ye2PY3ZU2a1PQ1JkiRJhUE/SZKkmg1F0BUMvEqSJA0nzuknSZIkSZIkNYxBP0mSJEmSJKlhDPpJkiRJkiRJDWPQT5IkSZIkSWoYg36SJEmSJElSwxj0kyRJkiRJkhpmVN0ZkCRJqtsW064aknRuPHibIUlHkiRJsqefJEmSJEmS1DAG/SRJkiRJkqSGMegnSZIkSZIkNYxBP0mSJEmSJKlhfJCHJEmShrUJ0ycOSTqzp8waknQkSZLAoJ8kSZJq5tOTJUmSBp/DeyVJkiRJkqSGMegnSZIkSZIkNUxbhvdGxEjgdGATYB6wd2bOaFm/D7Av8CxwZGb+JCL+DfgLcEf1tksy8+vtyJ8kSZIkSZLUZO2a029XYNXMnBQRWwHTgHcBRMTawFRgc2BV4JqIuALYFLgwMz/ZpjxJkiRJkiRJw0K7hvduDVwOkJk3UAJ8nbYErs3MeZn5GDAD2BjYDNg0In4bET+IiBe3KW+SJEmSJElSo7Ur6DcOeKzl9YKIGNXDuieANYC7gCMy883AD4FT25Q3SZIkSZIkqdHaNbz3cWBsy+uRmflsD+vGAnOA3wFPVcsuAb7SdadjxqzCqFEdg5/bYWb8+NHDOv261f35605/MHV0jGzU51nRWP4rnrrrq+70lwd1l0Hd6det7s9fd/qDyTagXpZ//ayDFU/d9VV3+nWr+/PXnX5d2hX0uxbYBbiomtPv9pZ1vweOiohVgVWAV1Ee3nEecDFwEbA98IeuO507d16bsju8zJnzVLfLJ9Sc/nBh+Xdvi2lXDUk6Nx68zZCkMxyMHz96uf0+qXu91ddQnIP8vtgG1M3yHzy2AfWy/OtnHax4bAPqZfm314QJY7td3q6g3yXAjhFxHTAC2CsiDgJmZOalEXEKcDVlePHnM/PpiPgccE5E7A88CezdprxJkiRJkiRJjdaWoF9mLgT267L4rpb1ZwNnd9nmXmDbduRHkiRJkiRJGk7a9SAPSZIkSZIkSTVp1/BeSVqhTJg+cUjSmT1l1pCkI0mSJEka3uzpJ0mSJEmSJDWMQT9JkiRJkiSpYQz6SZIkSZIkSQ1j0E+SJEmSJElqGIN+kiRJkiRJUsMY9JMkSZIkSZIaxqCfJEmSJEmS1DAG/SRJkiRJkqSGMegnSZIkSZIkNYxBP0mSJEmSJKlhDPpJkiRJkiRJDWPQT5IkSZIkSWoYg36SJEmSJElSwxj0kyRJkiRJkhrGoJ8kSZIkSZLUMKPqzoAkSZIkSXXZYtpVbU/jxoO3aXsaktSVQT9JkiRJkmoyYfrEtqcxe8qstqchafnj8F5JkiRJkiSpYQz6SZIkSZIkSQ3j8F5JkiRJqpFzykmS2sGgnyRpSH5sgD84JEmSJGmoOLxXkiRJkiRJahh7+kmSlgs+uU6SJEmSBo89/SRJkiRJkqSGsaefJEmSJA1j9raXpGayp58kSZIkSZLUMAb9JEmSJEmSpIYx6CdJkiRJkiQ1jHP6SZIkScPYFtOuGpJ0bjx4myFJR5IkFfb0kyRJkiRJkhrGoJ8kSZIkSZLUMAb9JEmSJEmSpIZxTj9JkiRJtZkwfeKQpDN7yqwhSUeSpOWFPf0kSZIkSZKkhmlLT7+IGAmcDmwCzAP2zswZLev3AfYFngWOzMyfRMSLgO8BqwEPAntl5lPtyJ8kSZIkSZLUZO3q6bcrsGpmTgI+B0zrXBERawNTgTcCbwW+FhGrAF8EvpeZbwJuoQQFJUmSJEmSJA1Qu4J+WwOXA2TmDcDmLeu2BK7NzHmZ+RgwA9i4dRvgZ8AObcqbJEmSJEmS1GgjFi1aNOg7jYhvAhdn5s+q1/cBG2TmsxGxB7BRZn62Wvcd4DvAN6rl/4qIDYDvZObWg545SZIkSZIkqeHa1dPvcWBsazqZ+WwP68YCc7os71wmSZIkSZIkaYDaFfS7FngbQERsBdzesu73wJsiYtWIWAN4FXBH6zbAzsDVbcqbJEmSJEmS1GjtGt7b+fTejYERwF6UgN6MzLy0enrvxylBx6Mz8+KIWAs4j9LL7xHgg5n55KBnTpIkSZIkSWq4tgT99JyIuAa4MjO/VHdellZEnAuMysw96s5LdwarjCNiHPBTYAvghMz8wiBkr7t09gSOzMyJbdj3+sC9wCsyc8Zg73+wDWLdzaSU6Te7Wbc+K1CZtEvd5yLr4TlDVRcR8VpgbGZeHRFvAX4NrNQy3YZWAMt7GzxYbINXXLbByz/rYeitqG1wle//BdYG/l/nHP3Vui8BO7Rj3v2I2AG4IjNHDPa+q/1PpDws9BXA1Mw8qx3ptKT3FtpY5xFxJOVBqOfSvjZtTWDbzPx+P9+/LTA7M+9YyvSWqCPgsi6vzwR2zMwrl2b/w1m7hvdKK6KPUk4qrwVOrDkvGjz3Ay+mXOxKw8klQNSdCamfbIObyTZYw9WK2gZ/Cbgb2BD4TZd1JwDvHOL8DJZDgIWUz/XfNedlMH0feF2b9n0ssMsA3v8rSrB4aXWto66vXwxctQz7H7ZG1Z0BaTmyBvDXzLyr7oxo8GTmAuDvdedDqkFb7pZLbWIb3EC2wRrGVtQ2eA3gusyc2XVFZs4d+uwMmjWAOzLznrozMpgy81/Av9q0+6H+Di9RR9XzH1rr7PEhzk9jGPRbChExCTgO2AxYRHnoyMcy84GI2I0SFV8H+CYtvSkjYl3gLOCNwLPAj4BP9ucEGhGrAacC7wXmAl+kdHH998ycGREvqNJ9F7Aa8ONq3/+otn8VcBLwhmr7s4CvZObCav3uwFcoEfQfUg7y7G74WUTsChwFvAy4Czg8My/vT9n111CXcTV86iPV34uqzzYPOAXYARgN/An4VGZeVb3vZZQ6eTPwGHB6Zh5drZsInAbsCDwKfA/4YmY+05LmkZSuynOB/8rMb7Ws2xP4TJWPPwEHZ+ZvqnWrUu7C7Q78G+WuygGZ+bduPtfHKT0mdsjMG3org8FSx/FR2TAibqY8HOi3wEcy8+GuQ1qq+v0I5e7RBsCNwN6Z+dcqH5sB04FNgFuAK4FtMvMtS1ci7VPDcfJ+4Axgrc7vckTsBFxAubP3Ino5ZirvjIj9q3z9klJPj7Z8nuMpdyxnA8dn5vSW9Pekh+OibjW1Cz2eCyLiN8B6wNkR0Tn8A2CfiPg88ALgYuDjmfl0tb8ez+3V/u4AdqK0MZtk5iMDLafBUH1/DgFeAvylyudP+jrv9vX96iPN4dYGD2kZ2wYPnjqOj8pwbIOH+jixDe5BTe1C49rgaqj+esBbqnZpPUp7dxCl5+J9tAzvrT7bicBGwD3AMZn53WrduZRz85qU3oEPUObyP7daP47Sjr4DeBBYfA7uRz73BPYDZlHq9uBq+88DnwDGANdT2uG7q/J7c7XtBzNzRBVQOgXYlRI0u5TynX6iGpp7PqU9/hCl/V6f0jZvDKwLbAvMr9ZtDawE3ATsm5l39vezDOAzb0i5dtiU8gDUGS1lcWRmTuwu35n5paot+hylLm4FDszMG6vtR1OuX98HdAA/AaZQyrSzXd46M9fvI38zqz+viIgvAzN5fh19n1Je7wTGU9qGw7M83+E3tNQRpR3pWmeLqIb39pTvzHyi/6U6fDi8d4AiYizN+XtdAAAQsElEQVRlzpkrgVcDkykXLZ+vDsaLKA3yZsCqwKSWzU+jnBw2p3z5J1FOTv1xCuWEshPly/0Zyhe80yWUITG7ANtTupN3nnRfRPkB+iDwesrJsPNg7jxhf5vSZXtT4Mkqje4+/ybVfo+hnODPAi6p5n8YFDWV8aeAaZSnS7+YMhzlu5QT+BsoFwP3A9+o8rgK8IsqrUnAx4DPRMTuETGCUh//rPK4O6VB+1pLeutQ6msScBhwejWXRefJezqljDep0rksIl5abfsN4P8BHwa2ogTvL42I1u8DEfFOyol11yH8sVHX8QGwL+XCbQtKvV3cy3uPAA4E3kK5UO78obgGcDnlh8brKBeHhw0gD0OmprK+lPJ9m9yy7D+B/8nM+fRyzLTYC/ggpexfR1W+VVDkV5Ru+6+j1NFxEfHeav2e9H5c1KbG731v54J3Uy60Dqac3zr9J7AzJTj1bmDv6jP059y+F7An8K4aA36vA06mHL9BuYC8qAq69Xje7ev71Q/DqQ2uo4xtgwdBjccHDL82uI6ytg3uRo3f+ya2wVtQgmUnU25yAWxDuUY5pvWNEbE2Zc61C6o8fwU4NSJah4N+gnI8bwr8HDgjIv6tWvcN4JWUwM5USmBxIF5PGYa8JeVG2wGUuvhQtW4G8MsqOPRuyrXYxZQ2BuAcSqD8TcDbKd+dc1v2vw4wrsp75/I9qs+5MyXwfSnwN0p78gbKdcHxA/wcfaravJ9SgmSbUr7X+/Tw9iXyXdXHVynHx+soc+T9KiI6y+FMynXLuymBzNdQArkn8FyZbdGPbHa+5z+rbeH5dXQS5cbQZMr18lWUwPgqPL+OuquzVj3lW92wp9/ArU65OJmWmYuAeyPiYsqBvhdwbWaeBBARB7DkvAfrA38EZmbmMxHxbkqPkF5FxBjKSWyXzLy+WjaVcmFERGxMOWFumJl/rpbtDtwVEa+mHBBPU+48zAf+XB3oX6WcmPanXDCcUW37CeCtPWTnEOCczrs4wF8j4vXAJykX3YNhyMs4Mx+LiLnA/Mz8e7XvHwP/m5n3V6+nA5dXPyh2oNxN3DwzHwPuiIgplB9r21F+8G9VDWu5q1r3i4j4bJXkM8CeVaN9Z3VnZj9K0GAqcFpmfqd672HV+k9GxNGUxuwdmfnrKl+7Uy7sdgI67yxNojxB+8NDPNnpkNddi29k5oXVvj9Wpf0aSi+Ork7OzF9W7z0D+HS1/H2Uu32fzDLp7l0R8Ua6b2zqVsdx8lRE/IhyIfiTiFgZ2K36B6VB7+mY6fTZzPx9tf4iyo8HKBcvf8zMw6vXf6kuyD8D/IBejgvg0L7y3mZ1tAsvoJdzQWb+NCIWAI9X57fOTadk5p+A2yPiCp4r//6c23+WmdcMqGQG3/qU8vlb1Zvia5SeQm+g9/NuX9+vHg3DNnh9hriMbYMHzfoMcd21GG5t8PoM/XFiG9y99Rn6dqGRbXBmzo6IZyjn0tnV4q+39MRtffsU4NeZ+fXq9YyIeCXleP5xtez2zDyu2vbzlMDcayLiNkpwaIfMvLlafySl9/ZAHN3ZsysiPkN5QMevqtefBN5GeRjJdyPiX5SHYv09Il5OOW5elM/1yP8wMDPKCIxOx3X57Ldk5iXV69UpozdOz2p0RpTejYcz+HYAJgCfqNK6KyK2owQtu9Oa7+9SemD+qFp3VJSbXHtHxCnA+4GdM/Pq6v37AW/OzLktZTa7mzSWUH13AP5Zbdu5qrWOrqF8n26vXp9ACXy/JDPvba2jav0SrztFuUHUbb77yudwZdBvgKoTxbnAgVHuvmxIOWH/rvr7tpb3zq9Oap2+SLn79K6I+AUlct2fp+G8EliZ0oB1ur7l71cBT3T+2KjSzoj4Z7XuVcDN1Y+NTtcBL4rSA2FjWrpUZ+azEXFTD3l5FbBRdUHXaSXK3flBUVMZd+cM4P0R8QZKHWxWLe+o8jGj+rHRmZcLYPGP+vFAayM/glKH61Wv7+1yl+5myg8OKGV8ZJe8XF8t/w9KD93ftaT7j4jIan3nD46zKcf384YbtVPNdbf4+Mgy3O4flDK5sZv3/rXl78cp32Eox8ItueRTtq6n3EVartRY1t8Dvlf92JgMPMVzk+r2dsx0ai37xyg936DU1e9Y0nWUi8rO9T0dF7WqqS76Ohf8tIfteiv/vs7tM/uRr3b7OXANcEtE3EG5y/4tyoV9b+fdvr5fvRlWbTD1lHF3bIMHrs66G1ZtMPWVtW3w89VRF8OpDe4p3VcBO1c3bDqN4rlgIbR83sx8vKqHlSjl10HL9RFlaOxAPNoSTBoDTAQuiIiFLe9ZtUqru7yPAO7rEsikev+C6u+ZXdYtfp2ZT0bE6cCHImJzyjG3KWUY+WDbkDLnbWtZ30S52dSdmS1/vwo4OiK+2rJsFUpP1P+g1NkfOldUNwUG65picR1VvgPsGhH70PM5qj/ane/GMeg3QBGxDuUgu4XSyJxN6RLc+ejyrhNeLr7Iz8wfVXcPdqV0C/425W7+nn0k23nx07rv1r+f7mG7jupfd+s7Wv5/tpt89zRx5yhKl91vd1k+r4f3D1hNZdw1DyOBKyhzdPw35Y7VypRH2EPpJdCTUZSuzO/oZt391f8Luywf2bLP7iZj7a0uW9d3OoLSzfn0iNgqq3mj2q3muuutTLvqurwzXwM5FmpVY1l3DqnbkdLb4PuZubAfx0ynBV1ed+azp/NUZzvV23FRq5rqor/ngq56Kv/+nNt7SnPIVD1ddqCU7TuA91B6Dkyj9/NuX9+v3gyrNrimMl6CbfDSqbnuhlUbXGNZ2wZ3UVNdDKc2uKd0RwEXUnqtt2r9jN2dA3pqS+d3feMA8tVZZ++nDLttNaebbUdRejN2NzXGQzw3VLXrZ1/8ugo03gj8gzKH3oWUQNbn+pH3pdHjtWQ3upbNwZRzR6u5wFqDkK/edC2/71DmsP4u5SbFQyx5E7W/ersGUDec02/gdqN01X5bZn696lK6AeVAvIOWMe9R5nTYuOX1kcDEzDw7Mzvnceh23p4uZlC+3Ju1LGv9+y5gbNUlvTOtDSnj+RP4M7BpRKzUss0kyklqNuXO9GYt23bQ/UmQan8bZOaMzn+U7u279fD+pVFHGXe1IWUOi8mZeVRm/pTnhpeMoFxEvDzKJLSdaX+56u2TlAleH20po7Upc4h0HnMvqxqLTltS6glKfb6+S362qvY7g3JRvHh9RLwQeEW1vtPFwGcpd3d6mvOhHeqsu41a9vUKyt3dgT4F8k5gk1hybqbNenpzzWop66oHxv9Q5qPZiXKRA30fM335M8//3k/iue91b8dF3epqF/o6FwxkePxQnNuXWZRJ17+QmVdnZuc57mFKWfd23u3r+9WbYdUG11TGXdkGL4Wa625YtcF1lbVt8PPV2C4Muza4iwT+o0ue30o1T2E/tp3PknPFvW6pM5I5B/g/4MUtebmXMvXKJt1tQpmapaPl/VDmhBvXzfu78xbK9+stmXl8lqkcXkp7blTcAfx7lGHlnfpbXgms26WeDqbk/x5KkHbxviJih4j4S3UjYSDf4V5V7fUHgQ9m5hezDJPunN9xoGXWV77VhT39Bu5RYJ2I2JHSZfm9lElcb6GM6/9URHQO19qf0tW406uA06qhJ09V2/2BPmQZF/9t4KSI6DyRnlL9v6gaRvQT4Lwoc1VAmUvmmsy8NSJmAF8GzoyI4ykN0peBM6q7g6cBv43y1JzfUu6OrU/3B/pJwDUR8XvKXcQdKJPOv6uvzzEAQ17G3ZhDuWv9voi4hNIofblatwqlN8/9lMlHv0Qpr6mU4UG/oJyMLoiIwyhPUPsmcFtmPl11I1+ZUl9HUO54vJcy9wiUO5PnRcSdwA2UOcFeC3y06kr+DeCUKE9ieoTyxMgHKPNLrd35ATLzwYg4itKl++Icmon366y7T0XErZQfg6cDP6mOjfUHsI8LKReDJ1fHxZsodw2vHcA+hkqdZf09yvf8gaye/kXfx0xfTgc+HWXOrHMpPyam8NwE2D0eFwPId7vU0S70dS6Achf3lfHcpNm9GYpz+2D4F/DFiPg/ynn4tZSL7hvo/bzb1/erR8OwDR7yMu6GbfDSqbPuhlsbXGdZ2wYvqY52Ybi2wa1OB6ZGmUPxHEpw7TjK/IS9yjLU97vA16M8JGY14L+WMT8nAl+NiIcpQbJDKT1iP931jZn554i4HPhulLn/nqb0POvIzIcinj/mtxuPUr5T746I31HqrPNabrBdSZku4pwocyNuRenR2p+HRJ1YbXcXZRj8HpTj9swsTyr+NqUe9qH0LD0O+GV1fTIXeG1ErJOZD/QjrbnAqyOiu2kd/kXpXfnuiHiIMkT3tGpdf85Ri/WV74Hsa7gwEjpwF1G6pF5E+WG2Pc89LWoW5cl9/0l5HPaLeO7ED+UJRg8Av6TMHzOKEvHuj0OqfV5JuYN8QbW8s3vrRygXWr+kXAjcSTVZfJbx/zsBL6f8CJ0OfJ0ylxRZJibfn3KyvZVyd/Zauuk6m+UJdLtT7lzfWX32vao7ioOlrjJeLDNnVfs6mNJN/HDKD4r5wOuyTAj8LsodipspT6D6SmZ+v1r3TsodiOsojffVLHnn61bKyfuGat97ZeZNVdoXU7qGf4Uywf+2lMeTd84V9BnKRc3/VPufB2yXmd11vz+J0pvkuIGWwVKqs+6Op1zg/o7Se2avgWa+OlZ2oQwR+SNliOX5LJ/dyOss62soF7idPQz6PGb62mG1/dspd4lvp5yPDs7Mb1br+zou6lRXXfR1LjiN8kTNs/va0RCd25dZZt5KOS4/Tel5ciLle/Jzejnv9vX96odh0wbXWMatebANXgo1192waoNrLmvb4BY11sWwa4NbZebfKEOnd6AE2aYBR2T1UKp+OIDyXf4FZVjzQB/i0dUJlLZgOuU7+hrgrZn5YA/v/xCl3f4F5YbbAwwgyFq13V+m5PuPlHPe/sALY5Cfap1lTuC3UXoh/gH4OCXo2p9tv085do+gfLd2ozxV/tbqLQdS5sK7nDJFwE08F7j9DuXa5bZY8oFAPTmJ8pTnI3r4DHtU6f+Z8oTooyjlvjS9PHvLt7oYsWjRoPXaVBtFxK7Alfnc04G2oPwoWD2XnBx8afa9JfBYZmbLsjuB4zPz3GXZt7QiiYiXAetky9PRojz9bvXM3LO2jC1nImI0ZejMltny8AKpqWyDpfazDe4f22BJ0kA4vHfFcQSwS9WFeizljuqPlvXHRmUSpXv2hygTan6A0i3+8l63kppnDeCXEbE7ZXLezSh3Aj9Qa66WIxHxHsqd89v8saFhxDZYaj/b4D7YBkuSBsqg33Kgmntjx17e8hlKl+9TKUODngF+ROnWOhimAy+jPOFrDcqwl50z8++DtP/a9aeMM7Nf3aQ1tIay7qr5t6ZQ5hRaF7gPOGh5Hl4xmPp5Lvo0pe1YnueZWeF5zho6tsHt5/d5xWUbPHRsg5cfnrPqU/Wk/3Uvb5mTmRN7Wa9BFhFrUubH7M06mfnYUORHA+fw3uVARLyY8gShnsz2IFo2lvGKy7obOpb18sO6GDqWdftZxisu627oWNbLD+uiPhGxCiXo35MFmXnvUOVHEOWJ6i/r4233+BCN5ZdBP0mSJEmSJKlhfHqvJEmSJEmS1DAG/SRJkiRJkqSGMegnSZIkSZIkNYxBP0mSJEmSJKlhDPpJkiRJkiRJDfP/ARO2hIK7NyuTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a0b472f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# positive cases only\n",
    "converted_control = control_group[control_group['converted']==1]\n",
    "converted_test = test_group[test_group['converted']==1]\n",
    "\n",
    "onehot_opsys_control = onehot_encode_function(converted_control['source'])\n",
    "onehot_opsys_test = onehot_encode_function(converted_test['source'])\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(22,4))\n",
    "plt.bar(np.arange(0, 36, 3), np.sum(onehot_opsys_control, axis=0)/np.sum(onehot_opsys_control), label='control')\n",
    "plt.bar(np.arange(1, 37, 3), np.sum(onehot_opsys_test, axis=0)/np.sum(onehot_opsys_test), label='test')\n",
    "plt.xticks(np.arange(0, 36, 3),list(source_cats.keys()), fontsize=14)\n",
    "plt.legend(loc=1, fontsize=14)\n",
    "plt.ylabel('fraction')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aha - it looks like in the test group that Google-sourced users were more likely to convert than Facebook-source users, relative to the control group. It's not immediately clear why this would be, but we can test the significance of the difference with a Chi-squared test, which compares distributions of categorical variables. Our null hypothesis here is that there is no difference between the test and control groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chi-squared p-value: 0.006847648891839333\n"
     ]
    }
   ],
   "source": [
    "# count Google, Facebook conversions in the control and test groups\n",
    "control_google = converted_control[converted_control['source']==0.0]['source'].count()\n",
    "test_google = converted_test[converted_test['source']==0.0]['source'].count()\n",
    "control_fb = converted_control[converted_control['source']==1.0]['source'].count()\n",
    "test_fb = converted_test[converted_test['source']==1.0]['source'].count()\n",
    "\n",
    "\n",
    "test_pop = [test_google, test_fb]\n",
    "control_pop = [control_google, control_fb]\n",
    "norm_factor = sum(test_pop)/sum(control_pop)\n",
    "control_pop = [int(i*norm_factor) for i in control_pop]\n",
    "# small issue with the total of the two samples being different - adjust by downscaling the control pop to have \n",
    "# total size = test pop (this will lower significance, which is okay)\n",
    "chisq, pval_chi = chisquare(control_pop, test_pop)\n",
    "print('Chi-squared p-value:',pval_chi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This doesn't meet our 3-sigma threshold we used previously, but as this is a less high-stakes insight we might be able to accept a 2-sigma (95%) significance level. \n",
    "\n",
    "There are a few ways forward with this insight - we could step up Google advertising, since that channel has a higher conversion rate than Facebook, or we could try and incentivize Facebook users with deals, etc. The best way may be a mix of both. \n",
    "\n",
    "# 4. Conclusion\n",
    "\n",
    "To briefly summarize, we conducted a followup analysis of an A/B test designed to determine the effects of increasing the price of a software product. Data was collected by randomly presenting users with the normal software price (the control group) and a 50% increased price (the test group) over a period of 3 months. \n",
    "\n",
    "### Key Insights\n",
    "* **Effects of increasing the price on conversion rate.** As expected, increasing the software price decreased conversion rate over the test period by 22% (>99.99% confidence). If the higher price is adopted, it may be worth conducting periodic longer-term conversion rate monitoring to ensure that, e.g., increased price dissatisfaction and a corresponding drop in referrals doesn't result in steadily declining sales over the long term.\n",
    "\n",
    "* **Effects of increasing the price on total revenue.** While conversion rate decreased by 22%, net revenue per test user actually increased by 18% relative to the control group (>99.99 confidence), due to the higher price of the software.\n",
    "\n",
    "* **Retaining customers under an increase in price.** Considering the test group, analysis shows with high confidence (>95%) that the fraction of conversions coming from Google ads increased while those coming from Facebook ads decreased. If the higher pricing is adopted, incentive programs can be designed to maintain the current Facebook conversion rate and/or Google ads can be increased to reach additional users. \n",
    "\n",
    "* **A/B testing efficiency.** If the primary goal was to address the two above questions, then analysis shows that a high level of confidence (99.7%) could have been reached with approximately 10% of the total data. However, the customer retention analysis above was only made possible by the large sample we collected. My recommendation is to first define the goals of the A/B test and a level of significance desired, then design the test to meet those objectives. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}