DV0101EN-2-3-1-Pie-Charts-Box-Plots-Scatter-Plots-and-Bubble-Plots-py-v2.0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<a href=\"https://cognitiveclass.ai\"><img src = \"https://ibm.box.com/shared/static/9gegpsmnsoo25ikkbl4qzlvlyjbgxs5x.png\" width = 400> </a>\n",
    "\n",
    "<h1 align=center><font size = 5>Pie Charts, Box Plots, Scatter Plots, and Bubble Plots</font></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Introduction\n",
    "\n",
    "In this lab session, we continue exploring the Matplotlib library. More specificatlly, we will learn how to create pie charts, box plots, scatter plots, and bubble charts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Table of Contents\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "\n",
    "1. [Exploring Datasets with *p*andas](#0)<br>\n",
    "2. [Downloading and Prepping Data](#2)<br>\n",
    "3. [Visualizing Data using Matplotlib](#4) <br>\n",
    "4. [Pie Charts](#6) <br>\n",
    "5. [Box Plots](#8) <br>\n",
    "6. [Scatter Plots](#10) <br>\n",
    "7. [Bubble Plots](#12) <br> \n",
    "</div>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Exploring Datasets with *pandas* and Matplotlib<a id=\"0\"></a>\n",
    "\n",
    "Toolkits: The course heavily relies on [*pandas*](http://pandas.pydata.org/) and [**Numpy**](http://www.numpy.org/) for data wrangling, analysis, and visualization. The primary plotting library we will explore in the course is [Matplotlib](http://matplotlib.org/).\n",
    "\n",
    "Dataset: Immigration to Canada from 1980 to 2013 - [International migration flows to and from selected countries - The 2015 revision](http://www.un.org/en/development/desa/population/migration/data/empirical2/migrationflows.shtml) from United Nation's website.\n",
    "\n",
    "The dataset contains annual data on the flows of international migrants as recorded by the countries of destination. The data presents both inflows and outflows according to the place of birth, citizenship or place of previous / next residence both for foreigners and nationals. In this lab, we will focus on the Canadian Immigration data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Downloading and Prepping Data <a id=\"2\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Import primary modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np  # useful for many scientific computing in Python\n",
    "import pandas as pd # primary data structure library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's download and import our primary Canadian Immigration dataset using *pandas* `read_excel()` method. Normally, before we can do that, we would need to download a module which *pandas* requires to read in excel files. This module is **xlrd**. For your convenience, we have pre-installed this module, so you would not have to worry about that. Otherwise, you would need to run the following line of code to install the **xlrd** module:\n",
    "```\n",
    "!conda install -c anaconda xlrd --yes\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Download the dataset and read it into a *pandas* dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Solving environment: | "
     ]
    }
   ],
   "source": [
    "!conda install -c anaconda xlrd --yes\n",
    "df_can = pd.read_excel('https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DV0101EN/labs/Data_Files/Canada.xlsx',\n",
    "                       sheet_name='Canada by Citizenship',\n",
    "                       skiprows=range(20),\n",
    "                       skipfooter=2\n",
    "                      )\n",
    "\n",
    "print('Data downloaded and read into a dataframe!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's take a look at the first five items in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_can.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's find out how many entries there are in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# print the dimensions of the dataframe\n",
    "print(df_can.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Clean up data. We will make some modifications to the original dataset to make it easier to create our visualizations. Refer to *Introduction \n",
    "to Matplotlib and Line Plots* and *Area Plots, Histograms, and Bar Plots* for a detailed description of this preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# clean up the dataset to remove unnecessary columns (eg. REG) \n",
    "df_can.drop(['AREA', 'REG', 'DEV', 'Type', 'Coverage'], axis=1, inplace=True)\n",
    "\n",
    "# let's rename the columns so that they make sense\n",
    "df_can.rename(columns={'OdName':'Country', 'AreaName':'Continent','RegName':'Region'}, inplace=True)\n",
    "\n",
    "# for sake of consistency, let's also make all column labels of type string\n",
    "df_can.columns = list(map(str, df_can.columns))\n",
    "\n",
    "# set the country name as index - useful for quickly looking up countries using .loc method\n",
    "df_can.set_index('Country', inplace=True)\n",
    "\n",
    "# add total column\n",
    "df_can['Total'] = df_can.sum(axis=1)\n",
    "\n",
    "# years that we will be using in this lesson - useful for plotting later on\n",
    "years = list(map(str, range(1980, 2014)))\n",
    "print('data dimensions:', df_can.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Visualizing Data using Matplotlib<a id=\"4\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Import `Matplotlib`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "mpl.style.use('ggplot') # optional: for ggplot-like style\n",
    "\n",
    "# check for latest version of Matplotlib\n",
    "print('Matplotlib version: ', mpl.__version__) # >= 2.0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Pie Charts <a id=\"6\"></a>\n",
    "\n",
    "A `pie chart` is a circualr graphic that displays numeric proportions by dividing a circle (or pie) into proportional slices. You are most likely already familiar with pie charts as it is widely used in business and media. We can create pie charts in Matplotlib by passing in the `kind=pie` keyword.\n",
    "\n",
    "Let's use a pie chart to explore the proportion (percentage) of new immigrants grouped by continents for the entire time period from 1980 to 2013. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 1: Gather data. \n",
    "\n",
    "We will use *pandas* `groupby` method to summarize the immigration data by `Continent`. The general process of `groupby` involves the following steps:\n",
    "\n",
    "1. **Split:** Splitting the data into groups based on some criteria.\n",
    "2. **Apply:** Applying a function to each group independently:\n",
    "       .sum()\n",
    "       .count()\n",
    "       .mean() \n",
    "       .std() \n",
    "       .aggregate()\n",
    "       .apply()\n",
    "       .etc..\n",
    "3. **Combine:** Combining the results into a data structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DV0101EN/labs/Images/Mod3Fig4SplitApplyCombine.png\" height=400 align=\"center\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# group countries by continents and apply sum() function \n",
    "df_continents = df_can.groupby('Continent', axis=0).sum()\n",
    "\n",
    "# note: the output of the groupby method is a `groupby' object. \n",
    "# we can not use it further until we apply a function (eg .sum())\n",
    "print(type(df_can.groupby('Continent', axis=0)))\n",
    "\n",
    "df_continents.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 2: Plot the data. We will pass in `kind = 'pie'` keyword, along with the following additional parameters:\n",
    "- `autopct` -  is a string or function used to label the wedges with their numeric value. The label will be placed inside the wedge. If it is a format string, the label will be `fmt%pct`.\n",
    "- `startangle` - rotates the start of the pie chart by angle degrees counterclockwise from the x-axis.\n",
    "- `shadow` - Draws a shadow beneath the pie (to give a 3D feel)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# autopct create %, start angle represent starting point\n",
    "df_continents['Total'].plot(kind='pie',\n",
    "                            figsize=(5, 6),\n",
    "                            autopct='%1.1f%%', # add in percentages\n",
    "                            startangle=90,     # start angle 90° (Africa)\n",
    "                            shadow=True,       # add shadow      \n",
    "                            )\n",
    "\n",
    "plt.title('Immigration to Canada by Continent [1980 - 2013]')\n",
    "plt.axis('equal') # Sets the pie chart to look like a circle.\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The above visual is not very clear, the numbers and text overlap in some instances. Let's make a few modifications to improve the visuals:\n",
    "\n",
    "* Remove the text labels on the pie chart by passing in `legend` and add it as a seperate legend using `plt.legend()`.\n",
    "* Push out the percentages to sit just outside the pie chart by passing in `pctdistance` parameter.\n",
    "* Pass in a custom set of colors for continents by passing in `colors` parameter.\n",
    "* **Explode** the pie chart to emphasize the lowest three continents (Africa, North America, and Latin America and Carribbean) by pasing in `explode` parameter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "colors_list = ['gold', 'yellowgreen', 'lightcoral', 'lightskyblue', 'lightgreen', 'pink']\n",
    "explode_list = [0.1, 0, 0, 0, 0.1, 0.1] # ratio for each continent with which to offset each wedge.\n",
    "\n",
    "df_continents['Total'].plot(kind='pie',\n",
    "                            figsize=(15, 6),\n",
    "                            autopct='%1.1f%%', \n",
    "                            startangle=90,    \n",
    "                            shadow=True,       \n",
    "                            labels=None,         # turn off labels on pie chart\n",
    "                            pctdistance=1.12,    # the ratio between the center of each pie slice and the start of the text generated by autopct \n",
    "                            colors=colors_list,  # add custom colors\n",
    "                            explode=explode_list # 'explode' lowest 3 continents\n",
    "                            )\n",
    "\n",
    "# scale the title up by 12% to match pctdistance\n",
    "plt.title('Immigration to Canada by Continent [1980 - 2013]', y=1.12) \n",
    "\n",
    "plt.axis('equal') \n",
    "\n",
    "# add legend\n",
    "plt.legend(labels=df_continents.index, loc='upper left') \n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Question:** Using a pie chart, explore the proportion (percentage) of new immigrants grouped by continents in the year 2013.\n",
    "\n",
    "**Note**: You might need to play with the explore values in order to fix any overlapping slice values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "explode_list = [0.1, 0, 0, 0, 0.1, 0.2] # ratio for each continent with which to offset each wedge.\n",
    "-->\n",
    "\n",
    "<!--\n",
    "df_continents['2013'].plot(kind='pie',\n",
    "                            figsize=(15, 6),\n",
    "                            autopct='%1.1f%%', \n",
    "                            startangle=90,    \n",
    "                            shadow=True,       \n",
    "                            labels=None,                 # turn off labels on pie chart\n",
    "                            pctdistance=1.12,            # the ratio between the pie center and start of text label\n",
    "                            explode=explode_list         # 'explode' lowest 3 continents\n",
    "                            )\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # scale the title up by 12% to match pctdistance\n",
    "plt.title('Immigration to Canada by Continent in 2013', y=1.12) \n",
    "plt.axis('equal') \n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # add legend\n",
    "plt.legend(labels=df_continents.index, loc='upper left') \n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # show plot\n",
    "plt.show()\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Box Plots <a id=\"8\"></a>\n",
    "\n",
    "A `box plot` is a way of statistically representing the *distribution* of the data through five main dimensions: \n",
    "\n",
    "- **Minimun:** Smallest number in the dataset.\n",
    "- **First quartile:** Middle number between the `minimum` and the `median`.\n",
    "- **Second quartile (Median):** Middle number of the (sorted) dataset.\n",
    "- **Third quartile:** Middle number between `median` and `maximum`.\n",
    "- **Maximum:** Highest number in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DV0101EN/labs/Images/boxplot_complete.png\" width=440, align=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "To make a `box plot`, we can use `kind=box` in `plot` method invoked on a *pandas* series or dataframe.\n",
    "\n",
    "Let's plot the box plot for the Japanese immigrants between 1980 - 2013."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 1: Get the dataset. Even though we are extracting the data for just one country, we will obtain it as a dataframe. This will help us with calling the `dataframe.describe()` method to view the percentiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# to get a dataframe, place extra square brackets around 'Japan'.\n",
    "df_japan = df_can.loc[['Japan'], years].transpose()\n",
    "df_japan.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 2: Plot by passing in `kind='box'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_japan.plot(kind='box', figsize=(8, 6))\n",
    "\n",
    "plt.title('Box plot of Japanese Immigrants from 1980 - 2013')\n",
    "plt.ylabel('Number of Immigrants')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We can immediately make a few key observations from the plot above:\n",
    "1. The minimum number of immigrants is around 200 (min), maximum number is around 1300 (max), and  median number of immigrants is around 900 (median).\n",
    "2. 25% of the years for period 1980 - 2013 had an annual immigrant count of ~500 or fewer (First quartile).\n",
    "2. 75% of the years for period 1980 - 2013 had an annual immigrant count of ~1100 or fewer (Third quartile).\n",
    "\n",
    "We can view the actual numbers by calling the `describe()` method on the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_japan.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "One of the key benefits of box plots is comparing the distribution of multiple datasets. In one of the previous labs, we observed that China and India had very similar immigration trends. Let's analyize these two countries further using box plots.\n",
    "\n",
    "**Question:** Compare the distribution of the number of new immigrants from India and China for the period 1980 - 2013."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 1: Get the dataset for China and India and call the dataframe **df_CI**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "df_CI= df_can.loc[['China', 'India'], years].transpose()\n",
    "df_CI.head()\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's view the percentages associated with both countries using the `describe()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "df_CI.describe()\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 2: Plot data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "df_CI.plot(kind='box', figsize=(10, 7))\n",
    "-->\n",
    "\n",
    "<!--\n",
    "plt.title('Box plots of Immigrants from China and India (1980 - 2013)')\n",
    "plt.xlabel('Number of Immigrants')\n",
    "-->\n",
    "\n",
    "<!--\n",
    "plt.show()\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We can observe that, while both countries have around the same median immigrant population (~20,000),  China's immigrant population range is more spread out than India's. The maximum population from India for any year (36,210) is around 15% lower than the maximum population from China (42,584).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "If you prefer to create horizontal box plots, you can pass the `vert` parameter in the **plot** function and assign it to *False*. You can also specify a different color in case you are not a big fan of the default red color."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# horizontal box plots\n",
    "df_CI.plot(kind='box', figsize=(10, 7), color='blue', vert=False)\n",
    "\n",
    "plt.title('Box plots of Immigrants from China and India (1980 - 2013)')\n",
    "plt.xlabel('Number of Immigrants')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Subplots**\n",
    "\n",
    "Often times we might want to plot multiple plots within the same figure. For example, we might want to perform a side by side comparison of the box plot with the line plot of China and India's immigration.\n",
    "\n",
    "To visualize multiple plots together, we can create a **`figure`** (overall canvas) and divide it into **`subplots`**, each containing a plot. With **subplots**, we usually work with the **artist layer** instead of the **scripting layer**. \n",
    "\n",
    "Typical syntax is : <br>\n",
    "```python\n",
    "    fig = plt.figure() # create figure\n",
    "    ax = fig.add_subplot(nrows, ncols, plot_number) # create subplots\n",
    "```\n",
    "Where\n",
    "- `nrows` and `ncols` are used to notionally split the figure into (`nrows` \\* `ncols`) sub-axes,  \n",
    "- `plot_number` is used to identify the particular subplot that this function is to create within the notional grid. `plot_number` starts at 1, increments across rows first and has a maximum of `nrows` * `ncols` as shown below.\n",
    "\n",
    "<img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DV0101EN/labs/Images/Mod3Fig5Subplots_V2.png\" width=500 align=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We can then specify which subplot to place each plot by passing in the `ax` paramemter in `plot()` method as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "fig = plt.figure() # create figure\n",
    "\n",
    "ax0 = fig.add_subplot(1, 2, 1) # add subplot 1 (1 row, 2 columns, first plot)\n",
    "ax1 = fig.add_subplot(1, 2, 2) # add subplot 2 (1 row, 2 columns, second plot). See tip below**\n",
    "\n",
    "# Subplot 1: Box plot\n",
    "df_CI.plot(kind='box', color='blue', vert=False, figsize=(20, 6), ax=ax0) # add to subplot 1\n",
    "ax0.set_title('Box Plots of Immigrants from China and India (1980 - 2013)')\n",
    "ax0.set_xlabel('Number of Immigrants')\n",
    "ax0.set_ylabel('Countries')\n",
    "\n",
    "# Subplot 2: Line plot\n",
    "df_CI.plot(kind='line', figsize=(20, 6), ax=ax1) # add to subplot 2\n",
    "ax1.set_title ('Line Plots of Immigrants from China and India (1980 - 2013)')\n",
    "ax1.set_ylabel('Number of Immigrants')\n",
    "ax1.set_xlabel('Years')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "** * Tip regarding subplot convention **\n",
    "\n",
    "In the case when `nrows`, `ncols`, and `plot_number` are all less than 10, a convenience exists such that the a 3 digit number can be given instead, where the hundreds represent `nrows`, the tens represent `ncols` and the units represent `plot_number`. For instance,\n",
    "```python\n",
    "   subplot(211) == subplot(2, 1, 1) \n",
    "```\n",
    "produces a subaxes in a figure which represents the top plot (i.e. the first) in a 2 rows by 1 column notional grid (no grid actually exists, but conceptually this is how the returned subplot has been positioned)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's try something a little more advanced. \n",
    "\n",
    "Previously we identified the top 15 countries based on total immigration from 1980 - 2013.\n",
    "\n",
    "**Question:** Create a box plot to visualize the distribution of the top 15 countries (based on total immigration) grouped by the *decades* `1980s`, `1990s`, and `2000s`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 1: Get the dataset. Get the top 15 countries based on Total immigrant population. Name the dataframe **df_top15**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "df_top15 = df_can.sort_values(['Total'], ascending=False, axis=0).head(15)\n",
    "df_top15\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 2: Create a new dataframe which contains the aggregate for each decade. One way to do that:\n",
    "   1. Create a list of all years in decades 80's, 90's, and 00's.\n",
    "   2. Slice the original dataframe df_can to create a series for each decade and sum across all years for each country.\n",
    "   3. Merge the three series into a new data frame. Call your dataframe **new_df**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "\\\\ # create a list of all years in decades 80's, 90's, and 00's\n",
    "years_80s = list(map(str, range(1980, 1990))) \n",
    "years_90s = list(map(str, range(1990, 2000))) \n",
    "years_00s = list(map(str, range(2000, 2010))) \n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # slice the original dataframe df_can to create a series for each decade\n",
    "df_80s = df_top15.loc[:, years_80s].sum(axis=1) \n",
    "df_90s = df_top15.loc[:, years_90s].sum(axis=1) \n",
    "df_00s = df_top15.loc[:, years_00s].sum(axis=1)\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # merge the three series into a new data frame\n",
    "new_df = pd.DataFrame({'1980s': df_80s, '1990s': df_90s, '2000s':df_00s}) \n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # display dataframe\n",
    "new_df.head()\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's learn more about the statistics associated with the dataframe using the `describe()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "new_df.describe()\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 3: Plot the box plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "new_df.plot(kind='box', figsize=(10, 6))\n",
    "-->\n",
    "\n",
    "<!--\n",
    "plt.title('Immigration from top 15 countries for decades 80s, 90s and 2000s')\n",
    "-->\n",
    "\n",
    "<!--\n",
    "plt.show()\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Note how the box plot differs from the summary table created. The box plot scans the data and identifies the outliers. In order to be an outlier, the data value must be:<br>\n",
    "* larger than Q3 by at least 1.5 times the interquartile range (IQR), or,\n",
    "* smaller than Q1 by at least 1.5 times the IQR.\n",
    "\n",
    "Let's look at decade 2000s as an example: <br>\n",
    "* Q1 (25%) = 36,101.5 <br>\n",
    "* Q3 (75%) = 105,505.5 <br>\n",
    "* IQR = Q3 - Q1 = 69,404 <br>\n",
    "\n",
    "Using the definition of outlier, any value that is greater than Q3 by 1.5 times IQR will be flagged as outlier.\n",
    "\n",
    "Outlier > 105,505.5 + (1.5 * 69,404) <br>\n",
    "Outlier > 209,611.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# let's check how many entries fall above the outlier threshold \n",
    "new_df[new_df['2000s']> 209611.5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "China and India are both considered as outliers since their population for the decade exceeds 209,611.5. \n",
    "\n",
    "The box plot is an advanced visualizaiton tool, and there are many options and customizations that exceed the scope of this lab. Please refer to [Matplotlib documentation](http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.boxplot) on box plots for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Scatter Plots <a id=\"10\"></a>\n",
    "\n",
    "A `scatter plot` (2D) is a useful method of comparing variables against each other. `Scatter` plots look similar to `line plots` in that they both map independent and dependent variables on a 2D graph. While the datapoints are connected together by a line in a line plot, they are not connected in a scatter plot. The data in a scatter plot is considered to express a trend. With further analysis using tools like regression, we can mathematically calculate this relationship and use it to predict trends outside the dataset.\n",
    "\n",
    "Let's start by exploring the following:\n",
    "\n",
    "Using a `scatter plot`, let's visualize the trend of total immigrantion to Canada (all countries combined) for the years 1980 - 2013."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 1: Get the dataset. Since we are expecting to use the relationship betewen `years` and `total population`, we will convert `years` to `int` type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# we can use the sum() method to get the total population per year\n",
    "df_tot = pd.DataFrame(df_can[years].sum(axis=0))\n",
    "\n",
    "# change the years to type int (useful for regression later on)\n",
    "df_tot.index = map(int, df_tot.index)\n",
    "\n",
    "# reset the index to put in back in as a column in the df_tot dataframe\n",
    "df_tot.reset_index(inplace = True)\n",
    "\n",
    "# rename columns\n",
    "df_tot.columns = ['year', 'total']\n",
    "\n",
    "# view the final dataframe\n",
    "df_tot.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 2: Plot the data. In `Matplotlib`, we can create a `scatter` plot set by passing in `kind='scatter'` as plot argument. We will also need to pass in `x` and `y` keywords to specify the columns that go on the x- and the y-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_tot.plot(kind='scatter', x='year', y='total', figsize=(10, 6), color='darkblue')\n",
    "\n",
    "plt.title('Total Immigration to Canada from 1980 - 2013')\n",
    "plt.xlabel('Year')\n",
    "plt.ylabel('Number of Immigrants')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Notice how the scatter plot does not connect the datapoints together. We can clearly observe an upward trend in the data: as the years go by, the total number of immigrants increases. We can mathematically analyze this upward trend using a regression line (line of best fit). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "So let's try to plot a linear line of best fit, and use it to  predict the number of immigrants in 2015.\n",
    "\n",
    "Step 1: Get the equation of line of best fit. We will use **Numpy**'s `polyfit()` method by passing in the following:\n",
    "- `x`: x-coordinates of the data. \n",
    "- `y`: y-coordinates of the data. \n",
    "- `deg`: Degree of fitting polynomial. 1 = linear, 2 = quadratic, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "x = df_tot['year']      # year on x-axis\n",
    "y = df_tot['total']     # total on y-axis\n",
    "fit = np.polyfit(x, y, deg=1)\n",
    "\n",
    "fit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The output is an array with the polynomial coefficients, highest powers first. Since we are plotting a linear regression `y= a*x + b`, our output has 2 elements `[5.56709228e+03, -1.09261952e+07]` with the the slope in position 0 and intercept in position 1. \n",
    "\n",
    "Step 2: Plot the regression line on the `scatter plot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_tot.plot(kind='scatter', x='year', y='total', figsize=(10, 6), color='darkblue')\n",
    "\n",
    "plt.title('Total Immigration to Canada from 1980 - 2013')\n",
    "plt.xlabel('Year')\n",
    "plt.ylabel('Number of Immigrants')\n",
    "\n",
    "# plot line of best fit\n",
    "plt.plot(x, fit[0] * x + fit[1], color='red') # recall that x is the Years\n",
    "plt.annotate('y={0:.0f} x + {1:.0f}'.format(fit[0], fit[1]), xy=(2000, 150000))\n",
    "\n",
    "plt.show()\n",
    "\n",
    "# print out the line of best fit\n",
    "'No. Immigrants = {0:.0f} * Year + {1:.0f}'.format(fit[0], fit[1]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Using the equation of line of best fit, we can estimate the number of immigrants in 2015:\n",
    "```python\n",
    "No. Immigrants = 5567 * Year - 10926195\n",
    "No. Immigrants = 5567 * 2015 - 10926195\n",
    "No. Immigrants = 291,310\n",
    "```\n",
    "When compared to the actuals from Citizenship and Immigration Canada's (CIC) [2016 Annual Report](http://www.cic.gc.ca/english/resources/publications/annual-report-2016/index.asp), we see that Canada accepted 271,845 immigrants in 2015. Our estimated value of 291,310 is within 7% of the actual number, which is pretty good considering our original data came from United Nations (and might differ slightly from CIC data).\n",
    "\n",
    "As a side note, we can observe that immigration took a dip around 1993 - 1997. Further analysis into the topic revealed that in 1993 Canada introcuded Bill C-86 which introduced revisions to the refugee determination system, mostly restrictive. Further amendments to the Immigration Regulations cancelled the sponsorship required for \"assisted relatives\" and reduced the points awarded to them, making it more difficult for family members (other than nuclear family) to immigrate to Canada. These restrictive measures had a direct impact on the immigration numbers for the next several years."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Question**: Create a scatter plot of the total immigration from Denmark, Norway, and Sweden to Canada from 1980 to 2013?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 1: Get the data:\n",
    "   1. Create a dataframe the consists of the numbers associated with Denmark, Norway, and Sweden only. Name it **df_countries**.\n",
    "   2. Sum the immigration numbers across all three countries for each year and turn the result into a dataframe. Name this new dataframe **df_total**.\n",
    "   3. Reset the index in place.\n",
    "   4. Rename the columns to **year** and **total**.\n",
    "   5. Display the resulting dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "\\\\ # create df_countries dataframe\n",
    "df_countries = df_can.loc[['Denmark', 'Norway', 'Sweden'], years].transpose()\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # create df_total by summing across three countries for each year\n",
    "df_total = pd.DataFrame(df_countries.sum(axis=1))\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # reset index in place\n",
    "df_total.reset_index(inplace=True)\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # rename columns\n",
    "df_total.columns = ['year', 'total']\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # change column year from string to int to create scatter plot\n",
    "df_total['year'] = df_total['year'].astype(int)\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # show resulting dataframe\n",
    "df_total.head()\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 2: Generate the scatter plot by plotting the total versus year in **df_total**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "\\\\ # generate scatter plot\n",
    "df_total.plot(kind='scatter', x='year', y='total', figsize=(10, 6), color='darkblue')\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # add title and label to axes\n",
    "plt.title('Immigration from Denmark, Norway, and Sweden to Canada from 1980 - 2013')\n",
    "plt.xlabel('Year')\n",
    "plt.ylabel('Number of Immigrants')\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # show plot\n",
    "plt.show()\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Bubble Plots <a id=\"12\"></a>\n",
    "\n",
    "A `bubble plot` is a variation of the `scatter plot` that displays three dimensions of data (x, y, z). The datapoints are replaced with bubbles, and the size of the bubble is determined by the third variable 'z', also known as the weight. In `maplotlib`, we can pass in an array or scalar to the keyword `s` to `plot()`, that contains the weight of each point.\n",
    "\n",
    "**Let's start by analyzing the effect of Argentina's great depression**.\n",
    "\n",
    "Argentina suffered a great depression from 1998 - 2002, which caused widespread unemployment, riots, the fall of the government, and a default on the country's foreign debt. In terms of income, over 50% of Argentines were poor, and seven out of ten Argentine children were poor at the depth of the crisis in 2002. \n",
    "\n",
    "Let's analyze the effect of this crisis, and compare Argentina's immigration to that of it's neighbour Brazil. Let's do that using a `bubble plot` of immigration from Brazil and Argentina for the years 1980 - 2013. We will set the weights for the bubble as the *normalized* value of the population for each year."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 1: Get the data for Brazil and Argentina. Like in the previous example, we will convert the `Years` to type int and bring it in the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_can_t = df_can[years].transpose() # transposed dataframe\n",
    "\n",
    "# cast the Years (the index) to type int\n",
    "df_can_t.index = map(int, df_can_t.index)\n",
    "\n",
    "# let's label the index. This will automatically be the column name when we reset the index\n",
    "df_can_t.index.name = 'Year'\n",
    "\n",
    "# reset index to bring the Year in as a column\n",
    "df_can_t.reset_index(inplace=True)\n",
    "\n",
    "# view the changes\n",
    "df_can_t.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 2: Create the normalized weights. \n",
    "\n",
    "There are several methods of normalizations in statistics, each with its own use. In this case, we will use [feature scaling](https://en.wikipedia.org/wiki/Feature_scaling) to bring all values into the range [0,1]. The general formula is:\n",
    "\n",
    "<img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DV0101EN/labs/Images/Mod3Fig3FeatureScaling.png\" align=\"center\">\n",
    "\n",
    "where *`X`* is an original value, *`X'`* is the normalized value. The formula sets the max value in the dataset to 1, and sets the min value to 0. The rest of the datapoints are scaled to a value between 0-1 accordingly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# normalize Brazil data\n",
    "norm_brazil = (df_can_t['Brazil'] - df_can_t['Brazil'].min()) / (df_can_t['Brazil'].max() - df_can_t['Brazil'].min())\n",
    "\n",
    "# normalize Argentina data\n",
    "norm_argentina = (df_can_t['Argentina'] - df_can_t['Argentina'].min()) / (df_can_t['Argentina'].max() - df_can_t['Argentina'].min())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 3: Plot the data. \n",
    "- To plot two different scatter plots in one plot, we can include the axes one plot into the other by passing it via the `ax` parameter. \n",
    "- We will also pass in the weights using the `s` parameter. Given that the normalized weights are between 0-1, they won't be visible on the plot. Therefore we will:\n",
    "    - multiply weights by 2000 to scale it up on the graph, and,\n",
    "    - add 10 to compensate for the min value (which has a 0 weight and therefore scale with x2000)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Brazil\n",
    "ax0 = df_can_t.plot(kind='scatter',\n",
    "                    x='Year',\n",
    "                    y='Brazil',\n",
    "                    figsize=(14, 8),\n",
    "                    alpha=0.5,                  # transparency\n",
    "                    color='green',\n",
    "                    s=norm_brazil * 2000 + 10,  # pass in weights \n",
    "                    xlim=(1975, 2015)\n",
    "                   )\n",
    "\n",
    "# Argentina\n",
    "ax1 = df_can_t.plot(kind='scatter',\n",
    "                    x='Year',\n",
    "                    y='Argentina',\n",
    "                    alpha=0.5,\n",
    "                    color=\"blue\",\n",
    "                    s=norm_argentina * 2000 + 10,\n",
    "                    ax = ax0\n",
    "                   )\n",
    "\n",
    "ax0.set_ylabel('Number of Immigrants')\n",
    "ax0.set_title('Immigration from Brazil and Argentina from 1980 - 2013')\n",
    "ax0.legend(['Brazil', 'Argentina'], loc='upper left', fontsize='x-large')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The size of the bubble corresponds to the magnitude of immigrating population for that year, compared to the 1980 - 2013 data. The larger the bubble, the more immigrants in that year.\n",
    "\n",
    "From the plot above, we can see a corresponding increase in immigration from Argentina during the 1998 - 2002 great depression. We can also observe a similar spike around 1985 to 1993. In fact, Argentina had suffered a great depression from 1974 - 1990, just before the onset of 1998 - 2002 great depression. \n",
    "\n",
    "On a similar note, Brazil suffered the *Samba Effect* where the Brazilian real (currency) dropped nearly 35% in 1999. There was a fear of a South American financial crisis as many South American countries were heavily dependent on industrial exports from Brazil. The Brazilian government subsequently adopted an austerity program, and the economy slowly recovered over the years, culminating in a surge in 2010. The immigration data reflect these events."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Question**: Previously in this lab, we created box plots to compare immigration from China and India to Canada. Create bubble plots of immigration from China and India to visualize any differences with time from 1980 to 2013. You can use **df_can_t** that we defined and used in the previous example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 1: Normalize the data pertaining to China and India."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "\\\\ # normalize China data\n",
    "norm_china = (df_can_t['China'] - df_can_t['China'].min()) / (df_can_t['China'].max() - df_can_t['China'].min())\n",
    "-->\n",
    "\n",
    "<!--\n",
    "# normalize India data\n",
    "norm_india = (df_can_t['India'] - df_can_t['India'].min()) / (df_can_t['India'].max() - df_can_t['India'].min())\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Step 2: Generate the bubble plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "jupyter": {
     "outputs_hidden": false
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Double-click __here__ for the solution.\n",
    "<!-- The correct answer is:\n",
    "\\\\ # China\n",
    "ax0 = df_can_t.plot(kind='scatter',\n",
    "                    x='Year',\n",
    "                    y='China',\n",
    "                    figsize=(14, 8),\n",
    "                    alpha=0.5,                  # transparency\n",
    "                    color='green',\n",
    "                    s=norm_china * 2000 + 10,  # pass in weights \n",
    "                    xlim=(1975, 2015)\n",
    "                   )\n",
    "-->\n",
    "\n",
    "<!--\n",
    "\\\\ # India\n",
    "ax1 = df_can_t.plot(kind='scatter',\n",
    "                    x='Year',\n",
    "                    y='India',\n",
    "                    alpha=0.5,\n",
    "                    color=\"blue\",\n",
    "                    s=norm_india * 2000 + 10,\n",
    "                    ax = ax0\n",
    "                   )\n",
    "-->\n",
    "\n",
    "<!--\n",
    "ax0.set_ylabel('Number of Immigrants')\n",
    "ax0.set_title('Immigration from China and India from 1980 - 2013')\n",
    "ax0.legend(['China', 'India'], loc='upper left', fontsize='x-large')\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Thank you for completing this lab!\n",
    "\n",
    "This notebook was created by [Jay Rajasekharan](https://www.linkedin.com/in/jayrajasekharan) with contributions from [Ehsan M. Kermani](https://www.linkedin.com/in/ehsanmkermani), and [Slobodan Markovic](https://www.linkedin.com/in/slobodan-markovic).\n",
    "\n",
    "This notebook was recently revamped by [Alex Aklson](https://www.linkedin.com/in/aklson/). I hope you found this lab session interesting. Feel free to contact me if you have any questions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "This notebook is part of a course on **Coursera** called *Data Visualization with Python*. If you accessed this notebook outside the course, you can take this course online by clicking [here](http://cocl.us/DV0101EN_Coursera_Week2_LAB2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "editable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<hr>\n",
    "\n",
    "Copyright &copy; 2019 [Cognitive Class](https://cognitiveclass.ai/?utm_source=bducopyrightlink&utm_medium=dswb&utm_campaign=bdu). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}