parser.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parsing Jupyter Notebooks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nbformat\n",
    "import io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading Notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metadata Params\n",
    "\n",
    "#### Required\n",
    "\n",
    "* author = set[ author1, author2, ... ]\n",
    "* level = pick1: beginner, novice, intermediate, advanced\n",
    "* duration = 45 (implicit: minutes)\n",
    "\n",
    "#### Derived\n",
    "* libraries = (compiled from cell)\n",
    "* git-info = dict{version; source; sha1; last-modified-date}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'cells': [{'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '<div style=\"text-align:center\">\\n<h1>Week 1: Code Camp Recap</h1>\\n<h2>7SSG2059 Geocomputation 2018/19</h2>\\n</div>'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}, 'nbpresent': {'id': 'cada64b6-f8c9-4dbb-9ab1-b5617d0cc16a'}}, 'source': '## Learning Outcomes\\n\\nThis practical gives you an opportunity to improve your understanding of the initial notebooks (2-5) in Code Camp. Exercises and questions will help you develop that understanding, and you can of course ask questions of lecturers and GTAs.  \\n\\nBy the end of this practical you should:\\n-  Have a better appreciation for what the Python programming language is\\n-  Understand the types of error (syntax, logical, semantic)\\n-  Understand variables and variable assignment\\n-  Have a better appreciation of data types, basic mathematic operators  and simple expressions\\n-  Understand conditional and nested execution\\n\\nUnderstanding code camp notebooks 2-5 is important as Quiz 1 will cover the content of these notebooks.\\n'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': 'This document assumes you have created a working directory as described here: http://bit.ly/WorkingDirectory'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '# 1. Python Errors (Code Camp Notebook 4)'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### Why are we starting with errors?\\n\\nBecause everyone makes mistakes. In programming, it is very easy to make errors. Because computers are stupid and do EXACTLY what we tell them, any small mistake can cause a code chunk or a program to fail to run. These errors in a program are often called bugs. There are other errors that mean that although a program will run, it may not do what you think it does. \\n\\nSo we can classify a few general types of error: \\n1.\\t**syntax errors** are when the ‘grammar’ rules of Python (known as the syntax) have been violated, making it impossible for the code to be parsed (i.e. understood by python); \\n2.\\t**logic errors**  have good syntax but there is an error in the order of statements or how they relate to one another;\\n3.\\t**semantic errors** make programs do something other than what the programmer intended.\\n\\n_“A good example of a logic error might be, “take a drink from your water bottle, put it in your backpack, walk to the library, and then put the top back on the bottle.”_ Severance (2014, p.14)\\n'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '#### TASK 1.1\\n\\nRun through the information and activites in Notebook 4 in the [Python Code Camp](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/) to refresh your understanding of Errors in Python. When you have done so return to this notebook to continue. Remember to right-click the link **save the notebook to file**, _then_ open it using Jupyter.   '}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '## 2. Back to Basics (Code Camp 2)'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': \"Before we go on, let's just pause to check we have a good hold on the basics of Python. \"}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': \"### TASK 2.1\\n\\nRun through the information and activites in Notebook 3 in the [Python Code Camp](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/) to refresh your understanding of the basics in Python. **Don't do the Code (general exercises)** - stop there and come back here to continue with this practical and the exercises below. \"}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '## Exercises'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### a) Variables and Assignment\\n\\nWithout running the code below, calculate what the values of `x`, `y` and `z` would be after all lines of code in the code block have been executed.'}, {'cell_type': 'code', 'execution_count': None, 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'outputs': [], 'source': 'x = 12\\ny = 10\\n\\nz = x + y\\nx = x + y\\ny = z + y'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': 'Once you have calculated what you think `x`, `y` and `z` are, add print statements to the code above and run it to check. Make sure you understand the results you find. **Ask someone if you need help to understand.**'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### b) Operator Precedence\\n\\nWithout running the script, calculate what the values of `x`, `y` and `z` would be after all lines of code in the code block have been executed.'}, {'cell_type': 'code', 'execution_count': None, 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'outputs': [], 'source': 'x = 1 + 2 * 3 / 4\\ny = (1 + 2) * (3 / 4)\\nz = 1 + 2 * (3 / 4)\\n\\nprint(x)\\nprint(y)\\nprint(z)'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': 'Once you have calculated what you think `x`, `y` and `z` are, add print statements to the code above and run it to check.'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### c) Integer Division\\n\\nWithout running the script, calculate what the values of `x`, `y` and `z` would be after all lines of code in the code block have been executed.'}, {'cell_type': 'code', 'execution_count': None, 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'outputs': [], 'source': 'width = 17\\nheight = 12.0\\n\\nx = width / 2\\ny = width / 2.0\\nz = height / 3'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': 'Once you have calculated what you think `x`, `y` and `z` are, add print statements to the code above and run it to check.'}, {'cell_type': 'markdown', 'metadata': {}, 'source': '**Save your notebook** [File -> Save and Checkpoint] to save your answers to disk!'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}, 'nbpresent': {'id': 'f730ab33-247e-4792-9e02-ab29c9f00c14'}}, 'source': '# 3. Conditionals (Code Camp 4)\\n\\n'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': 'This section of the practical (re)covers the concepts examined in Notebook 5 of [Python Code Camp](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/) but also add new questions and Tasks for you to tackle.  \\n\\nIn particular, this section looks at how to compare stuff (like numerical quantities but also words and strings) via *Comparison Operators* and how logical concepts like: \" `if` this condition is `True`, then do something...\" can be expressed in Python. '}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '## Conditional Execution'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### Comparison Operators'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### TASK 3.1\\n\\nRun through the information and activites in the **Comparisons** section of Notebook 5 in the [Python Code Camp](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/). Remember to right-click the link **save the notebook to file**, _then_ open it using Jupyter. \\n\\n**Stop** when you reach the **Conditions** section and return here (i.e. the last item to look at is _Testing for Inequality_.\\n\\n***'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': 'Welcome back. \\n\\nComparison operators are often used in conditional execution with _Boolean_ expressions (which produce True or False results). In conditional execution different lines of code are only executed under certain conditions (specified by the Boolean expression). For example, the code in the image below will only execute when the `hours` variable is greater than zero (as the output of the expression on line 2 is `true` in this circumstance).\\n\\n<img src=https://kingsgeocomputation.files.wordpress.com/2016/09/geocompweek2_task3_1a.jpg width = \"389\">'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### TASK 3.2\\nCopy the code from the image above into the code cell below, then add two lines of code so that `Hours are less than 10` is printed if that condition is `True`. Run your code three times, once each with the following initial values (on line 1):\\n1.\\t`hours = 10`\\n2.\\t`hours = 2`\\n3.\\t`hours = 0`\\n\\nIf you added your lines of code correctly, your output when running the code for the above initial values should be as follows:\\n1.\\t`Hours are greater than zero` \\n2.\\t`Hours are greater than zero`<br/>\\n    `Hours are less than 10`\\n3.\\t`Hours are less than 10`\\n'}, {'cell_type': 'code', 'execution_count': None, 'metadata': {}, 'outputs': [], 'source': ''}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### TASK 3.3\\nCopy your completed code from the previous task into the code cell below. Then add a new line so that when the initial value of `hours = 12` (on line 1) BOTH conditionals will execute (i.e. both `Hours are greater than zero` and `Hours are less than 10` are printed). \\n\\n_Hint: the line of code to add could read something like `hours = hours – 3` and will be indented._\\n'}, {'cell_type': 'code', 'execution_count': None, 'metadata': {}, 'outputs': [], 'source': ''}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### TASK 3.4\\nIn the code cell below, enter code to execute the flow chart shown in the image below. You will need to assign values to `a` and `b` on lines 1 and 2 before using the flow chart to complete the code such that it produces the following:\\n1.\\tWhen `a = 2` and `b = 2` four lines of output will be written\\n2.\\tWhen `a = 1` and `b = 2` one line of output will be written\\n\\n<img src=https://kingsgeocomputation.files.wordpress.com/2016/09/geocompweek2_task3_3.jpg width = 600>'}, {'cell_type': 'code', 'execution_count': None, 'metadata': {}, 'outputs': [], 'source': ''}, {'cell_type': 'markdown', 'metadata': {}, 'source': '**Save your notebook** [File -> Save and Checkpoint] to save your answers to disk!'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': \"### TASK 3.5\\nNow that you've had some practice with comparison operators, return to where you stopped in Notebook 5 in the [Python Code Camp](https://kingsgeocomputation.org/teaching/code-camp/code-camp-python/lessons/) (the section on **Conditions** after  _Testing for Inequality_). Run through all the remaining material in the notebook (**Conditions** and **Boolean Logic** sections), but **don't** do the _Code (Applied Geo-Example)_ at the end. Return here when you're familiar with the materials and continue with the tasks below. \\n\\n***\"}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': \"### Logical Operators\\nWelcome back. \\nAs you've just been reading, logical operators can be combined with expressions containing comparison operators to produce more complicated conditional execution statements. The main logical operators are `and`, `or`, and `not`. How these logical operators work is shown in the lecture slides for this week. Review those examples before attempting the tasks below.\"}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### TASK 3.6\\nRead the code in the code cell below. '}, {'cell_type': 'code', 'execution_count': None, 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'outputs': [], 'source': 'if x > 0 and x < 5:\\n    print(\"Joe\")\\n    \\nif x > 0 or x < 5:\\n    print(\"Aled\")\\n    \\nif not(x > 0):\\n    print(\"Sarah\")'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': 'Assume that the code executes for a given value of `x` and answer the following questions. Only test the code **AFTER** you have worked out the answers and typed them below.\\n\\n1.\\tWhat names are name(s) are printed when `x = 5`? \\n2.\\tWhat value(s) can `x` be when the names `Joe` and `Aled` are printed?  \\n3.\\tWhat name(s) are printed when `x = -1`?\\n4.\\tIs there any value for which all three names will be printed?\\n'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': \"**Answers** _Edit this cell!_\\n\\nTo edit this cell just double click on it (right _here_). When you're finished, click the _run cell_ button in the toolbar just below the menu near the top of the window.\\n\\n1.\\n\\n2.\\n\\n3.\\n\\n4.\"}, {'cell_type': 'markdown', 'metadata': {}, 'source': '**Save your notebook** [File -> Save and Checkpoint] to save your answers to disk!'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### Alternative Execution\\nThe reserved word `else` can be used with the reserved word `if` to produce what is known as an ‘alternative execution’. It is ‘alternative’ because there are two possibilities and a condition determines which alternative gets executed. In an alternative execution, since a condition must either be `true` or `false` exactly one of the alternatives will be executed. The alternatives are called branches, because they are branches in the flow of execution.'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### TASK 3.7\\nStudy the flow chart below.\\n\\n<img src=https://kingsgeocomputation.files.wordpress.com/2016/09/geocompweek2_task3_5.jpg width = 600>\\n\\nIn the code cell below, enter code to execute the alternative execution shown by the flow chart. Check your code works by trying different values of `x` and comparing the outcomes. Also check you understand why the Boolean expression using the percent symbol `%` allows us to evaluate odd vs even numbers. '}, {'cell_type': 'code', 'execution_count': None, 'metadata': {}, 'outputs': [], 'source': ''}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### Nested Conditional\\nConditional statements can be nested within one another. That is, once one conditional statement has been evaluated others may subsequently be evaluated (or not) depending on the result of the initial conditional statement. The code below shows an example of this. '}, {'cell_type': 'code', 'execution_count': None, 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'outputs': [], 'source': 'if x != y:                                 #line 1\\n    print(\"x is not equal to y\")\\n    \\n    if(x > y):                             #line 4\\n        print(\"x is greater than y\")\\n        \\n    else:                                  #line 7\\n        print(\"x is less than y\")\\n\\nelse:\\n    print(\"insert conclusion here\")'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': 'Note how the indentation is useful to work out which ‘level’ the code is operating at. In the code above, lines 4 and 7 are at the same indentation meaning that both will be skipped if the initial condition (on line 1) is `False`. \\n\\n### TASK 3.8\\nTo check you understand how the code above works:\\n1.\\tDraw a flow chart to illustrate the flow of execution (on paper or use LibreOffice Impress or Microsoft PowerPoint)\\n2.\\tChange `insert conclusion here` in the code above to a string that explains the condition of `x` and `y`\\n3.\\tFor `x = 2` and `y = 3`, type below what line(s) will be output \\n'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '**Answer** _Edit this cell!_\\n\\n3. '}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': 'Show the flow chart you have drawn to one of the GTAs or a lecturer (or post to Slack if after the practical) to check if you have got it right. \\n\\nGreat! You should now have a pretty good understanding of how conditional and logical operators work. This understanding will be handy in future as we work through other computational concepts. \\n\\n**Save your notebook** [File -> Save and Checkpoint] to save your answers to disk!'}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '## Exercises'}, {'cell_type': 'markdown', 'metadata': {}, 'source': 'The answers to the following exercises will be provided next week. '}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### 1. Alterative Execution\\nIn the code cell below, enter code that reports whether person A is older than person B or not. Whole numbers should be used as ages in years and these ages should be specified on the first two lines of code. Either “Person A is older than Person B” or “Person A is not older than Person B” should be reported to the used.'}, {'cell_type': 'code', 'execution_count': None, 'metadata': {}, 'outputs': [], 'source': ''}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### 2. Conditional Execution\\nCopy your final code from the previous exercise into the code block below. Now build on this code to check if Person A is older than Person B, but also whether they are the same age. One of following three responses should be reported to the user:\\n1.\\t“Person A is older than Person B”\\n2.\\t“Person B is older than Person A”\\n3.\\t“Person A and Person B are the same age”'}, {'cell_type': 'code', 'execution_count': None, 'metadata': {}, 'outputs': [], 'source': ''}, {'cell_type': 'markdown', 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'source': '### 3. Nested Execution\\nCopy your final code from the previous exercise into the code block below. Build on this code with a nested execution structure to report if Person A is much younger, a little younger, the same age, a little older or much older than Person B. Write your code with a nested structure so that first it evaluates whether one person is older than the other, and then, if so, whether there is a large difference in age or a small difference in age. You should create a variable to specify what the number of years is that indicates a ‘large difference’ in age. One of the following five responses should be reported to the user:\\n1.\\t“Person A is much younger than Person B” [if age difference > ‘large difference’]\\n2.\\t“Person A is a little younger than Person B” [if age difference <= ‘large difference’]\\n3.\\t“Person A is the same age as Person B”\\n4.\\t“Person A is a little older than Person B” [if age difference <= ‘large difference’]\\n5.\\t“Person A is a much older than Person B” [if age difference > ‘large difference’]\\n'}, {'cell_type': 'code', 'execution_count': None, 'metadata': {'geopyter': {'Contributors': 'James Millington (james.millington@kcl.ac.uk)', 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}}}, 'outputs': [], 'source': ''}, {'cell_type': 'markdown', 'metadata': {}, 'source': '\\n### Credits!\\n\\n#### Contributors:\\nThe following individuals have contributed to these teaching materials: James Millington (james.millington@kcl.ac.uk)\\n\\n#### License\\nThese teaching materials are licensed under a mix of the MIT and CC-BY licenses...\\n\\n#### Acknowledgements:\\nSupported by the [Royal Geographical Society](https://www.rgs.org/HomePage.htm) (with the Institute of British Geographers) with a Ray Y Gildea Jr Award.\\n\\n#### Potential Dependencies:\\nThis notebook may depend on the following libraries: None\\n'}], 'metadata': {'anaconda-cloud': {}, 'geopyter': {'Contributors': ['James Millington (james.millington@kcl.ac.uk)'], 'git': {'active_branch': 'master', 'author.name': 'Jon Reades', 'authored_date': '2017-08-17 19:06:58', 'committed_date': '2017-08-17 19:06:58', 'committer.name': 'Jon Reades', 'sha': '5e3b396ae18a982d693c4bfd86c721c7e1e21051'}, 'libs': {}}, 'kernelspec': {'display_name': 'Python [conda env:gsa2018]', 'language': 'python', 'name': 'conda-env-gsa2018-py'}, 'language_info': {'codemirror_mode': {'name': 'ipython', 'version': 3}, 'file_extension': '.py', 'mimetype': 'text/x-python', 'name': 'python', 'nbconvert_exporter': 'python', 'pygments_lexer': 'ipython3', 'version': '3.6.6'}}, 'nbformat': 4, 'nbformat_minor': 1, 'base_dir': 'https://raw.githubusercontent.com/kingsgeocomp/geocomputation/master/Practical-01-Code%20Camp%20Recap.ipynb'}\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import os \n",
    "import requests\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "def read_nb(nb_src, ext=True):\n",
    "    \"\"\"\n",
    "    Read a notebook file and return a notebook object.\n",
    "\n",
    "    Parameters\n",
    "    ==========\n",
    "    nb: String\n",
    "        Path to the notebook file; if the path does not end\n",
    "        in '.ipynb' then this will be appended unless you\n",
    "        override this by setting the 'ext' to False.\n",
    "    ext: boolean\n",
    "        Defaults to True, meaning that the '.ipynb'\n",
    "        extension will be automatically added. If you do not\n",
    "        want this behaviour for some reason then set ext to False.\n",
    "\n",
    "    Returns\n",
    "    =======\n",
    "    An object of class nbformat.notebooknode.NotebookNode\n",
    "    \"\"\"\n",
    "    m = re.search('^(.+geopyter)', os.getcwd(), re.IGNORECASE)\n",
    "    if m:\n",
    "        base_dir = m.group(0)\n",
    "    else:\n",
    "        base_dir = '.'\n",
    "    \n",
    "    # Append file extension if missing and ext is True\n",
    "    if not nb_src.endswith('.ipynb') and ext is True:\n",
    "        nb_src += '.ipynb'\n",
    "    \n",
    "    path = ''\n",
    "    loc = urlparse(nb_src)\n",
    "    if loc.scheme in ('http','ftp','https'):\n",
    "        # This doesn't support credentialed access at this time\n",
    "        # -- partly because it's a pain, and partly because you\n",
    "        # should be sharing... :-)\n",
    "        \n",
    "        nbd = requests.get(nb_src).text\n",
    "        \n",
    "        # Read-only in UTF-8, note NO_CONVERT.\n",
    "        with io.StringIO(r) as f:\n",
    "            nb = nbformat.read(f, nbformat.NO_CONVERT)\n",
    "            \n",
    "    elif loc.path is not None:\n",
    "        if os.path.exists(nb_src):\n",
    "            path = nb_src\n",
    "        elif os.path.exists(os.path.join(base_dir,\"atoms\",nb_src)):\n",
    "            path = os.path.join(base_dir,\"atoms\",nb_src)\n",
    "        else:\n",
    "            print(\"Doesn't look like there's a file at: \" + nb_src)\n",
    "        \n",
    "        # Read-only in UTF-8, note NO_CONVERT.\n",
    "        with open(nb, 'r', encoding='utf8') as f:\n",
    "            nb = nbformat.read(f, nbformat.NO_CONVERT)\n",
    "        \n",
    "    else:\n",
    "        print(\"Don't know what to do with this type of path: \" + nb_src)\n",
    "    \n",
    "    nb.base_dir = nb_src\n",
    "    \n",
    "    return nb\n",
    "\n",
    "nburl = 'https://raw.githubusercontent.com/kingsgeocomp/geocomputation/master/Practical-01-Code%20Camp%20Recap.ipynb'\n",
    "t = read_nb(nburl)\n",
    "print(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "\n",
      " \"cells\": [\n",
      "\n",
      "  {\n",
      "\n"
     ]
    }
   ],
   "source": [
    "r = requests.get(nburl).text\n",
    "#r.text.split(\"\\n\")\n",
    "\n",
    "f = io.StringIO(r)\n",
    "\n",
    "print(f.readline())\n",
    "\n",
    "print(f.readline())\n",
    "\n",
    "print(f.readline())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m \u001b[0mnbformat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mas_version\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Read a notebook from a file as a NotebookNode of the given version.\n",
       "\n",
       "The string can contain a notebook of any version.\n",
       "The notebook will be returned `as_version`, converting, if necessary.\n",
       "\n",
       "Notebook format errors will be logged.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "fp : file or str\n",
       "    A file-like object with a read method that returns unicode (use\n",
       "    ``io.open()`` in Python 2), or a path to a file.\n",
       "as_version: int\n",
       "    The version of the notebook format to return.\n",
       "    The notebook will be converted, if necessary.\n",
       "    Pass nbformat.NO_CONVERT to prevent conversion.\n",
       "\n",
       "Returns\n",
       "-------\n",
       "nb : NotebookNode\n",
       "    The notebook that was read.\n",
       "\u001b[0;31mFile:\u001b[0m      /anaconda3/lib/python3.6/site-packages/nbformat/__init__.py\n",
       "\u001b[0;31mType:\u001b[0m      function\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?nbformat.read"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def write_nb(nb, fn):\n",
    "    \"\"\"\n",
    "    Write a notebook to the path specified.\n",
    "    \n",
    "    Parameters\n",
    "    ==========\n",
    "    nb: nbformat.notebooknode.NotebookNode\n",
    "        A notebook object to write to disk.\n",
    "    fn: String\n",
    "        Path to which you want the notebook written. _Note:_ \n",
    "        for simplicity's sake this will automatically append \n",
    "        '.ipynb' to the filename; however we recommend that \n",
    "        you not get lazy and rely on this feature since it may\n",
    "        go away in the future.\n",
    "    \n",
    "    Returns\n",
    "    =======\n",
    "    Void.\n",
    "    \"\"\"\n",
    "    \n",
    "    m = re.search('^(.+geopyter)', os.getcwd(), re.IGNORECASE)\n",
    "    if m:\n",
    "        self.base_dir = m.group(0)\n",
    "    else:\n",
    "        self.base_dir = '.'\n",
    "\n",
    "    path = ''\n",
    "    loc = urlparse(ipynb)\n",
    "    if loc.scheme in ('http','ftp','https'):\n",
    "        print(\"Haven't implemented remote files yet\")\n",
    "        #path = requests.get(loc)\n",
    "    elif loc.path is not None:\n",
    "        if os.path.exists(ipynb):\n",
    "            path = ipynb\n",
    "        elif os.path.exists(os.path.join(self.base_dir,\"atoms\",ipynb)):\n",
    "            path = os.path.join(self.base_dir,\"atoms\",ipynb)\n",
    "        else:\n",
    "            print(\"Doesn't look like there's a file at: \" + ipynb)\n",
    "    else:\n",
    "        print(\"Don't know what to do with this type of path info: \" + ipynb)\n",
    "    \n",
    "    # Append file extension\n",
    "    if not fn.endswith('.ipynb'):\n",
    "        fn += '.ipynb'\n",
    "    \n",
    "    # Write raw notebook content\n",
    "    with io.open(fn, 'w', encoding='utf8') as f:\n",
    "        nbformat.write(nb, f, nbformat.NO_CONVERT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "def get_nb_structure(nb):\n",
    "    cell_types = defaultdict(list)\n",
    "    for i, cell in enumerate(nb['cells']):\n",
    "        cell_types[cell.cell_type].append(i)\n",
    "    return cell_types\n",
    "\n",
    "def dump_nb(nb, cells=5, lines=5):\n",
    "    \"\"\"\n",
    "    Dump content of a notebook to STDOUT to aid in debugging.\n",
    "    \n",
    "    Parameters\n",
    "    ==========\n",
    "    nb: nbformat.notebooknode.NotebookNode\n",
    "        A notebook object from which to dump content.\n",
    "    cells: int\n",
    "        Select an arbitrary number of cells to output. Defaults to 5.\n",
    "    lines: int\n",
    "        Select an arbitrary number of lines from each cell to output. Defaults to 5.\n",
    "    \n",
    "    Returns\n",
    "    =======\n",
    "    Void.\n",
    "    \"\"\"\n",
    "    \n",
    "    # For the cell-range specified\n",
    "    for c in xrange(0, cells):\n",
    "        \n",
    "        # Check we still have cells to read\n",
    "        if c < len(nb.cells):\n",
    "            \n",
    "            # And dump the contents to STDOUT\n",
    "            print(\"====== \" + nb.cells[c]['cell_type'] + \" ======\")\n",
    "            src = nb.cells[c]['source'].splitlines()\n",
    "            if len(src) > lines:\n",
    "                print('\\n'.join(src[0:lines]))\n",
    "                print(\"...\")\n",
    "            else:\n",
    "                print(nb.cells[c]['source'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from git import Repo\n",
    "\n",
    "def gitter(path='.'):\n",
    "    \"\"\"\n",
    "    Try to collect GitHub information to use in tracking \n",
    "    authorship contributions and allow specification of\n",
    "    particular versions of notebooks.\n",
    "    \n",
    "    Parameters\n",
    "    ==========\n",
    "    path: String\n",
    "        The path to a GitHub repository. Defaults to '.'\n",
    "    \n",
    "    Returns\n",
    "    =======\n",
    "    rp: dict\n",
    "        A dictionary containing relevant git metadata\n",
    "    \"\"\"\n",
    "    repo = Repo(path)\n",
    "    \n",
    "    rp = {}\n",
    "    \n",
    "    rp['active_branch'] = str(repo.active_branch)\n",
    "    \n",
    "    hc = repo.head.commit\n",
    "    rp['author.name'] = hc.author.name\n",
    "    rp['authored_date'] = datetime.datetime.fromtimestamp(hc.authored_date).strftime('%Y-%m-%d %H:%M:%S')\n",
    "    rp['committer.name'] = hc.committer.name\n",
    "    rp['committed_date'] = datetime.datetime.fromtimestamp(hc.committed_date).strftime('%Y-%m-%d %H:%M:%S')\n",
    "    rp['sha'] = hc.hexsha\n",
    "    \n",
    "    return rp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'author.name': u'jreades', 'committer.name': u'jreades', 'sha': u'1bc766046cf60d798d5bdef86ab8fb916eb781bf', 'committed_date': '2017-02-22 11:10:45', 'active_branch': 'master', 'authored_date': '2017-02-22 11:10:45'}\n"
     ]
    }
   ],
   "source": [
    "print(gitter())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import importlib\n",
    "def find_libraries(nb):\n",
    "    \"\"\"\n",
    "    Utility function to find libraries imported by notebooks \n",
    "    and assemble them into a group for reporting and testing\n",
    "    purposes.\n",
    "    \n",
    "    Parameters\n",
    "    ==========\n",
    "    nb: nbformat.notebooknode.NotebookNode\n",
    "        A notebook object to search for import statements\n",
    "        \n",
    "    Returns\n",
    "    =======\n",
    "    libs: Set\n",
    "        A set containing the libraries imported by the notebook\n",
    "    \"\"\"\n",
    "    \n",
    "    # Find and classify the cells by type [code, markdown]\n",
    "    cell_types = get_nb_structure(nb)\n",
    "    \n",
    "    libs  = set()\n",
    "    vlibs = {}\n",
    "    \n",
    "    # Iterate over the code cell-types\n",
    "    for c in cell_types['code']:\n",
    "        try:\n",
    "            #print(\"-\" * 25)\n",
    "            #print(nb.cells[c]['source'])\n",
    "            \n",
    "            # Convert the code into a block of lines\n",
    "            block = nb.cells[c]['source'].splitlines()\n",
    "            # Loop over the lines looking for import-type statements\n",
    "            for l in block: \n",
    "                m = re.match(\"(?:from|import) (\\S+)\", l)\n",
    "                if m:\n",
    "                    libs.add(m.group(1))\n",
    "        except IndexError: #Catch index error (not sure where this comes from)\n",
    "            pass\n",
    "    \n",
    "    # Try to get the versions in use on the machine\n",
    "    for l in libs: \n",
    "        l = l.split('.')[0]\n",
    "        #print(\"Checking version of \" + l)\n",
    "        mod = importlib.import_module(l)\n",
    "        ver = None\n",
    "        try:\n",
    "            ver = mod.__version__\n",
    "        except AttributeError:\n",
    "            try: \n",
    "                ver = mod.version\n",
    "            except AttributeError:\n",
    "                print(\"Unable to determine version for: \" + l)\n",
    "                print(\"Currently we check <module>.__version__ and <moduled>.version\")\n",
    "                pass\n",
    "        vlibs[l] = ver\n",
    "    return vlibs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{u'pysal': '1.13.0', u'scipy': '0.17.1', u'numpy': '1.11.1', u'seaborn': '0.7.1', u'sklearn': '0.17.1'}\n"
     ]
    }
   ],
   "source": [
    "#source_nb = 'atoms/foundations/Dictionaries.ipynb'\n",
    "source_nb = 'atoms/visualization/choropleth_classification.ipynb'\n",
    "inb = read_nb(source_nb)\n",
    "print(find_libraries(inb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def write_metadata(nb, nm, val, namespace=unicode('geopyter')):\n",
    "    \"\"\"\n",
    "    Add or append metadata values to the geopyter parameter.\n",
    "    \n",
    "    Parameters\n",
    "    ==========\n",
    "    nb: nbformat.notebooknode.NotebookNode\n",
    "        A notebook object to which to add Geopyter metadata.\n",
    "    nm: String\n",
    "        The name of the key within the Geopyter dictionary that we want to update.\n",
    "    val: String, List, Dictionary\n",
    "        The value to associate with the key.\n",
    "    \n",
    "    Returns\n",
    "    =======\n",
    "    Void.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Check for the namespace in the notebook metadata\n",
    "    if not namespace in nb.metadata:\n",
    "        nb.metadata[namespace] = {}\n",
    "    \n",
    "    # And write it\n",
    "    nb.metadata[namespace][nm] = val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_metadata(nb, nm, namespace=unicode('geopyter')):\n",
    "    \"\"\"\n",
    "    Retrieve metadata values from the geopyter parameter.\n",
    "    \n",
    "    Parameters\n",
    "    ==========\n",
    "    nb: nbformat.notebooknode.NotebookNode\n",
    "        A notebook object to which to add Geopyter metadata.\n",
    "    nm: String\n",
    "        The name of the key within the Geopyter dictionary that we want to retrieve.\n",
    "    \n",
    "    Returns\n",
    "    =======\n",
    "    Void.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Check for the namespace in the notebook metadata\n",
    "    if not nb.metadata.has_key(namespace):\n",
    "        nb.metadata[namespace] = {}\n",
    "    \n",
    "    # And write it\n",
    "    nb.metadata[namespace][nm] = val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 315,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def read_user_metadata(nb):\n",
    "    src = nb.cells[0]['source']\n",
    "    #print(src)\n",
    "    \n",
    "    meta = {}\n",
    "    \n",
    "    if not re.match(\"\\# \\w+\", src):\n",
    "        print(\"The first cell should be of level h1 and contain a bulleted list of metadata.\")\n",
    "    \n",
    "    for l in src.splitlines():\n",
    "        m = re.match(\"- ([^\\:]+?)\\: (.+)\", l)\n",
    "        if m is not None:\n",
    "            val = m.group(2).split(';')\n",
    "            if len(val)==1:\n",
    "                val = val[0]\n",
    "            meta[m.group(1)] = val\n",
    "\n",
    "    return meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 313,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#source_nb = 'atoms/foundations/Dictionaries.ipynb'\n",
    "source_nb = 'atoms/visualization/choropleth_classification.ipynb'\n",
    "inb = read_nb(source_nb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 317,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for (key, val) in read_user_metadata(inb).iteritems():\n",
    "    write_metadata(inb, key, val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 319,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "write_metadata(inb, unicode('libraries'), find_libraries(inb))\n",
    "write_metadata(inb, unicode('git'), gitter())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'nbformat_minor', u'cells', u'nbformat', u'metadata']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inb.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 320,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'anaconda-cloud': {},\n",
       " u'geopyter': {u'Contributors': [u'Serge Rey (srey@asu.edu)',\n",
       "   u' Jon Reades (jonathan.reades@kcl.ac.uk)'],\n",
       "  u'Duration': u'2 hours',\n",
       "  u'Keywords': [u'choropleth',\n",
       "   u' mapping',\n",
       "   u' visualization',\n",
       "   u' pysal',\n",
       "   u' classifiers'],\n",
       "  u'Level': u'Novice',\n",
       "  u'git': {'active_branch': 'master',\n",
       "   'author.name': u'jreades',\n",
       "   'authored_date': '2017-02-22 11:12:08',\n",
       "   'committed_date': '2017-02-22 11:12:08',\n",
       "   'committer.name': u'jreades',\n",
       "   'sha': u'eda8ee306bed2ab3120b83b6659fa72848d1274c'},\n",
       "  u'libraries': {u'numpy': '1.11.1',\n",
       "   u'pysal': '1.13.0',\n",
       "   u'scipy': '0.17.1',\n",
       "   u'seaborn': '0.7.1',\n",
       "   u'sklearn': '0.17.1'}},\n",
       " u'kernelspec': {u'display_name': u'Python [Root]',\n",
       "  u'language': u'python',\n",
       "  u'name': u'Python [Root]'},\n",
       " u'language_info': {u'codemirror_mode': {u'name': u'ipython', u'version': 2},\n",
       "  u'file_extension': u'.py',\n",
       "  u'mimetype': u'text/x-python',\n",
       "  u'name': u'python',\n",
       "  u'nbconvert_exporter': u'python',\n",
       "  u'pygments_lexer': u'ipython2',\n",
       "  u'version': u'2.7.12'}}"
      ]
     },
     "execution_count": 320,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inb.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write_nb(inb, 'test-metadata.ipynb')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "====== markdown ======\n",
      "# Notebook-6: Dictionaries\n",
      "====== markdown ======\n",
      "### Lesson Content \n",
      "\n",
      "Welcome back to the fifth Code Camp notebook! In this lesson we'll contiune our exploration of more advanced data structures. Last time we took a peek at a way to represent ordered collections of items via **lists**.\n",
      "\n",
      "This time we'll use **dictionaries** to create collections of unordered items (this is just an easy distinction - there's much more to it - but it's a good way to start wrapping your head around the subject).\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "dump_nb(inb, cells=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "c0 = snb.cells[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nbformat.notebooknode.NotebookNode"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(c0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'source', u'cell_type', u'metadata']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c0.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'markdown'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c0['cell_type']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'## Introduction\\n\\n* what is classification\\n* role in choropleth mapping\\n* explore classification using PySAL'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c0['source']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'collapsed': True}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c0['metadata']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of cells in  this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "def get_structure(cells):\n",
    "    cell_types = defaultdict(list)\n",
    "    for i, cell in enumerate(cells):\n",
    "        cell_types[cell.cell_type].append(i)\n",
    "    return cell_types\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cell_types = get_structure(snb.cells)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'code', u'markdown']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cell_types.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cell Type: code\t 38 cells\n",
      "Cell Type: markdown\t 11 cells\n"
     ]
    }
   ],
   "source": [
    "for ct, cells in cell_types.items():\n",
    "    print('Cell Type: %s\\t %d cells'% (ct, len(cells)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "code_cell_idx = cell_types['code'][0]\n",
    "code_cell_idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'cell_type': u'code',\n",
       " u'execution_count': 3,\n",
       " u'metadata': {u'collapsed': False},\n",
       " u'outputs': [{u'data': {u'text/plain': u'{\\'description\\': \\'Mexican states regional income 1940-2000\\',\\n \\'explanation\\': [\\'Data used in   Rey, S.J. and M.L.  Sastre Gutierrez. (2010) \"Interregional inequality\\',\\n  \\'dynamics in Mexico.\" Spatial Economic Analysis, 5: 277-298\\',\\n  \\'* mexico.csv: attribute data\\',\\n  \\'* mexico.gal: spatial weights in GAL format\\',\\n  \\'Polygon data, n=32, k=13\\'],\\n \\'name\\': \\'mexico\\'}'},\n",
       "   u'execution_count': 3,\n",
       "   u'metadata': {},\n",
       "   u'output_type': u'execute_result'}],\n",
       " u'source': u\"ps.examples.explain('mexico')\"}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snb.cells[code_cell_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mkd_cell_idx = cell_types['markdown'][0]\n",
    "mkd_cell_idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'cell_type': u'markdown',\n",
       " u'metadata': {u'collapsed': True},\n",
       " u'source': u'## Introduction\\n\\n* what is classification\\n* role in choropleth mapping\\n* explore classification using PySAL'}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snb.cells[mkd_cell_idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Turning Output Cells OFF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def remove_outputs(nb):\n",
    "    \"\"\"Set output attribute of all code cells to be empty\"\"\"\n",
    "    for cell in nb.cells:\n",
    "        if cell.cell_type == 'code':\n",
    "            cell.outputs = []\n",
    "\n",
    "def clear_notebook(old_ipynb, new_ipynb):\n",
    "    with io.open(old_ipynb, 'r') as f:\n",
    "        nb = nbformat.read(f, nbformat.NO_CONVERT)\n",
    "\n",
    "    remove_outputs(nb)\n",
    "    \n",
    "    with io.open(new_ipynb, 'w', encoding='utf8') as f:\n",
    "        nbformat.write(nb, f, nbformat.NO_CONVERT)\n",
    "\n",
    "source_nb = 'atoms/visualization/choropleth_classification.ipynb'\n",
    "\n",
    "new_nb = 'nout.ipynb'\n",
    "clear_notebook(source_nb, new_nb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Class for Querying"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "source_nb = 'atoms/foundations/Dictionaries-Test.ipynb'\n",
    "nb = read_nb(source_nb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------New Cell--------------------\n",
      "Notebook-6: Dictionaries\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "Lesson Content, In this Notebook\n",
      "--------------------New Cell--------------------\n",
      "Dictionaries\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "Accessing Dictionaries\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "Creating a Simple Phone Book\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "Useful Dictionary Methods\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "Are You On the List? (Part 2)\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "What Do You Do if You're Not On the List?\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "Lists of Lists, Dictionaries of Lists, Dictionaries of Dictionaries... Oh my!\n",
      "A Data Set of City Attributes\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "A Phonebook+\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "Dictionary-of-Dictionaries\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "Code (Applied Geo-example)\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "\n",
      "--------------------New Cell--------------------\n",
      "\n",
      "\n",
      "Further references:\n"
     ]
    }
   ],
   "source": [
    "source_nb = 'atoms/foundations/Dictionaries-Test.ipynb'\n",
    "nb = read_nb(source_nb)\n",
    "\n",
    "import re\n",
    "import markdown\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "md = markdown.Markdown()\n",
    "\n",
    "cell_types = get_nb_structure(nb)    \n",
    "\n",
    "# Iterate over the code cell-types\n",
    "for c in cell_types['markdown']:\n",
    "    \n",
    "    # Delete code blocks -- this is a bit brutal \n",
    "    # and it might be better to escape them in some\n",
    "    # way... but this at least works well enough\n",
    "    src = re.sub(r'```.+?```', '', nb.cells[c]['source'], flags=re.S)\n",
    "    \n",
    "    print(\"-\"*20 + \"New Cell\" + \"-\"*20)\n",
    "    soup = BeautifulSoup(md.convert(src), 'html.parser')\n",
    "    \n",
    "    h1 = soup.findAll('h1')\n",
    "    print( \", \".join([x.contents[0] for x in h1]))\n",
    "    \n",
    "    h2 = soup.findAll('h2')\n",
    "    print( \", \".join([x.contents[0] for x in h2]))\n",
    "    \n",
    "    h3 = soup.findAll('h3')\n",
    "    print( \", \".join([x.contents[0] for x in h3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import re\n",
    "rh1 = re.compile('^# ')\n",
    "rh2 = re.compile('^## ')\n",
    "rh3 = re.compile('^### ')\n",
    "rh4 = re.compile('^#### ')\n",
    "rh = re.compile('^#+')\n",
    "\n",
    "class NoteBook(object):\n",
    "    def __init__(self, ipynb):\n",
    "        self.nb = read_nb(ipynb)\n",
    "        self.structure = get_structure(self.nb.cells)\n",
    "        \n",
    "    def get_cells_by_type(self, cell_type=None):\n",
    "        if cell_type:\n",
    "            cell_type = cell_type.lower()\n",
    "            return [self.nb.cells[i] for i in self.structure[cell_type]]\n",
    "        else:\n",
    "            return self.nb.cells\n",
    "    \n",
    "    def get_cells_by_id(self, ids=[]):\n",
    "        return [self.nb.cells[i] for i in ids]\n",
    "    \n",
    "    def get_header_cells(self):\n",
    "        hs = []\n",
    "        if 'markdown' in self.structure:\n",
    "            idxs = self.structure['markdown']\n",
    "            pairs = zip(idxs, self.get_cells_by_type('markdown'))\n",
    "            hs = [(idx, cell) for idx, cell in pairs if rh.match(cell['source'])]\n",
    "        return hs\n",
    "        \n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nb = NoteBook(source_nb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cid = nb.get_cells_by_id()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cid = nb.get_cells_by_id([7, 10, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{u'cell_type': u'code',\n",
       "  u'execution_count': 5,\n",
       "  u'metadata': {u'collapsed': True},\n",
       "  u'outputs': [],\n",
       "  u'source': u\"y = f.by_col_array('pcgdp2000')\"},\n",
       " {u'cell_type': u'markdown',\n",
       "  u'metadata': {},\n",
       "  u'source': u'#### Sample Mean\\n\\n$\\\\bar{y} = \\\\sum_{i=1}^n y_i$'},\n",
       " {u'cell_type': u'code',\n",
       "  u'execution_count': 1,\n",
       "  u'metadata': {u'collapsed': True},\n",
       "  u'outputs': [],\n",
       "  u'source': u'import pysal as ps'}]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'# Classification for Choropleth Mapping\\n'}),\n",
       " (1,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {u'collapsed': True},\n",
       "   u'source': u'## Introduction\\n\\n* what is classification\\n* role in choropleth mapping\\n* explore classification using PySAL'}),\n",
       " (3,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'## Data Set: Mexico State Gross Domestic Product'}),\n",
       " (9,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'### Numerical summaries'}),\n",
       " (10,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'#### Sample Mean\\n\\n$\\\\bar{y} = \\\\sum_{i=1}^n y_i$'}),\n",
       " (12,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'#### Sample Standard Deviation\\n\\n$\\\\hat{\\\\sigma} = \\\\sqrt{\\\\frac{\\\\sum_{i=1}^n (y_i-\\\\bar{y})^2}{n-1}}$'}),\n",
       " (14, {u'cell_type': u'markdown', u'metadata': {}, u'source': u'#### Median'}),\n",
       " (22,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'### Univariate Distribution Visualization'}),\n",
       " (27,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'## PySAL Classifiers'}),\n",
       " (28,\n",
       "  {u'cell_type': u'markdown', u'metadata': {}, u'source': u'### Quantiles'}),\n",
       " (35,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'### Equal Interval'}),\n",
       " (37,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'### Maximum Breaks'}),\n",
       " (39,\n",
       "  {u'cell_type': u'markdown', u'metadata': {}, u'source': u'### Box Plot'}),\n",
       " (41,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'### Standard Deviation'}),\n",
       " (43,\n",
       "  {u'cell_type': u'markdown', u'metadata': {}, u'source': u'### Head Tail'}),\n",
       " (45,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'### Jenks Caspall'}),\n",
       " (48,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'### Fisher Jenks'}),\n",
       " (50, {u'cell_type': u'markdown', u'metadata': {}, u'source': u'### max-p'}),\n",
       " (52, {u'cell_type': u'markdown', u'metadata': {}, u'source': u'### Fit'}),\n",
       " (64,\n",
       "  {u'cell_type': u'markdown',\n",
       "   u'metadata': {},\n",
       "   u'source': u'## Consensus classification'})]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb.get_header_cells()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdict = defaultdict(list)\n",
    "for idx, cell in nb.get_header_cells():\n",
    "    level = cell['source'].count(\"#\")\n",
    "    hdict[level].append(idx)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(list,\n",
       "            {1: [0],\n",
       "             2: [1, 3, 27, 64],\n",
       "             3: [9, 22, 28, 35, 37, 39, 41, 43, 45, 48, 50, 52],\n",
       "             4: [10, 12, 14]})"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'list' object has no attribute 'copy'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-43-af303cfc889b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mkeys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhdict\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mkeys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msort\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreverse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mall_keys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkeys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mstart_end\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mlast_stop\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcells\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'list' object has no attribute 'copy'"
     ]
    }
   ],
   "source": [
    "# find the start and end cells for each H? block\n",
    "keys = list(hdict.keys())\n",
    "keys.sort(reverse=True)\n",
    "all_keys = keys.copy()\n",
    "start_end = []\n",
    "last_stop = len(nb.nb.cells)\n",
    "while keys:\n",
    "    current = keys.pop(0)\n",
    "    for element in hdict[current]:\n",
    "        above = [k for k in all_keys.copy() if k <= current]\n",
    "        stop = last_stop\n",
    "        while above:\n",
    "            key_above = above.pop()\n",
    "            larger = [v for v in hdict[key_above] if v > element]\n",
    "            if larger:\n",
    "                if larger[0] < stop:\n",
    "                    stop = larger[0]\n",
    "        start_end.append([element, stop])\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[10, 12],\n",
       " [12, 14],\n",
       " [14, 22],\n",
       " [9, 22],\n",
       " [22, 27],\n",
       " [28, 35],\n",
       " [35, 37],\n",
       " [37, 39],\n",
       " [39, 41],\n",
       " [41, 43],\n",
       " [43, 45],\n",
       " [45, 48],\n",
       " [48, 50],\n",
       " [50, 52],\n",
       " [52, 64],\n",
       " [1, 3],\n",
       " [3, 27],\n",
       " [27, 64],\n",
       " [64, 99],\n",
       " [0, 99]]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start_end # for each H? cell report the start and end cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(list,\n",
       "            {1: [0],\n",
       "             2: [1, 3, 27, 64],\n",
       "             3: [9, 22, 28, 35, 37, 39, 41, 43, 45, 48, 50, 52],\n",
       "             4: [10, 12, 14]})"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(start_end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(nb.get_header_cells())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## Data Set: Mexico State Gross Domestic Product\n",
      "ps.examples.available()\n",
      "ps.examples.explain('mexico')\n",
      "f = ps.open(ps.examples.get_path('mexico.csv'))\n",
      "f.header\n",
      "y = f.by_col_array('pcgdp2000')\n",
      "y\n",
      "### Numerical summaries\n",
      "#### Sample Mean\n",
      "\n",
      "$\\bar{y} = \\sum_{i=1}^n y_i$\n",
      "y_mean = y.mean()\n",
      "y_mean\n",
      "#### Sample Standard Deviation\n",
      "\n",
      "$\\hat{\\sigma} = \\sqrt{\\frac{\\sum_{i=1}^n (y_i-\\bar{y})^2}{n-1}}$\n",
      "y_std = y.std()\n",
      "y_std\n",
      "#### Median\n",
      "import numpy as np\n",
      "y_median = np.median(y)\n",
      "y_median\n",
      "(y < y_mean).sum()\n",
      "(y > y_mean).sum()\n",
      "y_sorted = np.sort(y, axis=0)\n",
      "y_sorted\n",
      "y_sorted[15]\n",
      "y_sorted[16]\n",
      "(y_sorted[15]+y_sorted[16])/2.\n",
      "### Univariate Distribution Visualization\n",
      "%pylab inline\n",
      "import seaborn as sns\n",
      "sns.distplot(y)\n",
      "sns.distplot(y, kde=False, rug=True)\n",
      "sns.distplot(y, bins=5, kde=False, rug=True)\n",
      "sns.distplot(y, hist=False,  rug=True)\n"
     ]
    }
   ],
   "source": [
    "# second h2 section with all children\n",
    "se2 = [ v for v in start_end if v[0]==3][0]\n",
    "block = nb.get_cells_by_id(range(*se2))\n",
    "for cell in block:\n",
    "    print(cell['source'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "### Numerical summaries\n",
      "#### Sample Mean\n",
      "\n",
      "$\\bar{y} = \\sum_{i=1}^n y_i$\n",
      "y_mean = y.mean()\n",
      "y_mean\n",
      "#### Sample Standard Deviation\n",
      "\n",
      "$\\hat{\\sigma} = \\sqrt{\\frac{\\sum_{i=1}^n (y_i-\\bar{y})^2}{n-1}}$\n",
      "y_std = y.std()\n",
      "y_std\n",
      "#### Median\n",
      "import numpy as np\n",
      "y_median = np.median(y)\n",
      "y_median\n",
      "(y < y_mean).sum()\n",
      "(y > y_mean).sum()\n",
      "y_sorted = np.sort(y, axis=0)\n",
      "y_sorted\n",
      "y_sorted[15]\n",
      "y_sorted[16]\n",
      "(y_sorted[15]+y_sorted[16])/2.\n"
     ]
    }
   ],
   "source": [
    "# first h3 section in second h2 section with all children\n",
    "se3 = [ v for v in start_end if v[0]==9][0]\n",
    "block = nb.get_cells_by_id(range(*se3))\n",
    "for cell in block:\n",
    "    print(cell['source'])"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}