YelpDataAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset: Yelp review data\n",
    "\n",
    "Download the training_data.json file, a collection of 40,000 json-encoded Yelp reviews we sampled from the [Yelp Dataset Challenge](https://www.yelp.com/dataset_challenge).\n",
    "\n",
    "Each line corresponds to a review on a particular business. The label (class) information of each review is in the \"label\" field. It is **either \"Food-relevant\" or \"Food-irrelevant\"**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Using jsonlines library. Data Format: one valid json per line\n",
    "import jsonlines\n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Read the training_data file\n",
    "review_list = []\n",
    "label_list = []\n",
    "with jsonlines.open('training_data.json') as reader:\n",
    "    for json_obj in reader:\n",
    "        review_list.append(json_obj[\"text\"])\n",
    "        label_list.append(json_obj[\"label\"])\n",
    "\n",
    "review_df = pd.DataFrame({'text': review_list, \n",
    "                         'label': label_list})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>dr. goldberg offers everything i look for in a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Unfortunately, the frustration of being Dr. Go...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Dr. Goldberg has been my doctor for years and ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Been going to Dr. Goldberg for over 10 years. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Got a letter in the mail last week that said D...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>I don't know what Dr. Goldberg was like before...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Top notch doctor in a top notch practice. Can'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Dr. Eric Goldberg is a fantastic doctor who ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Food-relevant</td>\n",
       "      <td>Pretty good dinner with a nice selection of fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Food-relevant</td>\n",
       "      <td>Good truck stop dining at the right price. We ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             label                                               text\n",
       "0  Food-irrelevant  dr. goldberg offers everything i look for in a...\n",
       "1  Food-irrelevant  Unfortunately, the frustration of being Dr. Go...\n",
       "2  Food-irrelevant  Dr. Goldberg has been my doctor for years and ...\n",
       "3  Food-irrelevant  Been going to Dr. Goldberg for over 10 years. ...\n",
       "4  Food-irrelevant  Got a letter in the mail last week that said D...\n",
       "5  Food-irrelevant  I don't know what Dr. Goldberg was like before...\n",
       "6  Food-irrelevant  Top notch doctor in a top notch practice. Can'...\n",
       "7  Food-irrelevant  Dr. Eric Goldberg is a fantastic doctor who ha...\n",
       "8    Food-relevant  Pretty good dinner with a nice selection of fo...\n",
       "9    Food-relevant  Good truck stop dining at the right price. We ..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unicode"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(review_df['label'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>isFoodRelevant</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>dr. goldberg offers everything i look for in a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Unfortunately, the frustration of being Dr. Go...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Dr. Goldberg has been my doctor for years and ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Been going to Dr. Goldberg for over 10 years. ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Got a letter in the mail last week that said D...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>I don't know what Dr. Goldberg was like before...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Top notch doctor in a top notch practice. Can'...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Food-irrelevant</td>\n",
       "      <td>Dr. Eric Goldberg is a fantastic doctor who ha...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Food-relevant</td>\n",
       "      <td>Pretty good dinner with a nice selection of fo...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Food-relevant</td>\n",
       "      <td>Good truck stop dining at the right price. We ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             label                                               text  \\\n",
       "0  Food-irrelevant  dr. goldberg offers everything i look for in a...   \n",
       "1  Food-irrelevant  Unfortunately, the frustration of being Dr. Go...   \n",
       "2  Food-irrelevant  Dr. Goldberg has been my doctor for years and ...   \n",
       "3  Food-irrelevant  Been going to Dr. Goldberg for over 10 years. ...   \n",
       "4  Food-irrelevant  Got a letter in the mail last week that said D...   \n",
       "5  Food-irrelevant  I don't know what Dr. Goldberg was like before...   \n",
       "6  Food-irrelevant  Top notch doctor in a top notch practice. Can'...   \n",
       "7  Food-irrelevant  Dr. Eric Goldberg is a fantastic doctor who ha...   \n",
       "8    Food-relevant  Pretty good dinner with a nice selection of fo...   \n",
       "9    Food-relevant  Good truck stop dining at the right price. We ...   \n",
       "\n",
       "   isFoodRelevant  \n",
       "0               0  \n",
       "1               0  \n",
       "2               0  \n",
       "3               0  \n",
       "4               0  \n",
       "5               0  \n",
       "6               0  \n",
       "7               0  \n",
       "8               1  \n",
       "9               1  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_df['isFoodRelevant'] = review_df['label'].apply(lambda label: 1 if label == u'Food-relevant' else 0)\n",
    "review_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import string\n",
    "from nltk.stem.porter import *\n",
    "import re\n",
    "\n",
    "stemmer = PorterStemmer()\n",
    "# Returns a review after stemming, casefolding, removing punctuation\n",
    "def preprocess_text(review):\n",
    "    # Replaces every punctuation with a whitespace\n",
    "    #replace_punctuation = string.maketrans(string.punctuation, ' '*len(string.punctuation))\n",
    "    word_list = []\n",
    "    \n",
    "    #This for loop turns the text into lower case, splits the text on punctuations and white spaces\n",
    "    # followed by applying Porter stemmer on each word.\n",
    "    text = re.split('\\W+', review.lower())\n",
    "    for word in text:\n",
    "        word = stemmer.stem(word)\n",
    "        word_list.append(word)\n",
    "    return ' '.join(word_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'review_df' (DataFrame)\n"
     ]
    }
   ],
   "source": [
    "review_df['review_clean'] = review_df['text'].apply(preprocess_text)\n",
    "review_df.head(3)\n",
    "%store review_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%store -r review_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unique tokens?\n",
    "Size of your feature space. That is, how many unique tokens are present?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'review_matrix' (csr_matrix)\n"
     ]
    }
   ],
   "source": [
    "count_vect = CountVectorizer(token_pattern=r'\\b\\w+\\b')\n",
    "review_matrix = count_vect.fit_transform(review_df['review_clean'])\n",
    "%store review_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique word count is:  36555\n"
     ]
    }
   ],
   "source": [
    "word_dict= count_vect.vocabulary_\n",
    "\n",
    "n = len(word_dict)\n",
    "print 'Unique word count is: ', n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Most Popular Words\n",
    "\n",
    "Great, now we can tokenize the documents. Let's make a list of the most popular words in our reviews. For this step, you should maintain a count of how many times each word occurs. Then print out the top-20 words in your reviews.\n",
    "\n",
    "Output should look like this:\n",
    "\n",
    "Rank Token Count\n",
    "\n",
    "1 awesome 78\n",
    "\n",
    "... ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unique_words_count = review_matrix.sum(axis=0)\n",
    "words_freq = [(word, unique_words_count[0, idx]) for word, idx in count_vect.vocabulary_.items()]\n",
    "sorted_word_dict = sorted(words_freq, key = lambda x: x[1], reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 \tthe \t246309\n",
      "2 \ti \t168931\n",
      "3 \tand \t168589\n",
      "4 \ta \t134904\n",
      "5 \tto \t128139\n",
      "6 \tit \t78867\n",
      "7 \tof \t76237\n",
      "8 \twa \t74020\n",
      "9 \tis \t63496\n",
      "10 \tfor \t60867\n",
      "11 \tin \t60523\n",
      "12 \tthat \t50804\n",
      "13 \tmy \t50565\n",
      "14 \tyou \t45881\n",
      "15 \tthey \t43635\n",
      "16 \tthi \t39940\n",
      "17 \twith \t39340\n",
      "18 \thave \t39082\n",
      "19 \tbut \t37967\n",
      "20 \ton \t35388\n"
     ]
    }
   ],
   "source": [
    "rank = 1\n",
    "for word, count in sorted_word_dict[:20]:\n",
    "    print rank, \"\\t\", word, \"\\t\", count\n",
    "    rank = rank + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zipf's Law\n",
    "\n",
    "Let's see if this law applies to our Yelp reviews. Using matplotlib to plot the log-base10 term counts on the y-axis versus the log-base10 rank on the x-axis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: 246309,\n",
       " 2: 168931,\n",
       " 3: 168589,\n",
       " 4: 134904,\n",
       " 5: 128139,\n",
       " 6: 78867,\n",
       " 7: 76237,\n",
       " 8: 74020,\n",
       " 9: 63496,\n",
       " 10: 60867,\n",
       " 11: 60523,\n",
       " 12: 50804,\n",
       " 13: 50565,\n",
       " 14: 45881,\n",
       " 15: 43635,\n",
       " 16: 39940,\n",
       " 17: 39340,\n",
       " 18: 39082,\n",
       " 19: 37967,\n",
       " 20: 35388,\n",
       " 21: 34969,\n",
       " 22: 34289,\n",
       " 23: 31608,\n",
       " 24: 29853,\n",
       " 25: 27347,\n"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rank_count_dict = {}\n",
    "count = 1\n",
    "for k,v in sorted_word_dict:\n",
    "    rank_count_dict[count] = v\n",
    "    count += 1\n",
    "\n",
    "rank_count_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,u'log10 (count)')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd4VVW+xvHvL40kkEJJAAlVioD0\nIs0CWLCMjmIfGR1RdMQy6jSv43XmOsXRkRn7iOKIotgLo4ggooA06QKhC1KTUBMgIW3dP85BI4Zw\ngJzsU97P8+yHnJ199n45yi8ra6+9ljnnEBGRyBfjdQAREakZKvgiIlFCBV9EJEqo4IuIRAkVfBGR\nKKGCLyISJVTwRUSihAq+iEiUUMEXEYkScV4HqKhBgwauRYsWXscQEQkbCxYs2OGcywjk2JAq+C1a\ntGD+/PlexxARCRtmtjHQY9WlIyISJVTwRUSihAq+iEiUUMEXEYkSKvgiIlFCBV9EJEqo4IuIRImI\nKPhPTF3DR0u3UVBU4nUUEZGQFVIPXh2PwuIyxs7awM79xcTHGr1b1mPwKQ0Z3D6T5vVrex1PRCRk\nWCgtYt6zZ093PE/alpaVs2jTHj7NzmFqdi5rc/cB0DqzDoPbZzL4lIZ0b5ZOXGxE/EIjIvIdM1vg\nnOsZ0LGRUPAPt3HnfqZm5/LZylzmfrOTkjJHenI8Z7XNYHD7hpzZLoPUxPhqSCwi4q2oL/gVFRSV\nMGPNDj7NzuHzVXns2l9MXIzRq0U9X+u/fUNaNlDXj4iEJxX8IygrdyzetJtPs3P5LDuXVTkFALTK\nqM3gU3zFv2fzuur6EZGwoYIfoE27DvDZylw+zc5h7vpdFJeVk5oYx1ntMr9r9ZuBYRW+5vuvzfcq\nLsYYdEombRqm1Fh2ERFQwT8u+w6WMnNNHlOzc5m2Kpcd+4qP+Rz9Tq7P9f1acHb7hsTG2NHfICJy\nglTwq4lzjkMfj/O//v7rQ/sdewtLeGv+ZsbN2ci2vUU0SU9iWN/mXNWzKXVrJ3iSXUSigwq+R0rL\nypmyIoeXZm1g7je7qBUXw0+7NuH6fi3ocFKq1/FEJAKp4IeA7G35vDx7A+8t2kJRSTnN6iXTtF4S\nTesmk1U3iab1fH+2alBHvwWIyHFTwQ8hew4U8/aCzSzetIfNuwvZvPvAD+4PJMTF8OpNp9GrRT0P\nU4pIuAqZgm9mG4ACoAwoPVqoSCz4lSksLmPLngNs2lXI/05YhmF8fNfp1K4V9jNdiEgNO5aCXxMD\nzgc657oGGigaJCXE0jozhYGnZPLYFV3ZtPsAf52Y7XUsEYlwesLIY71b1uOmAS15de63fLE6z+s4\nIhLBgl3wHTDZzBaY2YggXyts3XtuO9pk1uG3by9h7wFN8SwiwRHsgt/fOdcdOB8YaWZnHH6AmY0w\ns/lmNj8vLzpbuInxsYy6sis79xXz4IRlXscRkQgV1ILvnNvq/zMXeA/oXckxo51zPZ1zPTMyMoIZ\nJ6R1ykrj9kGteX/xVj7+epvXcUQkAgWt4JtZbTNLOfQ1cC6g5msVRg5sTacmafzPe18zNTuHNTkF\nHCgu9TqWiESIYI4DbAi8559gLA54zTk3KYjXC3vxsTGMurILP336S4aP/X54at3keJrUTaJJehJN\n0pNpUjeJvq3q6+ldETkmevAqBO05UMza3H1s2VPI5t2FbNlTyJYKfxaWlBEbY9w1uA0jB7bWRG0i\nUexYxuHrSZ8QlJ6cQM8W9ajsv6BzjryCg/xlYjajpqxmxpo8/nlVV7LqJtd4ThEJL2rhh7H3Fm3m\ngfeXU+4cvVvWo3uzunTKSiMtKZ7EuFgapyVqnh6RCKcWfpS4tFsWPZrV49kv1rJg426+WJ1HxZ/f\nSfGxjLvpNHo0r+tdSBEJGWrhR5D8ohJWbitgf3EpRcVl/H3SSnYfKOHtW/tqNS6RCKUWfpRKTYyn\nd8vvZ908tUkalz07i5+/OI/fDmlHUnwc7RqlaNF2kSilFn6EW751L9eMnkN+kW88f1yM8evz2jHi\n9FbEaHSPSNhTC1++0/GkNL78/SByCw5SWFzGM5+v5eGPV/JZdi43n9GKge0yiIvVHHoi0UAt/Cjj\nnOPN+Zt4bPJqcgsO0qxeMo9f3ZVuzXRjVyQchcwCKMdKBb/mlJSVMzU7lz9/tIKc/CJu8K+7m1En\nkXq1E2jbsI5a/iJhQF06clTxsTEMObURfVvV5773lvLCzG9+MKQzLSmeS7s14XdDTiEpIda7oCJS\nbVTwo1xacjzP/KwHRSVlbN59gJ37itmeX8S0lbmMnb2BWet2cP6pjWnXKIXeLevRoE4tryOLyHFS\nl44c0Rer8/jjhOVs2Lkf53wLrj93XQ8GnpLpdTQR8VMfvlSropIysrfl88AHy1ids49Hhnbmp92a\neB1LRAi9RcwlzCXGx9KtWV3GDT+NLllp/OqNxfx90kqvY4nIMVIfvgQsPTmB8Tf34YEPlvPs5+so\nL3f0Obk+HRunkpma6HU8ETkKFXw5JnGxMTx0SUd27y/muenreW76esxgQOsGDO2exZltMzRDp0iI\nUh++HLdd+30Ltcxat4O35m9my55CkuJjeeZn3XVjV6SG6Kat1LjycseiTXt4cMIylm3J57LuTfjH\n5V00X49IkOmmrdS4mBijR/O6vD6iLzef3pJ3F25h1JTVFJWUeR1NRPzUhy/Vqk6tOP7ngvZs3VPE\nU9PW8sqcjVzftznnd2rMSWlJpCXHex1RJGqpS0eCorzcMWf9Tl6evZFJy7d/t39Ix0Y8PLQT6cm6\nsStSHTSXjnguJsbo17oB/Vo34Jsd+1m2ZS/Ltu7l+enrmbYql9aZdTijbQaXdWui1bhEaoha+FKj\nVm7P5/V5m1ixNZ95G3aRUiuOP1zUnnM6NKKehnOKHDON0pGwsGp7AcPGzCW34CDtG6fywcj+JMRp\nHIHIsdAoHQkL7Rql8OXvB/HI5Z3J3pZPj4emcPtrC1m6eY/X0UQikvrwxVPxsTFc2bMpmSm1+Pjr\n7UxesZ3JK3J465a+dGma7nU8kYiiLh0JKXkFBzn/8Rns2HeQtg3r8MeLO9K3VX3M9ACXSGXUpSNh\nKyOlFhNu78/vzz+FwpIyrn1+Lr99eyl7DhR7HU0k7AW9hW9mscB8YItz7qKqjlULXyrKLyrhvne+\n5qOvt1ErLoYb+rfgzkFtqF1LPZEih4RaC/8uILsGriMRJjUxnqeu7cZrN5/GeR0b8dwX6xk2Zi5r\ncgq8jiYSloJa8M0sC7gQeCGY15HIZWb0O7kBT1zTjUcu78ya3H3c8J+vmLIih7yCg17HEwkrwf7d\n+F/Ab4EjPkppZiOAEQDNmjULchwJZ1f2bMopjVK4/sV53Pyyr+uvVYPa9D25Pvec05b6WmBdpEpB\nK/hmdhGQ65xbYGZnHek459xoYDT4+vCDlUciQ+esdGb+bhALNu5m1fYCvly3gze+2sSq7QXcf2F7\nujZN14gekSMI2k1bM/sbMAwoBRKBVOBd59x1R3qPbtrK8Rg/71seeH8ZpeWOLllpPPTTU+mcpTH8\nEh1CbmoFfwv/1xqlI8GSm1/E1JW5PDZ5Fbv2F/OniztyXZ/mau1LxAu1UToiQZeZmsg1vZvxwe0D\nOL1NBg98sJzL/z2bsbM2UFJW7nU8kZCgJ20l4pSXO16evYExX37Dpl2FdG+WznV9mnN2h4akJmoB\nFoksIdelEygVfKluY2Z+w5gZ69m6t4ik+FgeHtqJS7o28TqWSLVRl46I3/ABLZn5u0G8MaIPdRLj\nuOv1xfz+naVs31vkdTSRGqcWvkSN/KISRk1ezdjZG6gVF8PQ7llc0Kkx/Vs38DqayHFTl45IFdbm\n7uPJz9YwZUUOB4rLuKZ3M/5wYXvN0SNhSWvailShdWYdHr+6GwdLyxg1eTXPTV/Pp9k5XNipMed2\nbEi/k9Xil8ikFr5EvanZOYydvZE563ZSXFbOgNYNuK5PM/q0qk96stbZldCmFr7IMRjcviGD2zdk\n38FSxsz4hhdmrmfmuB0AdM5K48b+Lbmk60l6iEvCnlr4IofZvb+YJZv3sHTzXl6du5Gc/IP0aF6X\nUVd2oXn92l7HE/kB3bQVqSbFpeXc/cZipqzIITbGGHFGK0ac0Uo3eCVkaBy+SDVJiIvh6Z91Z+q9\nZ9L35Po8PnUNFzwxg40793sdTeSYBVzwzay2f7lCkajTtF4yL97Qi7E39mZHwUFueWUB6/L2eR1L\n5JgcseCbWYyZXWtmH5lZLrAS2GZmy83sUTNrU3MxRULDmW0zePa6HmzPL+LCJ2Ywadk2ryOJBKyq\nFv404GTgPqCRc66pcy4TOB2YAzxsZkec214kUp3RNoNPfnUGbTJTuHXcQq5/cR7Lt+71OpbIUR3x\npq2ZxTvnSqp8cwDHHAvdtJVwUlhcxpiZ63l62joKS8oY2C6Dv17WicZpSV5HkyhSLTdtDxVyM3ul\nkgu8UvEYkWiUlBDL7YPaMPu+QfzmvHbMWreTsx79nCenruFAcanX8UR+JJCbth0rvvDfuO0RnDgi\n4Sc9OYGRA1vz8V2n07tlPR6bsprhL81nz4Fir6OJ/EBVN23vM7MCoLOZ5fu3AiAX+KDGEoqEiVYZ\ndXhl+Gk89NNTmb1+Jxc9OZNPV+RQXh46z7pIdKuqS+dvzrkU4FHnXKp/S3HO1XfO3VeDGUXCyrA+\nzXn5xt44Bze9PJ/zH5/B15t1U1e8F9CTtmbWBGhOhbl3nHPTqzuMbtpKJCkpK+fDpVt56MNsdu0v\nZmj3LH5zXjsapSV6HU0iSLVOnmZmDwNXAyuAMv9uB1R7wReJJPGxMVzaLYsz22bywoz1/PuLdXy4\ndCvXntaM2we2pn6dWl5HlChz1Ba+ma0COjvnDgY7jFr4EsnW5u7j31+s4/1FW2iYmsjUe88kMV4P\nr8uJqe65dNYD8ScWSURaZ9bhH1d04d/X9WDLnkKuem424+d9S1FJ2dHfLFINAin4B4DFZvacmT1x\naAt2MJFIdXaHhjx2RRfyi0q5792vOeefX7Bsi27qSvAF0qVzfWX7nXNjqzuMunQkmjjnmLFmB3e/\nsZg9hSXcNKAltw1sTVqSfqGWwGk+fJEwkpNfxN8mZvP+4q2kJMZxzzlt+XnfFsTGaIUtObpqLfhm\n9g2+UTk/4JxrdXzxjkwFX6LZiq35PDxpJdNX59GlaTpPXt2NZvWTvY4lIa66b9r2BHr5t9OBJ4Bx\nxx9PRCrT4aRUxv6iF49d0YV1ufu45vk5moVTqtVRC75zbmeFbYtz7l/AoKO9z8wSzWyemS3xz6H/\np2pJLBLBzIyhPbJ4fUQfSsvLGfrsLM25L9XmqAXfzLpX2Hqa2a1ASgDnPggMcs51AboCQ8yszwnm\nFYkKpzZJ48M7Tqdp3WRuHbeQX45bwL6DmoFTTkwgKzE/VuHrUmADcOXR3uR8NwcOrQEX799C5w6x\nSIjLSKnFxLtO56nP1vLUtLWseGIGfx/amT6t6nsdTcLUUQu+c27g8Z7cP5XyAqA18LRzbu7xnksk\nGsXHxnD3OW05rWU97nvva64ePYdhfZpz/4Xt9ZSuHLNAunTSzGyUmc33b4+ZWVogJ3fOlTnnugJZ\nQG8zO7WS8484dO68vLxj/xuIRIF+rRvw8V2nc2P/lrwyZyO/HLeA3fs1374cm0BG6bwIFODrxrkS\nyAf+cywXcc7tAT4HhlTyvdHOuZ7OuZ4ZGRnHclqRqJKcEMf//qQDD1zUgZlrdzDk8enMWrvD61gS\nRgIp+Cc75x50zq33b38CjjoG38wyzCzd/3UScDaw8sTiisjwAS1577b+1K4Vx7UvzOXJqWsoLi33\nOpaEgUAKfqGZDTj0wsz6A4UBvK8xMM3MlgJfAVOccx8eX0wRqcg3imcAF3RqxGNTVnPN83NYm1vg\ndSwJcYE8adsVGAsc6rffDdzgnFtS3WH0pK3IsXtnwWb+9N/lHCwt52+XdeKy7lleR5IaVK0LoDjn\nFgNdzCzV/zr/BPOJSDUa2iOLM9pmMPK1hdzz5hKWbNrD/Rd2ICEukF/gJZoEMkrnr2aW7pzLd87l\nm1ldM/tzTYQTkcBkpNTitZtO46YBLRk7eyM/eXKm1tGVHwmkCXC+f5QNAM653cAFwYskIscjLjaG\nP1zUgRd+3pPdB4q5avRsZq/b6XUsCSGBFPxYM/tu8U3/iBstxikSos7u0JAP7xhAo9REfvbCHP7v\nvyu0qpYAgRX8ccBUMxtuZjcCU/DdxBWREJWZmsj7t/fn2tOa8eKX33DjS19pLh4JbAEUMxuCbxy9\nAZOdc58EI4xG6YhUv3cXbuY3by+l40mpvPSL3tSrneB1JKlG1TJKx8zMPwEazrlJwKSqjhGR0HRZ\n9yxSE+MZ+dpCzv3ndP7vko5c0Kmx17HEA1V16UwzszvMrFnFnWaWYGaDzGwsUOl6tyISWs7u0JC3\nbu1Lw9Ra3PbqQv45ZTUHitXFE22qKvhDgDJgvJltNbMV/uUO1wDXAP90zr1UAxlFpBp0zkrnjVv6\ncmGnxjw+dQ3Dxsxjv/r1o0qgffjxQAOgsOIQzeqmPnyRmvHeos3c8+YSejSry5jre5GWHO91JDlO\n1b2mLc65EufctmAWexGpOZd2y+Lxq7uxaNMeLnhiBpt3H/A6ktQAPXstEqUu7nISr4/oQ35RCUOf\nncXiTWrPRToVfJEo1qtFPV4Zfhpl5Y6hz85i2spcryNJEAUyl05D/wLm3cysYU2EEpGa07VpOp/e\ncyZtMutwyysL+O+SrV5HkiA5YsE3s65mNgffSlWPAI8CX5jZHDPrXkP5RKQGpCcn8PqIPnTKSuOO\n8Yt46rM1XkeSIKhqeuSXgFsOX3jczPrgW+KwSxBziUgNS09OYNzw0/jVG4v4x+TV1IqL5eYzjrq4\nnYSRqrp0ah9e7AGcc3OA2sGLJCJeSUqI5alru9O/dX3+MjGbsbM2eB1JqlFVBf9jM/vIzK4ys37+\n7Soz+4hKplkQkcgQHxvDizf0YtApmTw4YTmPTFpJWblmUIkEVT54ZWbnA5cATfBNnLYZmOCcmxiM\nMHrwSiR0FJeWc9urC/k0O4cuWWmMuqorJ2fU8TqWHOZYHrwK6EnbmqKCLxJanHOMm7ORv09aRbF/\nzdyhPbRmbiip9idtK7nA6ON5n4iEFzNjWN8WfHjHANo0rMO9by3hH5+sIpQaihK4qoZl1jvCVh8t\ncSgSVVo0qM07v+zHhZ0b89S0tQwfO18Tr4WhqoZl5gEb8fXdH+L8rzODGUpEQk9ifCxPXdONkzPq\n8MTUNVzx79m8PLw3DepoxdNwUVWXznrgLOdcywpbK+dcSyCnhvKJSAgxM+45py2jh/Vgbd4+hj47\ni2927Pc6lgSoqoL/L6DuEb73SBCyiEiYOLdjI8b+ojd7DpRwxb9nsTZ3n9eRJABHLPjOuaedc0uO\n8L0ngxdJRMJB35Pr8/qIPpSUOS5+aiarthd4HUmOIpDJ0y6rZBtsZurHF4ly7Run8srw3pSUlXP+\n49OZvHy715GkCoEMyxwOvAD8zL89D9wDfGlmw4KYTUTCQOesdD6683Sa1UtmxCsLeGTSSq8jyREE\nUvDLgfbOuaHOuaFAB+AgcBrwuyO9ycyamtk0M8s2s+Vmdlf1RBaRUNO2YQrv/LIfJ6Ul8szn67j3\nzSWUlJV7HUsOE0jBb+GcqzgqJxdo65zbBZRU8b5S4F7nXHugDzDSzDocf1QRCWX169Tis1+fxeU9\nsnhn4WYuemImRSVlXseSCgIp+DPM7EMzu97MrgcmANPNrDZwxDXR/GvgLvR/XQBk45uTR0QiVGJ8\nLP+4ogu3nNGKVTkFDPj7NA6WquiHikAK/kh88993BboBY4GRzrn9zrmBgVzEzFr43/uj6ZZFJPLc\nd0F7LurcmB37DjLi5QWUa7bNkHDUgu98k2bMBD4DPgWmu2OYSMPM6gDvAL9yzuVX8v0RZjbfzObn\n5eUFnlxEQtpT13bnih5ZfLE6jxte+kpTLIeAQIZlXgnMAy4HrgTmmtnlgZzczOLxFftXnXPvVnaM\nc260c66nc65nRkZG4MlFJOQ9ekUXbujXgumr8/jluAWadM1jVc2lc8j9QC/nXC6AmWXga+m/XdWb\nzMyAMUC2c27UiQYVkfD0x4s7crC0nPHzvuWWVxbw3LAe+MqD1LRA+vBjDhV7v50Bvq8/MAwYZGaL\n/Ztm2RSJQn+99FQu69aEyStyuG7MXPXpeySQFv4kM/sEGO9/fRVw1BWvnHMz+eFMmyISpcyMx67s\nwu4DxUxblceQx6cz+e4zvY4VdQK5afsbYDTQGegCjHbOHfGBKxGRypgZ//lFb/q2qs/qnH0Mf+kr\nryNFnYBWvHLOveOcu8c5d7dz7r1ghxKRyPXSjb0AmLoyl9teXeBxmuhS1YpXBWaWX8lWYGY/Gl4p\nIhKIWnGxrPi/8wCY+PV2bhqrdaxrSlXTI6c451Ir2VKcc6k1GVJEIktyQhxL/vdcAD7NzuGvE7M9\nThQdjmsRcxGRE5WWHP9d0R89fT2/e3upx4kinwq+iHgmLTme6b/xzdDyxvxN3PPGYo8TRTYVfBHx\nVLP6ycy5bzAA7y7awqOfaD79YFHBFxHPNUpLZN79vqL/9LR1fLR0m8eJIpMKvoiEhMyURN69rR8A\nI19byIKNuzxOFHlU8EUkZHRvVpe/D+0EwNBnZ7M6RwujVycVfBEJKVf1asbdZ7cF4Nx/TuebHfs9\nThQ5VPBFJOTcdXYbbh/YGoCB//icnfsOepwoMqjgi0hI+vV57bihXwsAevz5U3bvL/Y2UARQwReR\nkPXHizvyky4nAdDtoSnsP1jqcaLwpoIvIiHtyWu60btlPQA6PviJiv4JUMEXkZD3xog+dDzJN4VX\nv4c/0wIqx0kFX0RCnpkx4fYBZNVNYm9hCRc9OVPr4x4HFXwRCQuxMcan9/hWyVqxLZ8HPljmcaLw\no4IvImEjMT6WBX84G4Bxc75l9PR1HicKLyr4IhJW6tepxeS7zwDgrxNXsnjTHo8ThQ8VfBEJO20b\npjDqyi4A/PTpL9m2t9DjROFBBV9EwtJl3bO47ayTAThn1HTKNHLnqFTwRSRs/ea8drTOrMO+g6Vc\n98Jcr+OEPBV8EQlbZsZbt/QlMT6G2et38qBG7lRJBV9Ewlrd2glMuds3XHPs7I2MmfmNx4lClwq+\niIS9pvWS+WBkfwAe+nAFE5Zs9ThRaFLBF5GI0KVpOk9c0w2AO8cvYs76nR4nCj0q+CISMS7uctJ3\n8+jf8J95rMvb53Gi0BK0gm9mL5pZrpnpLoqI1Jhfn9eOs9plUFRSzp/+u0Jz7lQQzBb+S8CQIJ5f\nRKRSz/+8J92bpTN9dR6XPjPL6zghI2gF3zk3HdCy8yJS4+JjY/jLpZ3o26o+X2/Zy9hZG9TSJwT6\n8M1shJnNN7P5eXl5XscRkQjRvnEqP+/bHIAHJyxnba768z0v+M650c65ns65nhkZGV7HEZEIcn6n\nxoz9RW8Abnp5PmNnbfA2kMc8L/giIsHUpWkal3VvQkFRKZ8s3+51HE+p4ItIREtJjGfUlV3p1jSd\nOet30v2hKSzdHJ1TKgdzWOZ4YDbQzsw2m9nwYF1LRORobht4Mlf1asqu/cVkb8v3Oo4n4oJ1Yufc\nNcE6t4jIserRvB6tM1MYP28To6as5oPFWxlzfS+SEmK9jlZj1KUjIlEjNTGOmwa0pFFaErPW7WTT\n7gNeR6pRKvgiEjXMjD9c1IFfDW4DwMSvt/HfJVspLC7zOFnNUMEXkajTOD0RgH99uoY7xi/iv0uj\nY3ZNFXwRiTqnNEpl3v2D+ejOAQDkF5Z4nKhmBO2mrYhIKMtMSSQ9KQGA6Wt2UFruaFG/NkNObeRx\nsuBRwReRqBUfazSvn8z01XlMX51HbIyx5s/nExNjXkcLChV8EYlaZsZn955FcWk5L8xYz2NTVnOw\ntDxih2qqD19EolpsjJGUEEvtWr7278HSyB2xoxa+iAiQGO9r1Q967AtizEhNjOPNW/vSoE4tj5NV\nHxV8ERFgcPtMbshpQXFZOdv2FDJtVR7f7jqggi8iEmkapibyx4s7AjBr7Q6mrcqjpLTc41TVS334\nIiKHiY/zlcaSsshaJUstfBGRwyTE+gr+rHU7yC/yPZQVY9C/dQNSEuO9jHZCVPBFRA5Tr7bvgaxn\nPl/3g/33ntOWO/zz8IQjFXwRkcM0rZfMrN8PoqCo9Lt9P3lyJvvDfJI1FXwRkUqclJ70g9dxsUZp\nWXjfxNVNWxGRAMTGGKXl4X0TVwVfRCQA8bExlJarhS8iEvFiY4yyMG/hqw9fRCQA8THGB4u3MmPN\njh99LzOlFuNH9KFWXGhPuqaCLyISgDsHt2Hehl0/2r9hx34WfruHXfuLaZyWVMk7Q4cKvohIAK7u\n3Yyrezf70f43v9rEwm/3hEV3j/rwRUROwKHFUsLhfq4KvojICfDPwkC5UwtfRCSixZivhV+mgi8i\nEtkOFfxy9eGLiES22EN9+KFf74Nb8M1siJmtMrO1Zvb7YF5LRMQL/nof3aN0zCwWeBo4H+gAXGNm\nHYJ1PRERL3zXpRPlffi9gbXOufXOuWLgdeCSIF5PRKTGhVPBD+aDV02ATRVebwZOC+L1RERq3KE+\n/JGvLSTxOKdWqJucwJu39q3OWJUKZsG3Svb96EegmY0ARgA0a/bjp9hEREJZ16bpDO2eRWFJ6dEP\nPoLUGlo2MZgFfzPQtMLrLGDr4Qc550YDowF69uwZ+r8TiYhUULd2Ao9d2cXrGAEJZh/+V0AbM2tp\nZgnA1cCEIF5PRESqELQWvnOu1MxuBz4BYoEXnXPLg3U9ERGpWlBny3TOTQQmBvMaIiISGD1pKyIS\nJVTwRUSihAq+iEiUUMEXEYkSKvgiIlHCXAjN/2BmecDG43x7A+DHy8lHH30O39Nn8T19Ft+LtM+i\nuXMuI5ADQ6rgnwgzm++c6+l1Dq/pc/iePovv6bP4XjR/FurSERGJEir4IiJRIpIK/mivA4QIfQ7f\n02fxPX0W34vazyJi+vBFRKRqkdTCFxGRKoR9wddC6T5m9qKZ5ZrZMq+zeM3MmprZNDPLNrPlZnaX\n15m8YmaJZjbPzJb4P4s/eZ3sEVv4AAAEiklEQVTJS2YWa2aLzOxDr7N4IawLvhZK/4GXgCFehwgR\npcC9zrn2QB9gZBT/f3EQGOSc6wJ0BYaYWR+PM3npLiDb6xBeCeuCjxZK/45zbjqwy+scocA5t805\nt9D/dQG+f+BNvE3lDeezz/8y3r9F5Y07M8sCLgRe8DqLV8K94Fe2UHpU/sOWyplZC6AbMNfbJN7x\nd2MsBnKBKc65aP0s/gX8Fij3OohXwr3gB7RQukQnM6sDvAP8yjmX73UerzjnypxzXfGtK93bzE71\nOlNNM7OLgFzn3AKvs3gp3At+QAulS/Qxs3h8xf5V59y7XucJBc65PcDnROe9nv7AxWa2AV/X7yAz\nG+dtpJoX7gVfC6XLj5iZAWOAbOfcKK/zeMnMMsws3f91EnA2sNLbVDXPOXefcy7LOdcCX534zDl3\nncexalxYF3znXClwaKH0bODNaF0o3czGA7OBdma22cyGe53JQ/2BYfhacYv92wVeh/JIY2CamS3F\n10Ca4pyLyiGJoidtRUSiRli38EVEJHAq+CIiUUIFX0QkSqjgi4hECRV8EZEooYIvYcXM9h39qCO+\n93b/rKrOzBpU2G9m9oT/e0vNrPsR3p9kZl/4J+07YWb2kpldXsn+182sTXVcQ6QiFXyJJl/ie/Bo\n42H7zwfa+LcRwLNHeP+NwLvOubKKO6vrB0AFz+Kb80WkWqngS1jyt8ofNbNlZva1mV3l3x9jZs/4\n537/0MwmHmpFO+cWOec2VHK6S4CX/TNLzgHSzaxxJcf9DPjAf52z/HPuvwZ87d/3vpkt8F97RIWs\n+8zsL/456eeYWcNK/j4P+Vv8McAM4GwzizuhD0nkMCr4Eq4uwze/exd8rfZH/UX6MqAF0Am4Cegb\nwLmOOuuqf+qOVof9wOgN3O+cOzTX/o3OuR5AT+BOM6vv318bmOOfk346cPNh534EyAR+4Zwrd86V\nA2v9fzeRaqOCL+FqADDePxNkDvAF0Mu//y1/4dwOTAvgXIHMutoA2HPYvnnOuW8qvL7TzJYAc/BN\n6neoH74YODSdwQJ8P5AOeQBId87d4n742HsucFIA2UUCpoIv4aqyIl3V/qoEMutqIZB42L79313U\n7Cx8v2n09bfkF1U4vqRCMS8DKnbVfAX0MLN6h5070X9NkWqjgi/hajpwlX9xjwzgDGAeMBMY6u/L\nbwicFcC5JgA/998X6APsdc5tq3iAc243EGtmhxf9Q9KA3c65A2Z2Cr6lFQMxCXgY+MjMUirsbwtE\n5USAEjwq+BKu3gOWAkuAz4Df+rtw3sHXYl8GPIdvpau9AGZ2p5ltxteCX2pmh5a6mwisx9dv/jxw\n2xGuORlfl1FlJgFx/lkpH8LXrRMQ59xb/utO8A/9bAgUHv5DR+REabZMiThmVsc5t89/03Qe0N//\nw+BEz9sNuMc5N+yEQ1Z9nbuBfOfcmGBeR6KPhn1JJPrQv+hHAvBQdRR78A3r9A/FjD18LH412wO8\nEsTzS5RSC19EJEqoD19EJEqo4IuIRAkVfBGRKKGCLyISJVTwRUSihAq+iEiU+H+OyFABGUrChQAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7feb0a78e250>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from math import log\n",
    "x = [log(e,10) for e in rank_count_dict.keys()]\n",
    "y = [log(e,10) for e in rank_count_dict.values()]\n",
    "\n",
    "plt.plot(x,y)\n",
    "plt.xlabel('log10 (rank)')\n",
    "plt.ylabel('log10 (count)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zipf's law states that the frequency of any word is inversely proportional to its rank in the frequency table. \n",
    "\n",
    "Our graph plotted above is consistent with the Zipf's law. Initial set of words with very high frequency are useless in our text analysis. As can be seen above (in top20 words section), these words are mostly stop words. As the rank of the words in increasing, the corresponding frequency of the word is decreasing.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Represenation\n",
    "\n",
    "In this part  build feature vectors for each review. This will be input to our ML classifiers. Call parser from earlier, using all the same assumptions (e.g., casefolding, stemming). Each feature value should be the term count for that review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%store -r review_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prepare_data(dataframe):\n",
    "    dataframe['review_clean'] = review_df['text'].apply(preprocess_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 2, 1, 2, 1, 1, 1, 1, 1,\n",
       "       2, 3, 1, 1, 1, 1, 1, 1, 1, 1, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3,\n",
       "       1, 1, 1, 1, 1, 2, 1, 6, 1, 3, 1, 3, 3, 1, 1, 3, 3, 1, 1, 4, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 2])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_matrix.data[:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40000, 36555)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning Basics\n",
    "\n",
    "In this par evaluate a bunch of classifiers -- kNN, Decision tree, Naive Bayes, and SVM -- on the feature vectors generated in the previous task in two different settings. \n",
    "\n",
    "### Setting 1: Splitting data into train-test \n",
    "\n",
    "In the first setting, treat the first 70% of your data as training. The remaining 30% should be for testing. \n",
    "\n",
    "### Setting 2: Using 5 fold cross-validation\n",
    "\n",
    "In the second setting, use 5-folk cross-validation. \n",
    "\n",
    "### What to report\n",
    "\n",
    "* Report the overall accuracy for both settings.\n",
    "* For the class \"Food-relevant\", report the precision and recall for both settings.\n",
    "* For the class \"Food-irrelevant\", report the precision and recall for both settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setting 1 - Split data into 70:30 ratio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of original data:  40000\n",
      "Length of training data:  28000\n",
      "Length of test data:  12000\n"
     ]
    }
   ],
   "source": [
    "# TODO - Change to iloc to remove this deprecation warning\n",
    "partition_index = int(len(review_df) * .7)\n",
    "\n",
    "train_df = review_df.iloc[: partition_index] \n",
    "test_df = review_df.iloc[partition_index:]\n",
    "\n",
    "\n",
    "print \"Length of original data: \", len(review_df)\n",
    "print \"Length of training data: \", len(train_df)\n",
    "print \"Length of test data: \", len(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_vectorizer = CountVectorizer(token_pattern = r'\\b\\w+\\b')\n",
    "train_review_matrix = train_vectorizer.fit_transform(train_df['review_clean'])\n",
    "test_review_matrix = train_vectorizer.transform(test_df['review_clean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'train_review_matrix' (csr_matrix)\n",
      "Stored 'test_review_matrix' (csr_matrix)\n",
      "Stored 'train_df' (DataFrame)\n",
      "Stored 'test_df' (DataFrame)\n"
     ]
    }
   ],
   "source": [
    "%store train_review_matrix\n",
    "%store test_review_matrix\n",
    "%store train_df\n",
    "%store test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%store -r train_review_matrix\n",
    "%store -r test_review_matrix\n",
    "%store -r train_df\n",
    "%store -r test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score\n",
    "\n",
    "def calculateMetrics(predicted_labels, true_labels):\n",
    "    accuracy = accuracy_score(true_labels, predicted_labels, normalize=True)\n",
    "\n",
    "    precision_food_relevant = precision_score(true_labels, predicted_labels, pos_label = 1)\n",
    "    recall_food_relevant = recall_score(true_labels, predicted_labels, pos_label = 1)\n",
    "    \n",
    "    precision_food_irrelevant = precision_score(true_labels, predicted_labels, pos_label = 0)\n",
    "    recall_food_irrelevant = recall_score(true_labels, predicted_labels, pos_label = 0)\n",
    "   \n",
    "    return accuracy, precision_food_relevant, recall_food_relevant, precision_food_irrelevant, recall_food_irrelevant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_metrics(metrics):\n",
    "    metrics = [round(e, 5) for e in metrics] \n",
    "    print \"Accuracy: \", metrics[0]\n",
    "    print \"Precision is \", metrics[1], \" and recall is \", metrics[2], \" for Food Relevant class\"\n",
    "    print \"Precision is \", metrics[3] , \" and recall is \", metrics[4], \" for Food Irrelevant class\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decision Tree Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import tree\n",
    "\n",
    "def decision_tree_model(train_review_matrix, train_df, test_review_matrix, test_df):\n",
    "    dt_clf = tree.DecisionTreeClassifier()\n",
    "    dt_clf = dt_clf.fit(train_review_matrix, train_df['isFoodRelevant'])\n",
    "\n",
    "    predicted_classes = dt_clf.predict(test_review_matrix)\n",
    "    metrics = calculateMetrics(predicted_classes, test_df['isFoodRelevant'])\n",
    "    return metrics, dt_clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.76517\n",
      "Precision is  0.07428  and recall is  0.94142  for Food Relevant class\n",
      "Precision is  0.99844  and recall is  0.76158  for Food Irrelevant class\n"
     ]
    }
   ],
   "source": [
    "dt_metrics = decision_tree_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "print_metrics(dt_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Naive Bayes Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "def naive_bayes_model(train_review_matrix, train_df, test_review_matrix, test_df):\n",
    "    clf = MultinomialNB().fit(train_review_matrix, train_df['isFoodRelevant'])\n",
    "\n",
    "    predicted_classes = clf.predict(test_review_matrix)\n",
    "    return calculateMetrics(predicted_classes, test_df['isFoodRelevant']), clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.93717\n",
      "Precision is  0.23805  and recall is  0.97908  for Food Relevant class\n",
      "Precision is  0.99955  and recall is  0.93631  for Food Irrelevant class\n"
     ]
    }
   ],
   "source": [
    "nb_metrics = naive_bayes_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "print_metrics(nb_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SVM Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "\n",
    "def svm_model(train_review_matrix, train_df, test_review_matrix, test_df):\n",
    "    svm_clf = svm.LinearSVC().fit(train_review_matrix, train_df['isFoodRelevant'])\n",
    "    predicted_classes = svm_clf.predict(test_review_matrix)\n",
    "    return calculateMetrics(predicted_classes, test_df['isFoodRelevant']), svm_clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.861\n",
      "Precision is  0.12256  and recall is  0.97071  for Food Relevant class\n",
      "Precision is  0.99931  and recall is  0.85877  for Food Irrelevant class\n"
     ]
    }
   ],
   "source": [
    "svm_metrics = svm_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "print_metrics(svm_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest Classifier (Not able to run KNN Classifier)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def rf_model(train_review_matrix, train_df, test_review_matrix, test_df):\n",
    "    rf_clf = RandomForestClassifier()\n",
    "    rf_clf.fit(train_review_matrix, train_df['isFoodRelevant'])\n",
    "    predicted_classes = rf_clf.predict(test_review_matrix)\n",
    "    return calculateMetrics(predicted_classes, test_df['isFoodRelevant']), rf_clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.78475\n",
      "Precision is  0.08381  and recall is  0.98745  for Food Relevant class\n",
      "Precision is  0.99967  and recall is  0.78063  for Food Irrelevant class\n"
     ]
    }
   ],
   "source": [
    "rf_metrics = rf_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "print_metrics(rf_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setting 2: Cross Validation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_train_test_matrices(train_indices, test_indices, train_df, test_df):\n",
    "    train_vectorizer2 = CountVectorizer(token_pattern = r'\\b\\w+\\b')\n",
    "    train_review_matrix = train_vectorizer2.fit_transform(train_df['review_clean'])\n",
    "    test_review_matrix = train_vectorizer2.transform(test_df['review_clean'])\n",
    "    return train_review_matrix, test_review_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calculate_avg_metrics(metrics, model_index):\n",
    "    accurracy_avg = sum([tuple[0] for tuple in metrics[model_index]])/5\n",
    "    precision_food_relevant_avg = sum([tuple[1] for tuple in metrics[model_index]])/5\n",
    "    recall_food_relevant_avg = sum([tuple[2] for tuple in metrics[model_index]])/5\n",
    "    precision_food_irrelevant_avg = sum([tuple[3] for tuple in metrics[model_index]])/5\n",
    "    recall_food_irrelevant_avg = sum([tuple[4] for tuple in metrics[model_index]])/5\n",
    "    avg_metrics = (accurracy_avg, precision_food_relevant_avg, recall_food_relevant_avg, precision_food_irrelevant_avg, recall_food_irrelevant_avg)\n",
    "    print_metrics(avg_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def analyse_kfold_data(metrics):\n",
    "    print \"Decision Tree Model:\"\n",
    "    calculate_avg_metrics(metrics, 0)\n",
    "    print \"\"\n",
    "    print \"Naive Bayes Model\"\n",
    "    calculate_avg_metrics(metrics, 1)\n",
    "    print \"\"\n",
    "    print \"SVM\"\n",
    "    calculate_avg_metrics(metrics, 2)\n",
    "    print \"\"\n",
    "    print \"Random forest\"\n",
    "    calculate_avg_metrics(metrics, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sklearn/metrics/classification.py:1137: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples.\n",
      "  'recall', 'true', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "kf = KFold(n_splits = 5)\n",
    "\n",
    "m, itr = 4, 5 \n",
    "\n",
    "kfold_metrics = [[0 for x in range(itr)] for y in range(m)]\n",
    "idx = 0\n",
    "for train_indices, test_indices in kf.split(review_df):\n",
    "    train_df = review_df.iloc[train_indices]\n",
    "    test_df = review_df.iloc[test_indices]\n",
    "    train_review_matrix, test_review_matrix = get_train_test_matrices(train_indices, test_indices, train_df, test_df)\n",
    "    kfold_metrics[0][idx] = decision_tree_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "    kfold_metrics[1][idx] = naive_bayes_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "    kfold_metrics[2][idx] = svm_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "    kfold_metrics[3][idx] = rf_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "    idx = idx + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree Model:\n",
      "Accuracy:  0.8566\n",
      "Precision is  0.73125  and recall is  0.68607  for Food Relevant class\n",
      "Precision is  0.81424  and recall is  0.8633  for Food Irrelevant class\n",
      "\n",
      "Naive Bayes Model\n",
      "Accuracy:  0.93578\n",
      "Precision is  0.7591  and recall is  0.75242  for Food Relevant class\n",
      "Precision is  0.90816  and recall is  0.92092  for Food Irrelevant class\n",
      "\n",
      "SVM\n",
      "Accuracy:  0.91062\n",
      "Precision is  0.75654  and recall is  0.72685  for Food Relevant class\n",
      "Precision is  0.87042  and recall is  0.91541  for Food Irrelevant class\n",
      "\n",
      "Random forest\n",
      "Accuracy:  0.8728\n",
      "Precision is  0.76162  and recall is  0.66994  for Food Relevant class\n",
      "Precision is  0.8046  and recall is  0.92082  for Food Irrelevant class\n"
     ]
    }
   ],
   "source": [
    "analyse_kfold_data(kfold_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1.5: Improving  classifier \n",
    "\n",
    "I think we can do better!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Trying TFidf Transformer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "def get_train_test_matrices_v2(train_indices, test_indices, train_df, test_df):\n",
    "    tfidf_vectoriser = TfidfVectorizer(token_pattern = r'\\b\\w\\w\\w+\\b', stop_words='english')\n",
    "    train_review_matrix = tfidf_vectoriser.fit_transform(train_df['review_clean'])\n",
    "    test_review_matrix = tfidf_vectoriser.transform(test_df['review_clean'])\n",
    "    return train_review_matrix, test_review_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sklearn/metrics/classification.py:1137: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples.\n",
      "  'recall', 'true', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree Model:\n",
      "Accuracy:  0.8677\n",
      "Precision is  0.73749  and recall is  0.69792  for Food Relevant class\n",
      "Precision is  0.82902  and recall is  0.87058  for Food Irrelevant class\n",
      "\n",
      "Naive Bayes Model\n",
      "Accuracy:  0.91895\n",
      "Precision is  0.76601  and recall is  0.73799  for Food Relevant class\n",
      "Precision is  0.88303  and recall is  0.92748  for Food Irrelevant class\n",
      "\n",
      "SVM\n",
      "Accuracy:  0.93345\n",
      "Precision is  0.77229  and recall is  0.7375  for Food Relevant class\n",
      "Precision is  0.88769  and recall is  0.94656  for Food Irrelevant class\n",
      "\n",
      "Random forest\n",
      "Accuracy:  0.89355\n",
      "Precision is  0.76327  and recall is  0.69638  for Food Relevant class\n",
      "Precision is  0.8334  and recall is  0.92411  for Food Irrelevant class\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "kf = KFold(n_splits = 5)\n",
    "\n",
    "m, itr = 4, 5 \n",
    "\n",
    "kfold_metrics = [[0 for x in range(itr)] for y in range(m)]\n",
    "idx = 0\n",
    "for train_indices, test_indices in kf.split(review_df):\n",
    "    train_df = review_df.iloc[train_indices]\n",
    "    test_df = review_df.iloc[test_indices]\n",
    "    train_review_matrix, test_review_matrix = get_train_test_matrices_v2(train_indices, test_indices, train_df, test_df)\n",
    "    kfold_metrics[0][idx] = decision_tree_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "    kfold_metrics[1][idx] = naive_bayes_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "    kfold_metrics[2][idx] = svm_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "    kfold_metrics[3][idx] = rf_model(train_review_matrix, train_df, test_review_matrix, test_df)[0]\n",
    "    idx = idx + 1\n",
    "    \n",
    "analyse_kfold_data(kfold_metrics)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In the above approach, I am removing all the standard stop words in the English dictionary from the review text. Also, I am removing any word that has length less than 3 from the reviews. Following that, I am using Tf-idf scoring criteria for feature space.\n",
    "\n",
    "If we only compare the accuracy of this Tf-idf model with the vanilla term-frequency model, there hasn't been much change in the accuracy from one model to other.\n",
    "\n",
    "However, the there is improvement in the precision of Food-Relevant label from ~75% to ~77%. This is good improvement.\n",
    "\n",
    "\n",
    "### 2) Using Bi-gram language model with svm "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'bigram_vectoriser' (CountVectorizer)\n",
      "Stored 'train_bigram_matrix' (csr_matrix)\n",
      "Stored 'test_bigram_matrix' (csr_matrix)\n"
     ]
    }
   ],
   "source": [
    "bigram_vectoriser = CountVectorizer(ngram_range=(1,2), token_pattern = r'\\b\\w\\w\\w+\\b', stop_words='english')\n",
    "train_bigram_matrix = bigram_vectoriser.fit_transform(train_df[\"review_clean\"])\n",
    "test_bigram_matrix = bigram_vectoriser.transform(test_df[\"review_clean\"])\n",
    "%store bigram_vectoriser\n",
    "%store train_bigram_matrix\n",
    "%store test_bigram_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'south road': 720904,\n",
       " u'includ yelp': 390221,\n",
       " u'honest someth': 372738,\n",
       " u'case involv': 124992,\n",
       " u'contemporari rust': 172617,\n",
       " u'filet fix': 292768,\n",
       " u'pastrami virgin': 555082,\n",
       " u'includ doubl': 389644,\n",
       " u'haven chees': 359027,\n",
       " u'huge fanfar': 381328,\n",
       " u'hope beef': 374067,\n",
       " u'lot potenti': 455348,\n",
       " u'thing besid': 782607,\n",
       " u'theme peta': 778782,\n",
       " u'happier couldn': 356470,\n",
       " u'beauti gurgl': 69001,\n",
       " u'person add': 564347,\n",
       " u'gone won': 331875,\n",
       " u'middl stage': 488090,\n",
       " u'overcharg fifti': 543176,\n",
       " u'came helper': 117748,\n",
       " u'bar byron': 61519,\n",
       " u'burro total': 110981,\n",
       " u'inexpens delici': 392342,\n",
       " u'piec look': 571754,\n",
       " u'reflect littl': 626839,\n",
       " u'combin mexican': 159842,\n",
       " u'passion thi': 553910,\n",
       " u'possibl disgust': 588977,\n",
       " u'plu tri': 583484,\n",
       " u'ars littl': 42584,\n",
       " u'dri rib': 239358,\n",
       " u'place shadi': 577274,\n",
       " u'dancer like': 197624,\n",
       " u'sonic great': 717362,\n",
       " u'purchas thought': 607880,\n",
       " u'trojan': 810460,\n",
       " u'wine pitcher': 863880,\n",
       " u'strike food': 746536,\n",
       " u'fact chicago': 278971,\n",
       " u'littl prici': 445885,\n",
       " u'unit point': 820697,\n",
       " u'miss sometim': 493120,\n",
       " u'bracken great': 97130,\n",
       " u'run film': 654396,\n",
       " u'thi time': 782048,\n",
       " u'littl price': 445883,\n",
       " u'best sashimi': 78792,\n",
       " u'andouil potato': 27603,\n",
       " u'display larg': 228069,\n",
       " u'becaus ensalada': 70106,\n",
       " u'corpor came': 177954,\n",
       " u'week food': 853904,\n",
       " u'airflow mean': 19107,\n",
       " u'way year': 852080,\n",
       " u'outsid vibe': 542187,\n",
       " u'say san': 669674,\n",
       " u'cruz came': 191413,\n",
       " u'els attitud': 252221,\n",
       " u'jar thi': 404690,\n",
       " u'live theme': 447122,\n",
       " u'madison histori': 463217,\n",
       " u'sort bird': 718894,\n",
       " u'lemon wedg': 432602,\n",
       " u'bundl plu': 109093,\n",
       " u'split larg': 727276,\n",
       " u'pleas add': 581397,\n",
       " u'plate realli': 579979,\n",
       " u'guarante excel': 347346,\n",
       " u'singer guitar': 700787,\n",
       " u'scienc cours': 672352,\n",
       " u'start earlyish': 735541,\n",
       " u'good reput': 333965,\n",
       " u'clear jon': 151658,\n",
       " u'barn hop': 63513,\n",
       " u'time realli': 791117,\n",
       " u'creami mushroom': 187287,\n",
       " u'hang lunch': 354706,\n",
       " u'experi meet': 273477,\n",
       " u'finish cup': 294957,\n",
       " u'buy bonsai': 113243,\n",
       " u'grade math': 338543,\n",
       " u'rave later': 617108,\n",
       " u'need minor': 509108,\n",
       " u'possibl gave': 589023,\n",
       " u'quickli left': 611732,\n",
       " u'voic heavili': 838303,\n",
       " u'talk chang': 765680,\n",
       " u'grand children': 338791,\n",
       " u'pack veri': 546273,\n",
       " u'littl laid': 445517,\n",
       " u'extrem attract': 276489,\n",
       " u'south north': 720884,\n",
       " u'veal chop': 829890,\n",
       " u'ive hundr': 403322,\n",
       " u'accordingli filet': 9728,\n",
       " u'suppli did': 755959,\n",
       " u'bring compimentari': 101580,\n",
       " u'glanc insid': 328724,\n",
       " u'tartar lobster': 767961,\n",
       " u'shrimp sandwhich': 695901,\n",
       " u'cuban puerto': 191594,\n",
       " u'guess store': 348063,\n",
       " u'toast start': 793984,\n",
       " u'servic van': 685864,\n",
       " u'brush charm': 105438,\n",
       " u'friendli earli': 315279,\n",
       " u'guess stori': 348064,\n",
       " u'attent bear': 49481,\n",
       " u'word bianco': 868852,\n",
       " u'waitstaff just': 841483,\n",
       " u'dish brave': 226548,\n",
       " u'act 6th': 10586,\n",
       " u'kid chees': 415271,\n",
       " u'servic vax': 685868,\n",
       " u'tank becaus': 766960,\n",
       " u'tri dang': 807713,\n",
       " u'tri danc': 807712,\n",
       " u'oil lamb': 527422,\n",
       " u'water ridicul': 849965,\n",
       " u'annoy bean': 30090,\n",
       " u'wait opinion': 840013,\n",
       " u'washer spread': 847661,\n",
       " u'quit strong': 612878,\n",
       " u'price train': 597517,\n",
       " u'00am lunch': 96,\n",
       " u'bad mizthra': 57662,\n",
       " u'entertain alreadi': 259009,\n",
       " u'thing critic': 782809,\n",
       " u'ate juan': 47584,\n",
       " u'credit check': 187827,\n",
       " u'fresh daili': 311450,\n",
       " u'spinach salad': 726791,\n",
       " u'super pysch': 755297,\n",
       " u'hour juansday': 378024,\n",
       " u'wise tini': 865401,\n",
       " u'seen aida': 678207,\n",
       " u'locat anoth': 448837,\n",
       " u'skillet shrimp': 704218,\n",
       " u'rebandag check': 622158,\n",
       " u'cream huckleberri': 186872,\n",
       " u'place biltmor': 575270,\n",
       " u'option complet': 535810,\n",
       " u'center fresh': 128726,\n",
       " u'prepay anymor': 593754,\n",
       " u'soup thai': 720435,\n",
       " u'pleasant possibl': 582098,\n",
       " u'julian provid': 409850,\n",
       " u'germaphob realli': 325730,\n",
       " u'reproach great': 634415,\n",
       " u'lukewarm cold': 460348,\n",
       " u'pictur dark': 570863,\n",
       " u'varieti total': 829511,\n",
       " u'kudo waitress': 422437,\n",
       " u'miel fine': 488430,\n",
       " u'day xma': 201423,\n",
       " u'locat diamondback': 449070,\n",
       " u'gave singl': 323369,\n",
       " u'similar issu': 698270,\n",
       " u'roll cake': 648982,\n",
       " u'share fresh': 689256,\n",
       " u'easili impress': 246873,\n",
       " u'titl artist': 793591,\n",
       " u'member buri': 482070,\n",
       " u'coe great': 156646,\n",
       " u'noth pretti': 520643,\n",
       " u'pull headrest': 606449,\n",
       " u'bought ken': 94764,\n",
       " u'hang soon': 354800,\n",
       " u'addict summer': 12727,\n",
       " u'happen purs': 355524,\n",
       " u'notic glare': 521110,\n",
       " u'live pretent': 446996,\n",
       " u'acknowledg parti': 10375,\n",
       " u'member non': 482208,\n",
       " u'accord worker': 9715,\n",
       " u'owner sens': 545336,\n",
       " u'clean sanitari': 151020,\n",
       " u'best fri': 78165,\n",
       " u'month guarante': 497946,\n",
       " u'owner sent': 545337,\n",
       " u'short hang': 693882,\n",
       " u'seller use': 680488,\n",
       " u'short hand': 693881,\n",
       " u'declin took': 205304,\n",
       " u'sure sever': 757710,\n",
       " u'clearli lane': 151934,\n",
       " u'sausag omelet': 667059,\n",
       " u'becaus month': 70624,\n",
       " u'ask mother': 45015,\n",
       " u'valu base': 828375,\n",
       " u'profit place': 602607,\n",
       " u'limp year': 441706,\n",
       " u'mac cheeseburg': 462136,\n",
       " u'couldn perfect': 179904,\n",
       " u'strang couldn': 744910,\n",
       " u'listen idiot': 443844,\n",
       " u'bee problem': 72163,\n",
       " u'rudest peopl': 653621,\n",
       " u'await appar': 53319,\n",
       " u'push near': 608493,\n",
       " u'order incent': 537807,\n",
       " u'care young': 123203,\n",
       " u'nosomuch nielsen': 519324,\n",
       " u'pumpkin free': 606875,\n",
       " u'greasi tabl': 340282,\n",
       " u'lover good': 458862,\n",
       " u'desir start': 213734,\n",
       " u'inquiri friend': 394530,\n",
       " u'pain cute': 547381,\n",
       " u'phx sore': 569094,\n",
       " u'bonu shoe': 91609,\n",
       " u'like fajita': 438470,\n",
       " u'sweet befor': 760625,\n",
       " u'leather repair': 430074,\n",
       " u'escap establish': 261858,\n",
       " u'instal themselv': 396065,\n",
       " u'size wonder': 703520,\n",
       " u'hot offer': 376497,\n",
       " u'chocol die': 144236,\n",
       " u'rotten hoppiest': 652062,\n",
       " u'involv repair': 399864,\n",
       " u'volcan erupt': 838431,\n",
       " u'4pm staff': 4694,\n",
       " u'prefer old': 592702,\n",
       " u'stuck indi': 747593,\n",
       " u'compar wait': 164852,\n",
       " u'rock close': 648125,\n",
       " u'disgust egg': 226291,\n",
       " u'opt cobb': 535427,\n",
       " u'cent hot': 128438,\n",
       " u'age check': 16774,\n",
       " u'cycleri got': 196141,\n",
       " u'verbal inform': 832166,\n",
       " u'primo rate': 598370,\n",
       " u'fir': 295320,\n",
       " u'welcom invit': 856051,\n",
       " ...}"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bigram_vectoriser.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.9245\n",
      "Precision is  0.0  and recall is  0.0  for Food Relevant class\n",
      "Precision is  1.0  and recall is  0.9245  for Food Irrelevant class\n"
     ]
    }
   ],
   "source": [
    "naive_bayes_result = naive_bayes_model(train_bigram_matrix, train_df, test_bigram_matrix, test_df)\n",
    "nb_metrics = naive_bayes_result[0]\n",
    "nb_bigram_model = naive_bayes_result[1]\n",
    "print_metrics(nb_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.91887\n",
      "Precision is  0.0  and recall is  0.0  for Food Relevant class\n",
      "Precision is  1.0  and recall is  0.91887  for Food Irrelevant class\n"
     ]
    }
   ],
   "source": [
    "svm_result = svm_model(train_bigram_matrix, train_df, test_bigram_matrix, test_df)\n",
    "svm_metrics = svm_result[0]\n",
    "svm_bigram_model = svm_result[1]\n",
    "print_metrics(svm_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In the above model, we are trying to create a model of bigrams using TF-idf score criteria. Although, the accuracy of the model is almost same as the previous model - ~92%, the precision of the model for Food-Relevant data is 0, which is not desirable. Therefore, we need to tinker more with this model to make it usable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Other possible changes to improve the model**\n",
    "\n",
    "We can get the food related words from the wordnet database. Use those words in our dataset assigning them more weight, and then train our model.\n",
    "This approach will lead to increase in our precison for food relevant data and also improve accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This analysis is done on the data obtained from Scenario 1 ( 70:30 data split)\n",
    "#### 1) Using chi-squared stats to select the best 10 features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import chi2\n",
    "\n",
    "chi2score = chi2(train_tfidf_matrix, train_df[\"isFoodRelevant\"])[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wscores = zip(tfidf_vectoriser.get_feature_names(), chi2score)\n",
    "top_chi_scores = sorted(wscores, key = lambda score: score[1], reverse = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'food', 247.86341695810285),\n",
       " (u'car', 216.43206239117248),\n",
       " (u'zoo', 169.79510114756724),\n",
       " (u'exhibit', 164.57384069662658),\n",
       " (u'dr', 148.92894006448495),\n",
       " (u'hotel', 132.8543048433601),\n",
       " (u'wash', 119.96397070000924),\n",
       " (u'help', 119.54576501584486),\n",
       " (u'chicken', 118.39841654171589),\n",
       " (u'order', 116.36340838169082),\n",
       " (u'store', 113.95327200127554),\n",
       " (u'sandwich', 111.8770453089535),\n",
       " (u'repair', 108.27856431983842),\n",
       " (u'room', 107.85684986666467),\n",
       " (u'salon', 100.55230964550813),\n",
       " (u'lunch', 98.835295490317293),\n",
       " (u'appoint', 95.223255716213444),\n",
       " (u'menu', 95.102257119134464),\n",
       " (u'salad', 94.998212599730721),\n",
       " (u'delici', 93.619434154590948)]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_chi_scores[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using chi2 score and tf-idf criteria, the most informative words in the model are food, car, zoo,exhibit, hotel, chicken and other features listed above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2) Since the accuracy of SVM is the best among all the 4 classifiers used, we are using SVM coefficients to select the best features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'food', 5.6545514949900513),\n",
       " (u'walgreen', 3.6767937382963067),\n",
       " (u'sandwich', 3.3301366413290108),\n",
       " (u'eat', 3.1690883535734256),\n",
       " (u'costco', 3.1671206421533418),\n",
       " (u'groceri', 3.0855110245009549),\n",
       " (u'dine', 2.9106194049091836),\n",
       " (u'menu', 2.7139483467701049),\n",
       " (u'safeway', 2.5537099282064761),\n",
       " (u'produc', 2.5485950932757082),\n",
       " (u'wine', 2.5149116783688812),\n",
       " (u'order', 2.5099888579838501),\n",
       " (u'waiter', 2.4841887475464883),\n",
       " (u'restaur', 2.3880628110457427),\n",
       " (u'coffe', 2.3761644345899517),\n",
       " (u'starbuck', 2.3058105339204205),\n",
       " (u'chines', 2.2940349501068265),\n",
       " (u'bruschetta', 2.2766494285546086),\n",
       " (u'server', 2.2241605468262517),\n",
       " (u'bakeri', 2.147795214640976)]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_names = tfidf_vectoriser.get_feature_names()\n",
    "feature_scores = zip(features_names, svm_clf.coef_[0])\n",
    "feature_scores_sorted = sorted(feature_scores, key = lambda x: x[1], reverse = True)\n",
    "feature_scores_sorted[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The top 20 features as given by the SVM are listed above. {food, walgreen, sandwich, eat, costco and others}\n",
    "\n",
    "#### These features look more relevant (food related) as compared to the features obtained by the chi2 criteria"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}