Text Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification with Classical ML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step of this project is to develop a **classification model to predict the positive/negative labels** of movie reviews. This prediction will be **based solely on the text content** of the reviews.\n",
    "\n",
    "The data used in this project is the polarity dataset v2.0, http://www.cs.cornell.edu/people/pabo/movie-review-data/, of Cornell University."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Perform initial imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/moviereviews.tsv\", sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Check the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>neg</td>\n",
       "      <td>how do films like mouse hunt get into theatres...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "      <td>some talented actresses are blessed with a dem...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>pos</td>\n",
       "      <td>this has been an extraordinary year for austra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>pos</td>\n",
       "      <td>according to hollywood movies made in last few...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>neg</td>\n",
       "      <td>my first press screening of 1998 and already i...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                             review\n",
       "0   neg  how do films like mouse hunt get into theatres...\n",
       "1   neg  some talented actresses are blessed with a dem...\n",
       "2   pos  this has been an extraordinary year for austra...\n",
       "3   pos  according to hollywood movies made in last few...\n",
       "4   neg  my first press screening of 1998 and already i..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pos    1000\n",
       "neg    1000\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check number of both labels\n",
    "\n",
    "df['label'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "how do films like mouse hunt get into theatres ? \r\n",
      "isn't there a law or something ? \r\n",
      "this diabolical load of claptrap from steven speilberg's dreamworks studio is hollywood family fare at its deadly worst . \r\n",
      "mouse hunt takes the bare threads of a plot and tries to prop it up with overacting and flat-out stupid slapstick that makes comedies like jingle all the way look decent by comparison . \r\n",
      "writer adam rifkin and director gore verbinski are the names chiefly responsible for this swill . \r\n",
      "the plot , for what its worth , concerns two brothers ( nathan lane and an appalling lee evens ) who inherit a poorly run string factory and a seemingly worthless house from their eccentric father . \r\n",
      "deciding to check out the long-abandoned house , they soon learn that it's worth a fortune and set about selling it in auction to the highest bidder . \r\n",
      "but battling them at every turn is a very smart mouse , happy with his run-down little abode and wanting it to stay that way . \r\n",
      "the story alternates between unfunny scenes of the brothers bickering over what to do with their inheritance and endless action sequences as the two take on their increasingly determined furry foe . \r\n",
      "whatever promise the film starts with soon deteriorates into boring dialogue , terrible overacting , and increasingly uninspired slapstick that becomes all sound and fury , signifying nothing . \r\n",
      "the script becomes so unspeakably bad that the best line poor lee evens can utter after another run in with the rodent is : \" i hate that mouse \" . \r\n",
      "oh cringe ! \r\n",
      "this is home alone all over again , and ten times worse . \r\n",
      "one touching scene early on is worth mentioning . \r\n",
      "we follow the mouse through a maze of walls and pipes until he arrives at his makeshift abode somewhere in a wall . \r\n",
      "he jumps into a tiny bed , pulls up a makeshift sheet and snuggles up to sleep , seemingly happy and just wanting to be left alone . \r\n",
      "it's a magical little moment in an otherwise soulless film . \r\n",
      "a message to speilberg : if you want dreamworks to be associated with some kind of artistic credibility , then either give all concerned in mouse hunt a swift kick up the arse or hire yourself some decent writers and directors . \r\n",
      "this kind of rubbish will just not do at all . \r\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# check first negative review\n",
    "\n",
    "print(df['review'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this has been an extraordinary year for australian films . \r\n",
      " \" shine \" has just scooped the pool at the australian film institute awards , picking up best film , best actor , best director etc . to that we can add the gritty \" life \" ( the anguish , courage and friendship of a group of male prisoners in the hiv-positive section of a jail ) and \" love and other catastrophes \" ( a low budget gem about straight and gay love on and near a university campus ) . \r\n",
      "i can't recall a year in which such a rich and varied celluloid library was unleashed from australia . \r\n",
      " \" shine \" was one bookend . \r\n",
      "stand by for the other one : \" dead heart \" . \r\n",
      ">from the opening credits the theme of division is established . \r\n",
      "the cast credits have clear and distinct lines separating their first and last names . \r\n",
      "bryan | brown . \r\n",
      "in a desert settlement , hundreds of kilometres from the nearest town , there is an uneasy calm between the local aboriginals and the handful of white settlers who live nearby . \r\n",
      "the local police officer has the task of enforcing \" white man's justice \" to the aboriginals . \r\n",
      "these are people with a proud 40 , 000 year heritage behind them . \r\n",
      "naturally , this includes their own system of justice ; key to which is \" payback \" . \r\n",
      "an eye for an eye . \r\n",
      "revenge . \r\n",
      "usually extracted by the spearing through of the recipient's thigh . \r\n",
      "brown , as the officer , manages quite well to keep the balance . \r\n",
      "he admits that he has to 'bend the rules' a bit , including actively encouraging at least one brutal \" payback \" . \r\n",
      " ( be warned that this scene , near the start , is not for the squeamish ) . \r\n",
      "the local priest - an aboriginal , but in the \" white fellas \" church - has a foot on either side of the line . \r\n",
      "he is , figuratively and literally , in both camps . \r\n",
      "ernie dingo brings a great deal of understanding to this role as the man in the middle . \r\n",
      "he is part churchman and part politician . \r\n",
      "however the tension , like the heat , flies and dust , is always there . \r\n",
      "whilst her husband - the local teacher - is in church , white lady kate ( milliken ) and her aborginal friend tony , ( pedersen ) have gone off into the hills . \r\n",
      "he takes her to a sacred site , even today strictly men-only . \r\n",
      "she appears to not know this . \r\n",
      "tony tells her that this is a special place , an initiation place . \r\n",
      "he then makes love to her , surrounded by ancient rock art . \r\n",
      "the community finds out about this sacrilegious act and it's payback time . \r\n",
      "the fuse is lit and the brittle inter-racial peace is shattered . \r\n",
      "everyone is affected in the fall out . \r\n",
      "to say more is to give away the details of this finely crafted film . \r\n",
      "suffice to say it's a rewarding experience . \r\n",
      "bryan brown , acting and co-producing , is the pivotal character . \r\n",
      "his officer is real , human and therefore flawed . \r\n",
      "brown comments that he expects audiences to feel warmth towards the man , then suddenly feel angry about him . \r\n",
      "it wasn't long ago that i visited central australia - ayers rock ( uluru ) and alice springs - for the first time . \r\n",
      "the wide-screen cinematography shows the dead heart of australia in a way that captures it's vicious beauty , but never deteriorates into a moving slide show , in which the gorgeous background dominates those pesky actors in the foreground . \r\n",
      "the cultural clash has provided the thesis for many a film ; from the western to the birdcage . \r\n",
      "at least three excellent australian films have covered the aboriginal people and the line between them and we anglo-saxon 'invaders' : \" jedda \" , \" the chant of jimmie blacksmith \" and \" the last wave \" . \r\n",
      "in a year when the race 'debate' has reared up in australia , it is nourishing to see such an intelligent , non-judgemental film as \" dead heart \" . \r\n",
      "the aboriginal priest best sums this up . \r\n",
      "he is asked to say if he is a \" black fella or white fella \" . \r\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# check first positive review\n",
    "\n",
    "print(df['review'][2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Check missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "label      0\n",
       "review    35\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 35 missing reviews. We should delete these rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove rows with missing reviews\n",
    "\n",
    "df.dropna(inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "label     0\n",
       "review    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check missing values\n",
    "\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Check empty strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using the isspace() method\n",
    "\n",
    "empty_strings = []\n",
    "\n",
    "for i, lb, rv in df.itertuples():\n",
    "    if rv.isspace():\n",
    "        empty_strings.append(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[57, 71, 147, 151, 283, 307, 313, 323, 343, 351, 427, 501, 633, 675, 815, 851, 977, 1079, 1299, 1455, 1493, 1525, 1531, 1763, 1851, 1905, 1993]\n",
      "27\n"
     ]
    }
   ],
   "source": [
    "print(empty_strings)\n",
    "print(len(empty_strings))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 27 reviews that correspond to empty strings. These reviews are identified by the indices in the empty_strings list. We should remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove rows with empty strings\n",
    "\n",
    "df.drop(empty_strings, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1938"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check length\n",
    "\n",
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pos    969\n",
       "neg    969\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check number of both labels\n",
    "\n",
    "df['label'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have 1938 movie reviews (969 are positive and 969 are negative)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Split the data into train and test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = df['review']\n",
    "y = df['label']\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Build pipeline to vectorize the data and train/fit the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('tfidf',\n",
       "                 TfidfVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.float64'>,\n",
       "                                 encoding='utf-8', input='content',\n",
       "                                 lowercase=True, max_df=1.0, max_features=None,\n",
       "                                 min_df=1, ngram_range=(1, 1), norm='l2',\n",
       "                                 preprocessor=None, smooth_idf=True,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 sublinear_tf=False,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=None, use_idf=True,\n",
       "                                 vocabulary=None)),\n",
       "                ('clf',\n",
       "                 LinearSVC(C=1.0, class_weight=None, dual=True,\n",
       "                           fit_intercept=True, intercept_scaling=1,\n",
       "                           loss='squared_hinge', max_iter=1000,\n",
       "                           multi_class='ovr', penalty='l2', random_state=None,\n",
       "                           tol=0.0001, verbose=0))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "text_clf = Pipeline([('tfidf', TfidfVectorizer()),\n",
    "                    ('clf', LinearSVC())])\n",
    "\n",
    "text_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Make predictions with the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = text_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. Evaluate the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix, classification_report, accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[235  47]\n",
      " [ 41 259]]\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix\n",
    "\n",
    "print(confusion_matrix(y_test, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.85      0.83      0.84       282\n",
      "         pos       0.85      0.86      0.85       300\n",
      "\n",
      "    accuracy                           0.85       582\n",
      "   macro avg       0.85      0.85      0.85       582\n",
      "weighted avg       0.85      0.85      0.85       582\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# classification report\n",
    "\n",
    "print(classification_report(y_test, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8487972508591065\n"
     ]
    }
   ],
   "source": [
    "# accuracy score\n",
    "\n",
    "print(accuracy_score(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based solely on the text content of the reviews we've managed to correctly classify **84,9%** of them as positive or negative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10. Test the fitted model on new data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['pos' 'neg']\n"
     ]
    }
   ],
   "source": [
    "# make up some reviews and test the model\n",
    "\n",
    "my_review = \"The movie was great! The main actors were superb and the storyline was convincing.\"\n",
    "my_review_2 = \"Terrible movie! A complete waste of money.\"\n",
    "\n",
    "reviews = [my_review, my_review_2]\n",
    "\n",
    "print(text_clf.predict(reviews))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though this is a very simple model, everything seems to be working just fine!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've used **sklearn's TfidfVectorizer** - our text vectorization method - and the **Linear Support Vector Classification algorithm** to build our model. \n",
    "\n",
    "Let's consider other alternatives in terms of:\n",
    "* text normalization\n",
    "* text vectorization\n",
    "* learning algorithms\n",
    "\n",
    "Different **text classification** scenarios may require different approaches.\n",
    "\n",
    "In step 2 we'll be focusing on the different ways how we can **normalize** our documents (reviews)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text normalization allows us to **reduce our vocabulary size** and hence the number of dimensions of our feature space, which results in efficiency gains in terms of storage and processing.\n",
    "\n",
    "However, text normalization comes with the cost of **loss of information**.\n",
    "\n",
    "It is up to us to find the right balance between those efficiency gains and our model's accuracy, considering both recall and precision.\n",
    "\n",
    "Our ultimate goal is to **improve the overall performance of our NLP pipeline**, given the specific objectives of our project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how the text vectorization method we applied in step 1 deals with text normalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Check parameters of TfidfVectorizer  used in our pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'analyzer': 'word',\n",
       " 'binary': False,\n",
       " 'decode_error': 'strict',\n",
       " 'dtype': numpy.float64,\n",
       " 'encoding': 'utf-8',\n",
       " 'input': 'content',\n",
       " 'lowercase': True,\n",
       " 'max_df': 1.0,\n",
       " 'max_features': None,\n",
       " 'min_df': 1,\n",
       " 'ngram_range': (1, 1),\n",
       " 'norm': 'l2',\n",
       " 'preprocessor': None,\n",
       " 'smooth_idf': True,\n",
       " 'stop_words': None,\n",
       " 'strip_accents': None,\n",
       " 'sublinear_tf': False,\n",
       " 'token_pattern': '(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       " 'tokenizer': None,\n",
       " 'use_idf': True,\n",
       " 'vocabulary': None}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_clf.named_steps['tfidf'].get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regarding text normalization and having a look at the default parameters that were used before, we can see that:\n",
    "* **tokens of 2 or more alphanumeric characters** were selected (**punctuation was ignored**)\n",
    "* tokens were **lowercased**\n",
    "\n",
    "And that:\n",
    "* **no stopwords** were removed\n",
    "* apart from that, **no stemmer and no lemmatizer** were used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Check dataset size and vocabulary size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1938"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33855"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(text_clf.named_steps['tfidf'].vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a reasonably small dataset and vocabulary, and thus we can choose to keep most of the information from our reviews without hindering our NLP pipeline performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Supposing this was not the case, what could we do? In order to maximize the range of possibilities, we could make some changes to our pipeline by **creating custom transformers for both text normalization and text vectorization** and **building some different models** using several distinct algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by creating a **custom text normalization transformer** that allows us to:\n",
    "* lowercase words\n",
    "* remove punctuation\n",
    "* remove stopwords\n",
    "* stem words\n",
    "* lemmatize words\n",
    "* lemmatize and stem words (in this order)\n",
    "\n",
    "This is our step 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Perform necessary imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unicodedata\n",
    "import nltk\n",
    "from nltk import pos_tag, sent_tokenize, wordpunct_tokenize\n",
    "from nltk.corpus import wordnet as wn\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from sklearn.base import BaseEstimator, TransformerMixin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Create a custom text normalization transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TextNormalizer(BaseEstimator, TransformerMixin):\n",
    "\n",
    "    def __init__(self, language='english', lemmatizer=WordNetLemmatizer(), stemmer=None, vectorizer='gensim'):\n",
    "        '''\n",
    "        The default language is English.\n",
    "        The default lemmatizer is WordNetLemmatizer. If you don't want to use a lemmatizer,\n",
    "        set it to None.\n",
    "        To use a stemmer, set stemmer to 'porter' (PorterStemmer) or to 'snowball' (SnowballStemmer),\n",
    "        otherwise, it defaults to None.\n",
    "        The default vectorizer is Gensim - this way you get a list of tokens as an output.\n",
    "        If you need a string instead, set vectorizer to 'other'.\n",
    "        '''\n",
    "        self.language = language\n",
    "        self.stopwords = set(nltk.corpus.stopwords.words(self.language))\n",
    "        \n",
    "        self.lemmatizer = lemmatizer\n",
    "        self.stemmer = stemmer\n",
    "        \n",
    "        self.vectorizer = vectorizer\n",
    "        \n",
    "    def is_punct(self, token):\n",
    "        # returns True if all characters of a token are punctuation signs\n",
    "        return all(unicodedata.category(char).startswith('P') for char in token)\n",
    "    \n",
    "    def is_stopword(self, token):\n",
    "        # returns True if the lowercased token is a stopword\n",
    "        return token.lower() in self.stopwords\n",
    "    \n",
    "    def lemmatize(self, token, tag):\n",
    "        '''\n",
    "        Converts Penn Treebank part-of-speech tags (the default tag set in nltk.pos_tag)\n",
    "        to WordNet tags - defaults to wn.Noun if the first letter of the Penn Treebank pos tag\n",
    "        is neither 'N', 'V', 'R' or 'J'.\n",
    "        Returns lemmatized token.\n",
    "        '''        \n",
    "        wordnet_tag = {\n",
    "            'N': wn.NOUN, \n",
    "            'V': wn.VERB,\n",
    "            'R': wn.ADV,\n",
    "            'J': wn.ADJ\n",
    "        }.get(tag[0], wn.NOUN)\n",
    "        \n",
    "        return self.lemmatizer.lemmatize(token, wordnet_tag)\n",
    "    \n",
    "    def stem(self, token):\n",
    "        if self.stemmer == 'porter':\n",
    "            self.stemmer = PorterStemmer()\n",
    "        elif self.stemmer == 'snowball':\n",
    "            self.stemmer = SnowballStemmer(self.language)\n",
    "        return self.stemmer.stem(token)\n",
    "    \n",
    "    def normalize(self, review):\n",
    "        '''\n",
    "        Normalizes review by removing punctuation and stopwords, lowercasing tokens,\n",
    "        and by lemmatizing or/and stemming tokens.\n",
    "        '''\n",
    "        if not self.lemmatizer == None and self.stemmer == None:\n",
    "            return [self.lemmatize(token, tag).lower() \n",
    "                    for sentence in sent_tokenize(review) \n",
    "                    for (token, tag) in pos_tag(wordpunct_tokenize(sentence)) \n",
    "                    if not self.is_punct(token) and not self.is_stopword(token)]\n",
    "\n",
    "        elif self.lemmatizer == None and not self.stemmer == None:\n",
    "            return [self.stem(token).lower() \n",
    "                    for sentence in sent_tokenize(review) \n",
    "                    for (token, tag) in pos_tag(wordpunct_tokenize(sentence)) \n",
    "                    if not self.is_punct(token) and not self.is_stopword(token)]\n",
    "        \n",
    "        elif self.lemmatizer == None and self.stemmer == None:\n",
    "            return [token.lower() \n",
    "                    for sentence in sent_tokenize(review) \n",
    "                    for (token, tag) in pos_tag(wordpunct_tokenize(sentence)) \n",
    "                    if not self.is_punct(token) and not self.is_stopword(token)]\n",
    "\n",
    "        else:\n",
    "            return [self.stem(lemmatized_token)\n",
    "                    for lemmatized_token in [self.lemmatize(token, tag).lower() \n",
    "                    for sentence in sent_tokenize(review) \n",
    "                    for (token, tag) in pos_tag(wordpunct_tokenize(sentence)) \n",
    "                    if not self.is_punct(token) and not self.is_stopword(token)]]\n",
    "    \n",
    "    def fit(self, reviews, labels=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, reviews):\n",
    "        # returns a list of tokens if vectorizer is set to 'gensim', otherwise, returns a string\n",
    "        if self.vectorizer == 'gensim':\n",
    "            return [self.normalize(review) for review in reviews]\n",
    "        else:\n",
    "            return [' '.join(self.normalize(review)) for review in reviews]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Test the custom text normalization transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate object with default values\n",
    "\n",
    "normalizer = TextNormalizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our class, TextNormalizer, inherits from BaseEstimator. Its objects are then estimators with **methods get_params() and set_params()**.\n",
    "\n",
    "Let's check our object's parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'language': 'english',\n",
       " 'lemmatizer': <WordNetLemmatizer>,\n",
       " 'stemmer': None,\n",
       " 'vectorizer': 'gensim'}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normalizer.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The language is **English**, the lemmatizer we'll be using is **WorNetLemmatizer**, **no stemming** will be performed, and the output will be a **list of tokens**, ready to use as input to a Gensim's text vectorization method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a side note, both stemming and lemmatization allow us to reduce the size of our vocabulary, but they accomplish it in different ways:\n",
    "* **stemming** uses a series of rules to remove word affixes and the resulting token **might not be a valid word**\n",
    "* **lemmatization** uses a knowledge base and may also use the word's POS tag to return the word's lemma, which **is always a valid word**\n",
    "\n",
    "This is why **it makes sense to use a stemmer right after a lemmatizer, but not the other way around**, if you goal is to reduce dimensionality and improve recall."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TextNormalizer also inherits from TransformerMixin, so it has a **fit_transform method**.\n",
    "\n",
    "Let's test our custom normalizer!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['review_normalized'] = normalizer.fit_transform(df['review'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>review</th>\n",
       "      <th>review_normalized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>neg</td>\n",
       "      <td>how do films like mouse hunt get into theatres...</td>\n",
       "      <td>[film, like, mouse, hunt, get, theatre, law, s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>neg</td>\n",
       "      <td>some talented actresses are blessed with a dem...</td>\n",
       "      <td>[talented, actress, bless, demonstrated, wide,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>pos</td>\n",
       "      <td>this has been an extraordinary year for austra...</td>\n",
       "      <td>[extraordinary, year, australian, film, shine,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>pos</td>\n",
       "      <td>according to hollywood movies made in last few...</td>\n",
       "      <td>[accord, hollywood, movie, make, last, decade,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>neg</td>\n",
       "      <td>my first press screening of 1998 and already i...</td>\n",
       "      <td>[first, press, screening, 1998, already, get, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                             review  \\\n",
       "0   neg  how do films like mouse hunt get into theatres...   \n",
       "1   neg  some talented actresses are blessed with a dem...   \n",
       "2   pos  this has been an extraordinary year for austra...   \n",
       "3   pos  according to hollywood movies made in last few...   \n",
       "4   neg  my first press screening of 1998 and already i...   \n",
       "\n",
       "                                   review_normalized  \n",
       "0  [film, like, mouse, hunt, get, theatre, law, s...  \n",
       "1  [talented, actress, bless, demonstrated, wide,...  \n",
       "2  [extraordinary, year, australian, film, shine,...  \n",
       "3  [accord, hollywood, movie, make, last, decade,...  \n",
       "4  [first, press, screening, 1998, already, get, ...  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything seems to be working as expected. Our next step it to create a **custom Gensim vectorization transformer**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Perform necessary imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "from gensim.corpora import Dictionary\n",
    "from gensim.models import TfidfModel\n",
    "from gensim.matutils import sparse2full"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Create a custom Gensim vectorization transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GensimVectorizer(BaseEstimator, TransformerMixin):\n",
    "    \n",
    "    def __init__(self, dict_path=None, tfidf_path=None):\n",
    "        '''\n",
    "        Allows to set a path for both the dictionary and the tfidf model\n",
    "        to be used used by the Gensim vectorizer.\n",
    "        '''\n",
    "        self.dict_path = dict_path\n",
    "        self.tfidf_path = tfidf_path\n",
    "        self.id2word = None\n",
    "        self.tfidf = None\n",
    "        self.load()\n",
    "        \n",
    "    def load(self):\n",
    "        # loads existing dictionary and tfidf model\n",
    "        if not self.dict_path == None:\n",
    "            self.id2word = Dictionary.load_from_text(self.dict_path)\n",
    "            \n",
    "        if not self.tfidf_path == None:\n",
    "            self.tfidf = TfidfModel.load(self.tfidf_path)\n",
    "    \n",
    "    def save(self):\n",
    "        # saves dictionary as a tab-delimited text file\n",
    "        self.id2word.save_as_text('./reviews_dictionary.txt')\n",
    "        # saves tfidf model as a pickled sparse matrix\n",
    "        self.tfidf.save('./reviews_tfidf.pkl')\n",
    "    \n",
    "    def fit(self, reviews, labels=None):\n",
    "        # creates dictionary and tfidf model\n",
    "        if self.dict_path == None or self.tfidf_path == None:\n",
    "            self.id2word = Dictionary(reviews)\n",
    "            self.tfidf = TfidfModel(dictionary=self.id2word, normalize=True)\n",
    "            self.save()\n",
    "            \n",
    "        return self\n",
    "    \n",
    "    def transform(self, reviews):\n",
    "        # returns dense numpy array of tfidf vectors\n",
    "        return [sparse2full(self.tfidf[self.id2word.doc2bow(review)], \n",
    "                            len(self.id2word)) for review in reviews]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Test the custom Gensim vectorization transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate object with default values\n",
    "\n",
    "gensim_vectorizer = GensimVectorizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check our object's parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dict_path': None, 'tfidf_path': None}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gensim_vectorizer.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a `dict_path` and a `tfidf_path` that define where our dictionary and TF-IDF model should be loaded from. Gensim allows us to write dictionaries and models to disk, enabling us to load them later everytime they're needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if everything is working properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "gensim_tfidf = gensim_vectorizer.fit_transform(df['review_normalized'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('abandon',\n",
       " 'abode',\n",
       " 'action',\n",
       " 'adam',\n",
       " 'alone',\n",
       " 'alternate',\n",
       " 'another',\n",
       " 'appalling',\n",
       " 'arrive',\n",
       " 'arse')"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_, tokens = zip(*sorted(zip(gensim_vectorizer.id2word.token2id.values(), gensim_vectorizer.id2word.token2id.keys())))\n",
    "tokens[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_gensim_tfidf = pd.DataFrame(gensim_tfidf, columns=tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abandon</th>\n",
       "      <th>abode</th>\n",
       "      <th>action</th>\n",
       "      <th>adam</th>\n",
       "      <th>alone</th>\n",
       "      <th>alternate</th>\n",
       "      <th>another</th>\n",
       "      <th>appalling</th>\n",
       "      <th>arrive</th>\n",
       "      <th>arse</th>\n",
       "      <th>...</th>\n",
       "      <th>castor</th>\n",
       "      <th>compardre</th>\n",
       "      <th>converted</th>\n",
       "      <th>dietrich</th>\n",
       "      <th>hassler</th>\n",
       "      <th>megabyte</th>\n",
       "      <th>potrayal</th>\n",
       "      <th>recline</th>\n",
       "      <th>staggeringly</th>\n",
       "      <th>swain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.052881</td>\n",
       "      <td>0.196983</td>\n",
       "      <td>0.019049</td>\n",
       "      <td>0.052065</td>\n",
       "      <td>0.078179</td>\n",
       "      <td>0.065869</td>\n",
       "      <td>0.015404</td>\n",
       "      <td>0.087037</td>\n",
       "      <td>0.045431</td>\n",
       "      <td>0.125088</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.041786</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.016400</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.014172</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.011461</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31836 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    abandon     abode    action      adam     alone  alternate   another  \\\n",
       "0  0.052881  0.196983  0.019049  0.052065  0.078179   0.065869  0.015404   \n",
       "1  0.041786  0.000000  0.000000  0.000000  0.000000   0.000000  0.000000   \n",
       "2  0.000000  0.000000  0.000000  0.000000  0.000000   0.000000  0.000000   \n",
       "3  0.000000  0.000000  0.000000  0.000000  0.016400   0.000000  0.000000   \n",
       "4  0.000000  0.000000  0.014172  0.000000  0.000000   0.000000  0.011461   \n",
       "\n",
       "   appalling    arrive      arse  ...  castor  compardre  converted  dietrich  \\\n",
       "0   0.087037  0.045431  0.125088  ...     0.0        0.0        0.0       0.0   \n",
       "1   0.000000  0.000000  0.000000  ...     0.0        0.0        0.0       0.0   \n",
       "2   0.000000  0.000000  0.000000  ...     0.0        0.0        0.0       0.0   \n",
       "3   0.000000  0.000000  0.000000  ...     0.0        0.0        0.0       0.0   \n",
       "4   0.000000  0.000000  0.000000  ...     0.0        0.0        0.0       0.0   \n",
       "\n",
       "   hassler  megabyte  potrayal  recline  staggeringly  swain  \n",
       "0      0.0       0.0       0.0      0.0           0.0    0.0  \n",
       "1      0.0       0.0       0.0      0.0           0.0    0.0  \n",
       "2      0.0       0.0       0.0      0.0           0.0    0.0  \n",
       "3      0.0       0.0       0.0      0.0           0.0    0.0  \n",
       "4      0.0       0.0       0.0      0.0           0.0    0.0  \n",
       "\n",
       "[5 rows x 31836 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_gensim_tfidf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1938"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_gensim_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far so good! We have our **1938 movie reviews** and a new **vocabulary size of 31836 tokens**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to choose a model and **build our new pipeline**. That's our step 5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Build pipeline to normalize text, vectorize it and train/fit the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('normalizer',\n",
       "                 TextNormalizer(language='english',\n",
       "                                lemmatizer=<WordNetLemmatizer>, stemmer=None,\n",
       "                                vectorizer='gensim')),\n",
       "                ('vectorizer',\n",
       "                 GensimVectorizer(dict_path=None, tfidf_path=None)),\n",
       "                ('clf_model',\n",
       "                 LinearSVC(C=1.0, class_weight=None, dual=True,\n",
       "                           fit_intercept=True, intercept_scaling=1,\n",
       "                           loss='squared_hinge', max_iter=1000,\n",
       "                           multi_class='ovr', penalty='l2', random_state=None,\n",
       "                           tol=0.0001, verbose=0))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_clf = Pipeline([\n",
    "    ('normalizer', TextNormalizer()),\n",
    "    ('vectorizer', GensimVectorizer()),\n",
    "    ('clf_model', LinearSVC())\n",
    "])\n",
    "\n",
    "text_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Make predictions with the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = text_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Evaluate the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[226  56]\n",
      " [ 43 257]]\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix\n",
    "\n",
    "print(confusion_matrix(y_test, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.84      0.80      0.82       282\n",
      "         pos       0.82      0.86      0.84       300\n",
      "\n",
      "    accuracy                           0.83       582\n",
      "   macro avg       0.83      0.83      0.83       582\n",
      "weighted avg       0.83      0.83      0.83       582\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# classification report\n",
    "\n",
    "print(classification_report(y_test, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8298969072164949\n"
     ]
    }
   ],
   "source": [
    "# accuracy score\n",
    "\n",
    "print(accuracy_score(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our particular case, using the same algorithm as before (LinearSVC), we get a slightly lower accuracy.\n",
    "\n",
    "Even though we've tried to minimize the loss of information by choosing lemmatization over stemming, it seems that some information was lost.\n",
    "\n",
    "Let's try to improve our result by **tuning some of our hyperparameters with GridSearchCV**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Perform necessary imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "import tabulate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Check parameters of our pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'memory': None,\n",
       " 'steps': [('normalizer',\n",
       "   TextNormalizer(language='english', lemmatizer=<WordNetLemmatizer>, stemmer=None,\n",
       "                  vectorizer='gensim')),\n",
       "  ('vectorizer', GensimVectorizer(dict_path=None, tfidf_path=None)),\n",
       "  ('clf_model',\n",
       "   LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
       "             intercept_scaling=1, loss='squared_hinge', max_iter=1000,\n",
       "             multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,\n",
       "             verbose=0))],\n",
       " 'verbose': False,\n",
       " 'normalizer': TextNormalizer(language='english', lemmatizer=<WordNetLemmatizer>, stemmer=None,\n",
       "                vectorizer='gensim'),\n",
       " 'vectorizer': GensimVectorizer(dict_path=None, tfidf_path=None),\n",
       " 'clf_model': LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
       "           intercept_scaling=1, loss='squared_hinge', max_iter=1000,\n",
       "           multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,\n",
       "           verbose=0),\n",
       " 'normalizer__language': 'english',\n",
       " 'normalizer__lemmatizer': <WordNetLemmatizer>,\n",
       " 'normalizer__stemmer': None,\n",
       " 'normalizer__vectorizer': 'gensim',\n",
       " 'vectorizer__dict_path': None,\n",
       " 'vectorizer__tfidf_path': None,\n",
       " 'clf_model__C': 1.0,\n",
       " 'clf_model__class_weight': None,\n",
       " 'clf_model__dual': True,\n",
       " 'clf_model__fit_intercept': True,\n",
       " 'clf_model__intercept_scaling': 1,\n",
       " 'clf_model__loss': 'squared_hinge',\n",
       " 'clf_model__max_iter': 1000,\n",
       " 'clf_model__multi_class': 'ovr',\n",
       " 'clf_model__penalty': 'l2',\n",
       " 'clf_model__random_state': None,\n",
       " 'clf_model__tol': 0.0001,\n",
       " 'clf_model__verbose': 0}"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#all parameters\n",
    "\n",
    "text_clf.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since our loss of information should be related with our text normalization step, let's focus on this particular aspect of our pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'language': 'english',\n",
       " 'lemmatizer': <WordNetLemmatizer>,\n",
       " 'stemmer': None,\n",
       " 'vectorizer': 'gensim'}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_clf.named_steps['normalizer'].get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how **choosing lemmatization or/and stemming** affects our result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Tuning parameters with GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 21min 26s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise-deprecating',\n",
       "             estimator=Pipeline(memory=None,\n",
       "                                steps=[('normalizer',\n",
       "                                        TextNormalizer(language='english',\n",
       "                                                       lemmatizer=<WordNetLemmatizer>,\n",
       "                                                       stemmer=None,\n",
       "                                                       vectorizer='gensim')),\n",
       "                                       ('vectorizer',\n",
       "                                        GensimVectorizer(dict_path=None,\n",
       "                                                         tfidf_path=None)),\n",
       "                                       ('clf_model',\n",
       "                                        LinearSVC(C=1.0, class_weight=None,\n",
       "                                                  dual=True, fit_intercept=True,\n",
       "                                                  intercep...\n",
       "                                                  loss='squared_hinge',\n",
       "                                                  max_iter=1000,\n",
       "                                                  multi_class='ovr',\n",
       "                                                  penalty='l2',\n",
       "                                                  random_state=None, tol=0.0001,\n",
       "                                                  verbose=0))],\n",
       "                                verbose=False),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid=[{'normalizer__lemmatizer': [<WordNetLemmatizer>, None],\n",
       "                          'normalizer__stemmer': [None, 'porter', 'snowball']}],\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "param_grid = [{\n",
    "    'normalizer__lemmatizer': [WordNetLemmatizer(), None], \n",
    "    'normalizer__stemmer': [None, 'porter', 'snowball']\n",
    "}]\n",
    "\n",
    "grid_search = GridSearchCV(text_clf, param_grid, cv=3, scoring = 'accuracy')\n",
    "\n",
    "grid_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'normalizer__lemmatizer': None, 'normalizer__stemmer': None}"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we were expecting, the **best result is obtained without lemmatizing or stemming** our reviews.\n",
    "\n",
    "Let's check the results in more detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lemmatizer         stemmer            accuracy\n",
      "-----------------  ---------------  ----------\n",
      "None               None               0.820059\n",
      "WordNetLemmatizer  SnowballStemmer    0.814159\n",
      "None               SnowballStemmer    0.811947\n",
      "WordNetLemmatizer  None               0.811209\n",
      "WordNetLemmatizer  PorterStemmer      0.808997\n",
      "None               PorterStemmer      0.807522\n"
     ]
    }
   ],
   "source": [
    "columns = ['lemmatizer', 'stemmer', 'accuracy']\n",
    "table = []\n",
    "\n",
    "cvres = grid_search.cv_results_\n",
    "\n",
    "for mean_score, params in sorted(zip(cvres['mean_test_score'], cvres['params']), reverse=True):\n",
    "    \n",
    "    if params['normalizer__lemmatizer'] != None:\n",
    "        lem = 'WordNetLemmatizer'\n",
    "    else:\n",
    "        lem = 'None'\n",
    "    \n",
    "    if params['normalizer__stemmer'] == 'porter':\n",
    "        stem = 'PorterStemmer'\n",
    "    elif params['normalizer__stemmer'] == 'snowball':\n",
    "        stem = 'SnowballStemmer'\n",
    "    else:\n",
    "        stem = 'None'\n",
    "    \n",
    "    row=[lem, stem, mean_score]\n",
    "    table.append(row)\n",
    "\n",
    "print(tabulate.tabulate(table, headers=columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our results confirm what we have said before: **no lemmatization and no stemming give us the best results** because we **reduce the loss of information** associated with these techniques.\n",
    "\n",
    "Interestingly enough, the **second best result** is obtained by **combining lemmatization with stemming** (SnowballStemmer).\n",
    "\n",
    "The **worst** result is obtained by **only stemming** the reviews with PorterStemmer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What this means is that, despite they can be useful techniques in other scenarios, **for our particular case we should neither lemmatize nor stem our reviews**.\n",
    "\n",
    "In other situations, like large-scale information retrieval applications where we want to maximize recall, for example, some combination of both might be useful.\n",
    "\n",
    "Based on that, let's update our pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Recreate pipeline (no lemmatization and no stemming)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('normalizer',\n",
       "                 TextNormalizer(language='english', lemmatizer=None,\n",
       "                                stemmer=None, vectorizer='gensim')),\n",
       "                ('vectorizer',\n",
       "                 GensimVectorizer(dict_path=None, tfidf_path=None)),\n",
       "                ('clf_model',\n",
       "                 LinearSVC(C=1.0, class_weight=None, dual=True,\n",
       "                           fit_intercept=True, intercept_scaling=1,\n",
       "                           loss='squared_hinge', max_iter=1000,\n",
       "                           multi_class='ovr', penalty='l2', random_state=None,\n",
       "                           tol=0.0001, verbose=0))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_clf = Pipeline([\n",
    "    ('normalizer', TextNormalizer(lemmatizer=None)),\n",
    "    ('vectorizer', GensimVectorizer()),\n",
    "    ('clf_model', LinearSVC())\n",
    "])\n",
    "\n",
    "text_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Make predictions with the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = text_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Evaluate the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[228  54]\n",
      " [ 42 258]]\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix\n",
    "\n",
    "print(confusion_matrix(y_test, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.84      0.81      0.83       282\n",
      "         pos       0.83      0.86      0.84       300\n",
      "\n",
      "    accuracy                           0.84       582\n",
      "   macro avg       0.84      0.83      0.83       582\n",
      "weighted avg       0.84      0.84      0.83       582\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# classification report\n",
    "\n",
    "print(classification_report(y_test, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8350515463917526\n"
     ]
    }
   ],
   "source": [
    "# accuracy score\n",
    "\n",
    "print(accuracy_score(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now try different classifiers to see if we can finally improve our result! This is our step 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Perform necessary imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "\n",
    "from collections import defaultdict\n",
    "from sklearn.metrics import f1_score, precision_score, recall_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Define function to create pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_pipeline(classifier, reduction=False):\n",
    "    \n",
    "    steps=[\n",
    "        ('normalizer', TextNormalizer(lemmatizer=None)),\n",
    "        ('vectorizer', GensimVectorizer())\n",
    "    ]\n",
    "    \n",
    "    if reduction:\n",
    "        steps.append(('reduction', TruncatedSVD(n_components=1000))\n",
    "        )\n",
    "    \n",
    "    steps.append(('clf_model', classifier))\n",
    "    return Pipeline(steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to **create a new pipeline that allows us to test different classifiers**.\n",
    "\n",
    "We've also introduced an optional 'reduction' step to perform dimensionality reduction using truncated singular value decomposition (SVD). In this context, since it will be applied to a tf-idf matrix, it is known as **latent semantic analysis** (LSA)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Create a list with models to test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = []\n",
    "for form in (LogisticRegression, SGDClassifier):\n",
    "    models.append(create_pipeline(form(), True))\n",
    "    models.append(create_pipeline(form(), False))\n",
    "\n",
    "models.append(create_pipeline(MultinomialNB(), False))\n",
    "models.append(create_pipeline(GaussianNB(), True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Test models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model                                precision    recall    accuracy        f1\n",
      "---------------------------------  -----------  --------  ----------  --------\n",
      "SGDClassifier                         0.843854  0.846667    0.840206  0.845258\n",
      "LogisticRegression                    0.819355  0.846667    0.824742  0.832787\n",
      "LogisticRegression (TruncatedSVD)     0.819672  0.833333    0.819588  0.826446\n",
      "SGDClassifier (TruncatedSVD)          0.872587  0.753333    0.816151  0.808587\n",
      "MultinomialNB                         0.857678  0.763333    0.812715  0.80776\n",
      "GaussianNB (TruncatedSVD)             0.516464  0.993333    0.517182  0.67959\n",
      "Wall time: 9min 54s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "columns = ['model', 'precision', 'recall', 'accuracy', 'f1']\n",
    "table = []\n",
    "\n",
    "for model in models:\n",
    "    name = model.named_steps['clf_model'].__class__.__name__\n",
    "    if 'reduction' in model.named_steps:\n",
    "            name += \" (TruncatedSVD)\"\n",
    "\n",
    "    scores = {}\n",
    "    \n",
    "    model.fit(X_train, y_train)\n",
    "    y_pred = model.predict(X_test)\n",
    "    \n",
    "    scores['precision'] = precision_score(y_test, y_pred, pos_label='pos', average='binary')\n",
    "    scores['recall'] = recall_score(y_test, y_pred, pos_label='pos', average='binary')\n",
    "    scores['accuracy'] = accuracy_score(y_test, y_pred)\n",
    "    scores['f1'] = f1_score(y_test, y_pred, pos_label='pos', average='binary')\n",
    "\n",
    "    row = [name]\n",
    "    for column in columns[1:]:\n",
    "        row.append(scores[column])\n",
    "    \n",
    "    table.append(row)\n",
    "table.sort(key=lambda row: row[-1], reverse=True)\n",
    "print(tabulate.tabulate(table, headers=columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our **SGDClassifier** model we obtain the **best f1 score** and the **best accuracy**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though we were not able to improve our initial accuracy, we've shown how important **custom tranformers** can be for a flexible approach to this kind of problems and how easy it is to **integrate them in an NLP pipeline**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}