Movies Recommender.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Movie Recommender System"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This notebook mainly consists of a movies dataset which consists of various parameters including the movie's genre, popularity etc. of about 5000 movies. And so considering the usecase of dataset here, we are going to use NLP and its techniques to make a movie recommender system for us.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import ast\n",
    "\n",
    "import nltk\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = pd.read_csv(r'datasets/tmdb_5000_movies.csv')\n",
    "credits = pd.read_csv(r'datasets/tmdb_5000_credits.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>keywords</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>237000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.avatarmovie.com/</td>\n",
       "      <td>19995</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>150.437577</td>\n",
       "      <td>[{\"name\": \"Ingenious Film Partners\", \"id\": 289...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2009-12-10</td>\n",
       "      <td>2787965087</td>\n",
       "      <td>162.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Enter the World of Pandora.</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>11800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>300000000</td>\n",
       "      <td>[{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...</td>\n",
       "      <td>http://disney.go.com/disneypictures/pirates/</td>\n",
       "      <td>285</td>\n",
       "      <td>[{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...</td>\n",
       "      <td>en</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>139.082615</td>\n",
       "      <td>[{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2007-05-19</td>\n",
       "      <td>961000000</td>\n",
       "      <td>169.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>At the end of the world, the adventure begins.</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>4500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>245000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.sonypictures.com/movies/spectre/</td>\n",
       "      <td>206647</td>\n",
       "      <td>[{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...</td>\n",
       "      <td>en</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>107.376788</td>\n",
       "      <td>[{\"name\": \"Columbia Pictures\", \"id\": 5}, {\"nam...</td>\n",
       "      <td>[{\"iso_3166_1\": \"GB\", \"name\": \"United Kingdom\"...</td>\n",
       "      <td>2015-10-26</td>\n",
       "      <td>880674609</td>\n",
       "      <td>148.0</td>\n",
       "      <td>[{\"iso_639_1\": \"fr\", \"name\": \"Fran\\u00e7ais\"},...</td>\n",
       "      <td>Released</td>\n",
       "      <td>A Plan No One Escapes</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>6.3</td>\n",
       "      <td>4466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>250000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...</td>\n",
       "      <td>http://www.thedarkknightrises.com/</td>\n",
       "      <td>49026</td>\n",
       "      <td>[{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...</td>\n",
       "      <td>en</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>112.312950</td>\n",
       "      <td>[{\"name\": \"Legendary Pictures\", \"id\": 923}, {\"...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2012-07-16</td>\n",
       "      <td>1084939099</td>\n",
       "      <td>165.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>The Legend Ends</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>7.6</td>\n",
       "      <td>9106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>260000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://movies.disney.com/john-carter</td>\n",
       "      <td>49529</td>\n",
       "      <td>[{\"id\": 818, \"name\": \"based on novel\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>43.926995</td>\n",
       "      <td>[{\"name\": \"Walt Disney Pictures\", \"id\": 2}]</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2012-03-07</td>\n",
       "      <td>284139100</td>\n",
       "      <td>132.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Lost in our world, found in another.</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>6.1</td>\n",
       "      <td>2124</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      budget                                             genres  \\\n",
       "0  237000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "1  300000000  [{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...   \n",
       "2  245000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "3  250000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...   \n",
       "4  260000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "\n",
       "                                       homepage      id  \\\n",
       "0                   http://www.avatarmovie.com/   19995   \n",
       "1  http://disney.go.com/disneypictures/pirates/     285   \n",
       "2   http://www.sonypictures.com/movies/spectre/  206647   \n",
       "3            http://www.thedarkknightrises.com/   49026   \n",
       "4          http://movies.disney.com/john-carter   49529   \n",
       "\n",
       "                                            keywords original_language  \\\n",
       "0  [{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...                en   \n",
       "1  [{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...                en   \n",
       "2  [{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...                en   \n",
       "3  [{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...                en   \n",
       "4  [{\"id\": 818, \"name\": \"based on novel\"}, {\"id\":...                en   \n",
       "\n",
       "                             original_title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "2                                   Spectre   \n",
       "3                     The Dark Knight Rises   \n",
       "4                               John Carter   \n",
       "\n",
       "                                            overview  popularity  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...  150.437577   \n",
       "1  Captain Barbossa, long believed to be dead, ha...  139.082615   \n",
       "2  A cryptic message from Bond’s past sends him o...  107.376788   \n",
       "3  Following the death of District Attorney Harve...  112.312950   \n",
       "4  John Carter is a war-weary, former military ca...   43.926995   \n",
       "\n",
       "                                production_companies  \\\n",
       "0  [{\"name\": \"Ingenious Film Partners\", \"id\": 289...   \n",
       "1  [{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...   \n",
       "2  [{\"name\": \"Columbia Pictures\", \"id\": 5}, {\"nam...   \n",
       "3  [{\"name\": \"Legendary Pictures\", \"id\": 923}, {\"...   \n",
       "4        [{\"name\": \"Walt Disney Pictures\", \"id\": 2}]   \n",
       "\n",
       "                                production_countries release_date     revenue  \\\n",
       "0  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...   2009-12-10  2787965087   \n",
       "1  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...   2007-05-19   961000000   \n",
       "2  [{\"iso_3166_1\": \"GB\", \"name\": \"United Kingdom\"...   2015-10-26   880674609   \n",
       "3  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...   2012-07-16  1084939099   \n",
       "4  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...   2012-03-07   284139100   \n",
       "\n",
       "   runtime                                   spoken_languages    status  \\\n",
       "0    162.0  [{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...  Released   \n",
       "1    169.0           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "2    148.0  [{\"iso_639_1\": \"fr\", \"name\": \"Fran\\u00e7ais\"},...  Released   \n",
       "3    165.0           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "4    132.0           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "\n",
       "                                          tagline  \\\n",
       "0                     Enter the World of Pandora.   \n",
       "1  At the end of the world, the adventure begins.   \n",
       "2                           A Plan No One Escapes   \n",
       "3                                 The Legend Ends   \n",
       "4            Lost in our world, found in another.   \n",
       "\n",
       "                                      title  vote_average  vote_count  \n",
       "0                                    Avatar           7.2       11800  \n",
       "1  Pirates of the Caribbean: At World's End           6.9        4500  \n",
       "2                                   Spectre           6.3        4466  \n",
       "3                     The Dark Knight Rises           7.6        9106  \n",
       "4                               John Carter           6.1        2124  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[{\"cast_id\": 5, \"character\": \"John Carter\", \"c...</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "2  [{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...   \n",
       "3  [{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...   \n",
       "4  [{\"cast_id\": 5, \"character\": \"John Carter\", \"c...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  \n",
       "4  [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credits.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merging both the datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = movies.merge(credits, on='title')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>keywords</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>...</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>237000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.avatarmovie.com/</td>\n",
       "      <td>19995</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>150.437577</td>\n",
       "      <td>[{\"name\": \"Ingenious Film Partners\", \"id\": 289...</td>\n",
       "      <td>...</td>\n",
       "      <td>162.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Enter the World of Pandora.</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>11800</td>\n",
       "      <td>19995</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>300000000</td>\n",
       "      <td>[{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...</td>\n",
       "      <td>http://disney.go.com/disneypictures/pirates/</td>\n",
       "      <td>285</td>\n",
       "      <td>[{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...</td>\n",
       "      <td>en</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>139.082615</td>\n",
       "      <td>[{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...</td>\n",
       "      <td>...</td>\n",
       "      <td>169.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>At the end of the world, the adventure begins.</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>4500</td>\n",
       "      <td>285</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>245000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.sonypictures.com/movies/spectre/</td>\n",
       "      <td>206647</td>\n",
       "      <td>[{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...</td>\n",
       "      <td>en</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>107.376788</td>\n",
       "      <td>[{\"name\": \"Columbia Pictures\", \"id\": 5}, {\"nam...</td>\n",
       "      <td>...</td>\n",
       "      <td>148.0</td>\n",
       "      <td>[{\"iso_639_1\": \"fr\", \"name\": \"Fran\\u00e7ais\"},...</td>\n",
       "      <td>Released</td>\n",
       "      <td>A Plan No One Escapes</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>6.3</td>\n",
       "      <td>4466</td>\n",
       "      <td>206647</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>250000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...</td>\n",
       "      <td>http://www.thedarkknightrises.com/</td>\n",
       "      <td>49026</td>\n",
       "      <td>[{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...</td>\n",
       "      <td>en</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>112.312950</td>\n",
       "      <td>[{\"name\": \"Legendary Pictures\", \"id\": 923}, {\"...</td>\n",
       "      <td>...</td>\n",
       "      <td>165.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>The Legend Ends</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>7.6</td>\n",
       "      <td>9106</td>\n",
       "      <td>49026</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>260000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://movies.disney.com/john-carter</td>\n",
       "      <td>49529</td>\n",
       "      <td>[{\"id\": 818, \"name\": \"based on novel\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>43.926995</td>\n",
       "      <td>[{\"name\": \"Walt Disney Pictures\", \"id\": 2}]</td>\n",
       "      <td>...</td>\n",
       "      <td>132.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Lost in our world, found in another.</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>6.1</td>\n",
       "      <td>2124</td>\n",
       "      <td>49529</td>\n",
       "      <td>[{\"cast_id\": 5, \"character\": \"John Carter\", \"c...</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      budget                                             genres  \\\n",
       "0  237000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "1  300000000  [{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...   \n",
       "2  245000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "3  250000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...   \n",
       "4  260000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "\n",
       "                                       homepage      id  \\\n",
       "0                   http://www.avatarmovie.com/   19995   \n",
       "1  http://disney.go.com/disneypictures/pirates/     285   \n",
       "2   http://www.sonypictures.com/movies/spectre/  206647   \n",
       "3            http://www.thedarkknightrises.com/   49026   \n",
       "4          http://movies.disney.com/john-carter   49529   \n",
       "\n",
       "                                            keywords original_language  \\\n",
       "0  [{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...                en   \n",
       "1  [{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...                en   \n",
       "2  [{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...                en   \n",
       "3  [{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...                en   \n",
       "4  [{\"id\": 818, \"name\": \"based on novel\"}, {\"id\":...                en   \n",
       "\n",
       "                             original_title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "2                                   Spectre   \n",
       "3                     The Dark Knight Rises   \n",
       "4                               John Carter   \n",
       "\n",
       "                                            overview  popularity  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...  150.437577   \n",
       "1  Captain Barbossa, long believed to be dead, ha...  139.082615   \n",
       "2  A cryptic message from Bond’s past sends him o...  107.376788   \n",
       "3  Following the death of District Attorney Harve...  112.312950   \n",
       "4  John Carter is a war-weary, former military ca...   43.926995   \n",
       "\n",
       "                                production_companies  ... runtime  \\\n",
       "0  [{\"name\": \"Ingenious Film Partners\", \"id\": 289...  ...   162.0   \n",
       "1  [{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...  ...   169.0   \n",
       "2  [{\"name\": \"Columbia Pictures\", \"id\": 5}, {\"nam...  ...   148.0   \n",
       "3  [{\"name\": \"Legendary Pictures\", \"id\": 923}, {\"...  ...   165.0   \n",
       "4        [{\"name\": \"Walt Disney Pictures\", \"id\": 2}]  ...   132.0   \n",
       "\n",
       "                                    spoken_languages    status  \\\n",
       "0  [{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...  Released   \n",
       "1           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "2  [{\"iso_639_1\": \"fr\", \"name\": \"Fran\\u00e7ais\"},...  Released   \n",
       "3           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "4           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "\n",
       "                                          tagline  \\\n",
       "0                     Enter the World of Pandora.   \n",
       "1  At the end of the world, the adventure begins.   \n",
       "2                           A Plan No One Escapes   \n",
       "3                                 The Legend Ends   \n",
       "4            Lost in our world, found in another.   \n",
       "\n",
       "                                      title vote_average vote_count movie_id  \\\n",
       "0                                    Avatar          7.2      11800    19995   \n",
       "1  Pirates of the Caribbean: At World's End          6.9       4500      285   \n",
       "2                                   Spectre          6.3       4466   206647   \n",
       "3                     The Dark Knight Rises          7.6       9106    49026   \n",
       "4                               John Carter          6.1       2124    49529   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "2  [{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...   \n",
       "3  [{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...   \n",
       "4  [{\"cast_id\": 5, \"character\": \"John Carter\", \"c...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  \n",
       "4  [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 4809 entries, 0 to 4808\n",
      "Data columns (total 23 columns):\n",
      " #   Column                Non-Null Count  Dtype  \n",
      "---  ------                --------------  -----  \n",
      " 0   budget                4809 non-null   int64  \n",
      " 1   genres                4809 non-null   object \n",
      " 2   homepage              1713 non-null   object \n",
      " 3   id                    4809 non-null   int64  \n",
      " 4   keywords              4809 non-null   object \n",
      " 5   original_language     4809 non-null   object \n",
      " 6   original_title        4809 non-null   object \n",
      " 7   overview              4806 non-null   object \n",
      " 8   popularity            4809 non-null   float64\n",
      " 9   production_companies  4809 non-null   object \n",
      " 10  production_countries  4809 non-null   object \n",
      " 11  release_date          4808 non-null   object \n",
      " 12  revenue               4809 non-null   int64  \n",
      " 13  runtime               4807 non-null   float64\n",
      " 14  spoken_languages      4809 non-null   object \n",
      " 15  status                4809 non-null   object \n",
      " 16  tagline               3965 non-null   object \n",
      " 17  title                 4809 non-null   object \n",
      " 18  vote_average          4809 non-null   float64\n",
      " 19  vote_count            4809 non-null   int64  \n",
      " 20  movie_id              4809 non-null   int64  \n",
      " 21  cast                  4809 non-null   object \n",
      " 22  crew                  4809 non-null   object \n",
      "dtypes: float64(3), int64(5), object(15)\n",
      "memory usage: 901.7+ KB\n"
     ]
    }
   ],
   "source": [
    "movies.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Keeping the useful columns for further analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies = movies[['movie_id','overview','title','genres','keywords','cast','crew']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                           overview   title  \\\n",
       "0     19995  In the 22nd century, a paraplegic Marine is di...  Avatar   \n",
       "\n",
       "                                              genres  \\\n",
       "0  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking NuLL Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "movie_id    0\n",
       "overview    3\n",
       "title       0\n",
       "genres      0\n",
       "keywords    0\n",
       "cast        0\n",
       "crew        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"name\": \"Fantasy\"}, {\"id\": 878, \"name\": \"Science Fiction\"}]'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.iloc[0]['genres']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Formatting the 'genre' column text into more readable format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert(obj):\n",
    "    L=[]\n",
    "    for i in ast.literal_eval(obj):\n",
    "        L.append(i['name'])\n",
    "    return L       \n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['genres'] = movies['genres'].apply(convert)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Formatting the 'keywords' column text into more readable format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['keywords'] = movies['keywords'].apply(convert)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, Science Fiction]</td>\n",
       "      <td>[culture clash, future, space war, space colon...</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drug abuse, exotic island, east india ...</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, based on novel, secret agent, sequel, mi...</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dc comics, crime fighter, terrorist, secret i...</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, Science Fiction]</td>\n",
       "      <td>[based on novel, mars, medallion, space travel...</td>\n",
       "      <td>[{\"cast_id\": 5, \"character\": \"John Carter\", \"c...</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                           overview  \\\n",
       "0     19995  In the 22nd century, a paraplegic Marine is di...   \n",
       "1       285  Captain Barbossa, long believed to be dead, ha...   \n",
       "2    206647  A cryptic message from Bond’s past sends him o...   \n",
       "3     49026  Following the death of District Attorney Harve...   \n",
       "4     49529  John Carter is a war-weary, former military ca...   \n",
       "\n",
       "                                      title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "2                                   Spectre   \n",
       "3                     The Dark Knight Rises   \n",
       "4                               John Carter   \n",
       "\n",
       "                                          genres  \\\n",
       "0  [Action, Adventure, Fantasy, Science Fiction]   \n",
       "1                   [Adventure, Fantasy, Action]   \n",
       "2                     [Action, Adventure, Crime]   \n",
       "3               [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, Science Fiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [culture clash, future, space war, space colon...   \n",
       "1  [ocean, drug abuse, exotic island, east india ...   \n",
       "2  [spy, based on novel, secret agent, sequel, mi...   \n",
       "3  [dc comics, crime fighter, terrorist, secret i...   \n",
       "4  [based on novel, mars, medallion, space travel...   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "2  [{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...   \n",
       "3  [{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...   \n",
       "4  [{\"cast_id\": 5, \"character\": \"John Carter\", \"c...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  \n",
       "4  [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"credit_id\": \"5602a8a7c3a3685532001c9a\", \"gender\": 2, \"id\": 65731, \"name\": \"Sam Worthington\", \"order\": 0}, {\"cast_id\": 3, \"character\": \"Neytiri\", \"credit_id\": \"52fe48009251416c750ac9cb\", \"gender\": 1, \"id\": 8691, \"name\": \"Zoe Saldana\", \"order\": 1}, {\"cast_id\": 25, \"character\": \"Dr. Grace Augustine\", \"credit_id\": \"52fe48009251416c750aca39\", \"gender\": 1, \"id\": 10205, \"name\": \"Sigourney Weaver\", \"order\": 2}, {\"cast_id\": 4, \"character\": \"Col. Quaritch\", \"credit_id\": \"52fe48009251416c750ac9cf\", \"gender\": 2, \"id\": 32747, \"name\": \"Stephen Lang\", \"order\": 3}, {\"cast_id\": 5, \"character\": \"Trudy Chacon\", \"credit_id\": \"52fe48009251416c750ac9d3\", \"gender\": 1, \"id\": 17647, \"name\": \"Michelle Rodriguez\", \"order\": 4}, {\"cast_id\": 8, \"character\": \"Selfridge\", \"credit_id\": \"52fe48009251416c750ac9e1\", \"gender\": 2, \"id\": 1771, \"name\": \"Giovanni Ribisi\", \"order\": 5}, {\"cast_id\": 7, \"character\": \"Norm Spellman\", \"credit_id\": \"52fe48009251416c750ac9dd\", \"gender\": 2, \"id\": 59231, \"name\": \"Joel David Moore\", \"order\": 6}, {\"cast_id\": 9, \"character\": \"Moat\", \"credit_id\": \"52fe48009251416c750ac9e5\", \"gender\": 1, \"id\": 30485, \"name\": \"CCH Pounder\", \"order\": 7}, {\"cast_id\": 11, \"character\": \"Eytukan\", \"credit_id\": \"52fe48009251416c750ac9ed\", \"gender\": 2, \"id\": 15853, \"name\": \"Wes Studi\", \"order\": 8}, {\"cast_id\": 10, \"character\": \"Tsu\\'Tey\", \"credit_id\": \"52fe48009251416c750ac9e9\", \"gender\": 2, \"id\": 10964, \"name\": \"Laz Alonso\", \"order\": 9}, {\"cast_id\": 12, \"character\": \"Dr. Max Patel\", \"credit_id\": \"52fe48009251416c750ac9f1\", \"gender\": 2, \"id\": 95697, \"name\": \"Dileep Rao\", \"order\": 10}, {\"cast_id\": 13, \"character\": \"Lyle Wainfleet\", \"credit_id\": \"52fe48009251416c750ac9f5\", \"gender\": 2, \"id\": 98215, \"name\": \"Matt Gerald\", \"order\": 11}, {\"cast_id\": 32, \"character\": \"Private Fike\", \"credit_id\": \"52fe48009251416c750aca5b\", \"gender\": 2, \"id\": 154153, \"name\": \"Sean Anthony Moran\", \"order\": 12}, {\"cast_id\": 33, \"character\": \"Cryo Vault Med Tech\", \"credit_id\": \"52fe48009251416c750aca5f\", \"gender\": 2, \"id\": 397312, \"name\": \"Jason Whyte\", \"order\": 13}, {\"cast_id\": 34, \"character\": \"Venture Star Crew Chief\", \"credit_id\": \"52fe48009251416c750aca63\", \"gender\": 2, \"id\": 42317, \"name\": \"Scott Lawrence\", \"order\": 14}, {\"cast_id\": 35, \"character\": \"Lock Up Trooper\", \"credit_id\": \"52fe48009251416c750aca67\", \"gender\": 2, \"id\": 986734, \"name\": \"Kelly Kilgour\", \"order\": 15}, {\"cast_id\": 36, \"character\": \"Shuttle Pilot\", \"credit_id\": \"52fe48009251416c750aca6b\", \"gender\": 0, \"id\": 1207227, \"name\": \"James Patrick Pitt\", \"order\": 16}, {\"cast_id\": 37, \"character\": \"Shuttle Co-Pilot\", \"credit_id\": \"52fe48009251416c750aca6f\", \"gender\": 0, \"id\": 1180936, \"name\": \"Sean Patrick Murphy\", \"order\": 17}, {\"cast_id\": 38, \"character\": \"Shuttle Crew Chief\", \"credit_id\": \"52fe48009251416c750aca73\", \"gender\": 2, \"id\": 1019578, \"name\": \"Peter Dillon\", \"order\": 18}, {\"cast_id\": 39, \"character\": \"Tractor Operator / Troupe\", \"credit_id\": \"52fe48009251416c750aca77\", \"gender\": 0, \"id\": 91443, \"name\": \"Kevin Dorman\", \"order\": 19}, {\"cast_id\": 40, \"character\": \"Dragon Gunship Pilot\", \"credit_id\": \"52fe48009251416c750aca7b\", \"gender\": 2, \"id\": 173391, \"name\": \"Kelson Henderson\", \"order\": 20}, {\"cast_id\": 41, \"character\": \"Dragon Gunship Gunner\", \"credit_id\": \"52fe48009251416c750aca7f\", \"gender\": 0, \"id\": 1207236, \"name\": \"David Van Horn\", \"order\": 21}, {\"cast_id\": 42, \"character\": \"Dragon Gunship Navigator\", \"credit_id\": \"52fe48009251416c750aca83\", \"gender\": 0, \"id\": 215913, \"name\": \"Jacob Tomuri\", \"order\": 22}, {\"cast_id\": 43, \"character\": \"Suit #1\", \"credit_id\": \"52fe48009251416c750aca87\", \"gender\": 0, \"id\": 143206, \"name\": \"Michael Blain-Rozgay\", \"order\": 23}, {\"cast_id\": 44, \"character\": \"Suit #2\", \"credit_id\": \"52fe48009251416c750aca8b\", \"gender\": 2, \"id\": 169676, \"name\": \"Jon Curry\", \"order\": 24}, {\"cast_id\": 46, \"character\": \"Ambient Room Tech\", \"credit_id\": \"52fe48009251416c750aca8f\", \"gender\": 0, \"id\": 1048610, \"name\": \"Luke Hawker\", \"order\": 25}, {\"cast_id\": 47, \"character\": \"Ambient Room Tech / Troupe\", \"credit_id\": \"52fe48009251416c750aca93\", \"gender\": 0, \"id\": 42288, \"name\": \"Woody Schultz\", \"order\": 26}, {\"cast_id\": 48, \"character\": \"Horse Clan Leader\", \"credit_id\": \"52fe48009251416c750aca97\", \"gender\": 2, \"id\": 68278, \"name\": \"Peter Mensah\", \"order\": 27}, {\"cast_id\": 49, \"character\": \"Link Room Tech\", \"credit_id\": \"52fe48009251416c750aca9b\", \"gender\": 0, \"id\": 1207247, \"name\": \"Sonia Yee\", \"order\": 28}, {\"cast_id\": 50, \"character\": \"Basketball Avatar / Troupe\", \"credit_id\": \"52fe48009251416c750aca9f\", \"gender\": 1, \"id\": 1207248, \"name\": \"Jahnel Curfman\", \"order\": 29}, {\"cast_id\": 51, \"character\": \"Basketball Avatar\", \"credit_id\": \"52fe48009251416c750acaa3\", \"gender\": 0, \"id\": 89714, \"name\": \"Ilram Choi\", \"order\": 30}, {\"cast_id\": 52, \"character\": \"Na\\'vi Child\", \"credit_id\": \"52fe48009251416c750acaa7\", \"gender\": 0, \"id\": 1207249, \"name\": \"Kyla Warren\", \"order\": 31}, {\"cast_id\": 53, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acaab\", \"gender\": 0, \"id\": 1207250, \"name\": \"Lisa Roumain\", \"order\": 32}, {\"cast_id\": 54, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acaaf\", \"gender\": 1, \"id\": 83105, \"name\": \"Debra Wilson\", \"order\": 33}, {\"cast_id\": 57, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acabb\", \"gender\": 0, \"id\": 1207253, \"name\": \"Chris Mala\", \"order\": 34}, {\"cast_id\": 55, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acab3\", \"gender\": 0, \"id\": 1207251, \"name\": \"Taylor Kibby\", \"order\": 35}, {\"cast_id\": 56, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acab7\", \"gender\": 0, \"id\": 1207252, \"name\": \"Jodie Landau\", \"order\": 36}, {\"cast_id\": 58, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acabf\", \"gender\": 0, \"id\": 1207254, \"name\": \"Julie Lamm\", \"order\": 37}, {\"cast_id\": 59, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acac3\", \"gender\": 0, \"id\": 1207257, \"name\": \"Cullen B. Madden\", \"order\": 38}, {\"cast_id\": 60, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acac7\", \"gender\": 0, \"id\": 1207259, \"name\": \"Joseph Brady Madden\", \"order\": 39}, {\"cast_id\": 61, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acacb\", \"gender\": 0, \"id\": 1207262, \"name\": \"Frankie Torres\", \"order\": 40}, {\"cast_id\": 62, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acacf\", \"gender\": 1, \"id\": 1158600, \"name\": \"Austin Wilson\", \"order\": 41}, {\"cast_id\": 63, \"character\": \"Troupe\", \"credit_id\": \"52fe48019251416c750acad3\", \"gender\": 1, \"id\": 983705, \"name\": \"Sara Wilson\", \"order\": 42}, {\"cast_id\": 64, \"character\": \"Troupe\", \"credit_id\": \"52fe48019251416c750acad7\", \"gender\": 0, \"id\": 1207263, \"name\": \"Tamica Washington-Miller\", \"order\": 43}, {\"cast_id\": 65, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acadb\", \"gender\": 1, \"id\": 1145098, \"name\": \"Lucy Briant\", \"order\": 44}, {\"cast_id\": 66, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acadf\", \"gender\": 2, \"id\": 33305, \"name\": \"Nathan Meister\", \"order\": 45}, {\"cast_id\": 67, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acae3\", \"gender\": 0, \"id\": 1207264, \"name\": \"Gerry Blair\", \"order\": 46}, {\"cast_id\": 68, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acae7\", \"gender\": 2, \"id\": 33311, \"name\": \"Matthew Chamberlain\", \"order\": 47}, {\"cast_id\": 69, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acaeb\", \"gender\": 0, \"id\": 1207265, \"name\": \"Paul Yates\", \"order\": 48}, {\"cast_id\": 70, \"character\": \"Op Center Duty Officer\", \"credit_id\": \"52fe48019251416c750acaef\", \"gender\": 0, \"id\": 1207266, \"name\": \"Wray Wilson\", \"order\": 49}, {\"cast_id\": 71, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acaf3\", \"gender\": 2, \"id\": 54492, \"name\": \"James Gaylyn\", \"order\": 50}, {\"cast_id\": 72, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acaf7\", \"gender\": 0, \"id\": 1207267, \"name\": \"Melvin Leno Clark III\", \"order\": 51}, {\"cast_id\": 73, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acafb\", \"gender\": 0, \"id\": 1207268, \"name\": \"Carvon Futrell\", \"order\": 52}, {\"cast_id\": 74, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acaff\", \"gender\": 0, \"id\": 1207269, \"name\": \"Brandon Jelkes\", \"order\": 53}, {\"cast_id\": 75, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb03\", \"gender\": 0, \"id\": 1207270, \"name\": \"Micah Moch\", \"order\": 54}, {\"cast_id\": 76, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb07\", \"gender\": 0, \"id\": 1207271, \"name\": \"Hanniyah Muhammad\", \"order\": 55}, {\"cast_id\": 77, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb0b\", \"gender\": 0, \"id\": 1207272, \"name\": \"Christopher Nolen\", \"order\": 56}, {\"cast_id\": 78, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb0f\", \"gender\": 0, \"id\": 1207273, \"name\": \"Christa Oliver\", \"order\": 57}, {\"cast_id\": 79, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb13\", \"gender\": 0, \"id\": 1207274, \"name\": \"April Marie Thomas\", \"order\": 58}, {\"cast_id\": 80, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb17\", \"gender\": 0, \"id\": 1207275, \"name\": \"Bravita A. Threatt\", \"order\": 59}, {\"cast_id\": 81, \"character\": \"Mining Chief (uncredited)\", \"credit_id\": \"52fe48019251416c750acb1b\", \"gender\": 0, \"id\": 1207276, \"name\": \"Colin Bleasdale\", \"order\": 60}, {\"cast_id\": 82, \"character\": \"Veteran Miner (uncredited)\", \"credit_id\": \"52fe48019251416c750acb1f\", \"gender\": 0, \"id\": 107969, \"name\": \"Mike Bodnar\", \"order\": 61}, {\"cast_id\": 83, \"character\": \"Richard (uncredited)\", \"credit_id\": \"52fe48019251416c750acb23\", \"gender\": 0, \"id\": 1207278, \"name\": \"Matt Clayton\", \"order\": 62}, {\"cast_id\": 84, \"character\": \"Nav\\'i (uncredited)\", \"credit_id\": \"52fe48019251416c750acb27\", \"gender\": 1, \"id\": 147898, \"name\": \"Nicole Dionne\", \"order\": 63}, {\"cast_id\": 85, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb2b\", \"gender\": 0, \"id\": 1207280, \"name\": \"Jamie Harrison\", \"order\": 64}, {\"cast_id\": 86, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb2f\", \"gender\": 0, \"id\": 1207281, \"name\": \"Allan Henry\", \"order\": 65}, {\"cast_id\": 87, \"character\": \"Ground Technician (uncredited)\", \"credit_id\": \"52fe48019251416c750acb33\", \"gender\": 2, \"id\": 1207282, \"name\": \"Anthony Ingruber\", \"order\": 66}, {\"cast_id\": 88, \"character\": \"Flight Crew Mechanic (uncredited)\", \"credit_id\": \"52fe48019251416c750acb37\", \"gender\": 0, \"id\": 1207283, \"name\": \"Ashley Jeffery\", \"order\": 67}, {\"cast_id\": 14, \"character\": \"Samson Pilot\", \"credit_id\": \"52fe48009251416c750ac9f9\", \"gender\": 0, \"id\": 98216, \"name\": \"Dean Knowsley\", \"order\": 68}, {\"cast_id\": 89, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb3b\", \"gender\": 0, \"id\": 1201399, \"name\": \"Joseph Mika-Hunt\", \"order\": 69}, {\"cast_id\": 90, \"character\": \"Banshee (uncredited)\", \"credit_id\": \"52fe48019251416c750acb3f\", \"gender\": 0, \"id\": 236696, \"name\": \"Terry Notary\", \"order\": 70}, {\"cast_id\": 91, \"character\": \"Soldier (uncredited)\", \"credit_id\": \"52fe48019251416c750acb43\", \"gender\": 0, \"id\": 1207287, \"name\": \"Kai Pantano\", \"order\": 71}, {\"cast_id\": 92, \"character\": \"Blast Technician (uncredited)\", \"credit_id\": \"52fe48019251416c750acb47\", \"gender\": 0, \"id\": 1207288, \"name\": \"Logan Pithyou\", \"order\": 72}, {\"cast_id\": 93, \"character\": \"Vindum Raah (uncredited)\", \"credit_id\": \"52fe48019251416c750acb4b\", \"gender\": 0, \"id\": 1207289, \"name\": \"Stuart Pollock\", \"order\": 73}, {\"cast_id\": 94, \"character\": \"Hero (uncredited)\", \"credit_id\": \"52fe48019251416c750acb4f\", \"gender\": 0, \"id\": 584868, \"name\": \"Raja\", \"order\": 74}, {\"cast_id\": 95, \"character\": \"Ops Centreworker (uncredited)\", \"credit_id\": \"52fe48019251416c750acb53\", \"gender\": 0, \"id\": 1207290, \"name\": \"Gareth Ruck\", \"order\": 75}, {\"cast_id\": 96, \"character\": \"Engineer (uncredited)\", \"credit_id\": \"52fe48019251416c750acb57\", \"gender\": 0, \"id\": 1062463, \"name\": \"Rhian Sheehan\", \"order\": 76}, {\"cast_id\": 97, \"character\": \"Col. Quaritch\\'s Mech Suit (uncredited)\", \"credit_id\": \"52fe48019251416c750acb5b\", \"gender\": 0, \"id\": 60656, \"name\": \"T. J. Storm\", \"order\": 77}, {\"cast_id\": 98, \"character\": \"Female Marine (uncredited)\", \"credit_id\": \"52fe48019251416c750acb5f\", \"gender\": 0, \"id\": 1207291, \"name\": \"Jodie Taylor\", \"order\": 78}, {\"cast_id\": 99, \"character\": \"Ikran Clan Leader (uncredited)\", \"credit_id\": \"52fe48019251416c750acb63\", \"gender\": 1, \"id\": 1186027, \"name\": \"Alicia Vela-Bailey\", \"order\": 79}, {\"cast_id\": 100, \"character\": \"Geologist (uncredited)\", \"credit_id\": \"52fe48019251416c750acb67\", \"gender\": 0, \"id\": 1207292, \"name\": \"Richard Whiteside\", \"order\": 80}, {\"cast_id\": 101, \"character\": \"Na\\'vi (uncredited)\", \"credit_id\": \"52fe48019251416c750acb6b\", \"gender\": 0, \"id\": 103259, \"name\": \"Nikie Zambo\", \"order\": 81}, {\"cast_id\": 102, \"character\": \"Ambient Room Tech / Troupe\", \"credit_id\": \"52fe48019251416c750acb6f\", \"gender\": 1, \"id\": 42286, \"name\": \"Julene Renee\", \"order\": 82}]'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies['cast'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- It clearly seems that cast column entries are too big, so they needs to be brought down into simple more readable format too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert3(obj):\n",
    "    L=[]\n",
    "    counter=0\n",
    "    for i in ast.literal_eval(obj):\n",
    "        if counter !=3:\n",
    "            L.append(i['name'])\n",
    "            counter +=1\n",
    "        else:\n",
    "            break\n",
    "    return L       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Formatting the 'cast' column text into more readable format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['cast'] = movies['cast'].apply(convert3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, Science Fiction]</td>\n",
       "      <td>[culture clash, future, space war, space colon...</td>\n",
       "      <td>[Sam Worthington, Zoe Saldana, Sigourney Weaver]</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drug abuse, exotic island, east india ...</td>\n",
       "      <td>[Johnny Depp, Orlando Bloom, Keira Knightley]</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, based on novel, secret agent, sequel, mi...</td>\n",
       "      <td>[Daniel Craig, Christoph Waltz, Léa Seydoux]</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dc comics, crime fighter, terrorist, secret i...</td>\n",
       "      <td>[Christian Bale, Michael Caine, Gary Oldman]</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, Science Fiction]</td>\n",
       "      <td>[based on novel, mars, medallion, space travel...</td>\n",
       "      <td>[Taylor Kitsch, Lynn Collins, Samantha Morton]</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                           overview  \\\n",
       "0     19995  In the 22nd century, a paraplegic Marine is di...   \n",
       "1       285  Captain Barbossa, long believed to be dead, ha...   \n",
       "2    206647  A cryptic message from Bond’s past sends him o...   \n",
       "3     49026  Following the death of District Attorney Harve...   \n",
       "4     49529  John Carter is a war-weary, former military ca...   \n",
       "\n",
       "                                      title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "2                                   Spectre   \n",
       "3                     The Dark Knight Rises   \n",
       "4                               John Carter   \n",
       "\n",
       "                                          genres  \\\n",
       "0  [Action, Adventure, Fantasy, Science Fiction]   \n",
       "1                   [Adventure, Fantasy, Action]   \n",
       "2                     [Action, Adventure, Crime]   \n",
       "3               [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, Science Fiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [culture clash, future, space war, space colon...   \n",
       "1  [ocean, drug abuse, exotic island, east india ...   \n",
       "2  [spy, based on novel, secret agent, sequel, mi...   \n",
       "3  [dc comics, crime fighter, terrorist, secret i...   \n",
       "4  [based on novel, mars, medallion, space travel...   \n",
       "\n",
       "                                               cast  \\\n",
       "0  [Sam Worthington, Zoe Saldana, Sigourney Weaver]   \n",
       "1     [Johnny Depp, Orlando Bloom, Keira Knightley]   \n",
       "2      [Daniel Craig, Christoph Waltz, Léa Seydoux]   \n",
       "3      [Christian Bale, Michael Caine, Gary Oldman]   \n",
       "4    [Taylor Kitsch, Lynn Collins, Samantha Morton]   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  \n",
       "4  [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Formatting the 'crew' column text into more readable format by just extracting the DIrector's name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fetch_director(obj):\n",
    "    L=[]\n",
    "    for i in ast.literal_eval(obj):\n",
    "        if i['job'] == 'Director':\n",
    "            L.append(i['name'])\n",
    "            break\n",
    "    return L     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['crew'] = movies['crew'].apply(fetch_director)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, Science Fiction]</td>\n",
       "      <td>[culture clash, future, space war, space colon...</td>\n",
       "      <td>[Sam Worthington, Zoe Saldana, Sigourney Weaver]</td>\n",
       "      <td>[James Cameron]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drug abuse, exotic island, east india ...</td>\n",
       "      <td>[Johnny Depp, Orlando Bloom, Keira Knightley]</td>\n",
       "      <td>[Gore Verbinski]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, based on novel, secret agent, sequel, mi...</td>\n",
       "      <td>[Daniel Craig, Christoph Waltz, Léa Seydoux]</td>\n",
       "      <td>[Sam Mendes]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dc comics, crime fighter, terrorist, secret i...</td>\n",
       "      <td>[Christian Bale, Michael Caine, Gary Oldman]</td>\n",
       "      <td>[Christopher Nolan]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, Science Fiction]</td>\n",
       "      <td>[based on novel, mars, medallion, space travel...</td>\n",
       "      <td>[Taylor Kitsch, Lynn Collins, Samantha Morton]</td>\n",
       "      <td>[Andrew Stanton]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                           overview  \\\n",
       "0     19995  In the 22nd century, a paraplegic Marine is di...   \n",
       "1       285  Captain Barbossa, long believed to be dead, ha...   \n",
       "2    206647  A cryptic message from Bond’s past sends him o...   \n",
       "3     49026  Following the death of District Attorney Harve...   \n",
       "4     49529  John Carter is a war-weary, former military ca...   \n",
       "\n",
       "                                      title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "2                                   Spectre   \n",
       "3                     The Dark Knight Rises   \n",
       "4                               John Carter   \n",
       "\n",
       "                                          genres  \\\n",
       "0  [Action, Adventure, Fantasy, Science Fiction]   \n",
       "1                   [Adventure, Fantasy, Action]   \n",
       "2                     [Action, Adventure, Crime]   \n",
       "3               [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, Science Fiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [culture clash, future, space war, space colon...   \n",
       "1  [ocean, drug abuse, exotic island, east india ...   \n",
       "2  [spy, based on novel, secret agent, sequel, mi...   \n",
       "3  [dc comics, crime fighter, terrorist, secret i...   \n",
       "4  [based on novel, mars, medallion, space travel...   \n",
       "\n",
       "                                               cast                 crew  \n",
       "0  [Sam Worthington, Zoe Saldana, Sigourney Weaver]      [James Cameron]  \n",
       "1     [Johnny Depp, Orlando Bloom, Keira Knightley]     [Gore Verbinski]  \n",
       "2      [Daniel Craig, Christoph Waltz, Léa Seydoux]         [Sam Mendes]  \n",
       "3      [Christian Bale, Michael Caine, Gary Oldman]  [Christopher Nolan]  \n",
       "4    [Taylor Kitsch, Lynn Collins, Samantha Morton]     [Andrew Stanton]  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Handling the other column's text data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['overview'] = movies['overview'].apply(lambda x: x.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['genres'] = movies['genres'].apply(lambda x:[i.replace(\" \",\"\") for i in x])\n",
    "movies['keywords'] = movies['keywords'].apply(lambda x:[i.replace(\" \",\"\") for i in x])\n",
    "movies['cast'] = movies['cast'].apply(lambda x:[i.replace(\" \",\"\") for i in x])\n",
    "movies['crew'] = movies['crew'].apply(lambda x:[i.replace(\" \",\"\") for i in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>[In, the, 22nd, century,, a, paraplegic, Marin...</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, ScienceFiction]</td>\n",
       "      <td>[cultureclash, future, spacewar, spacecolony, ...</td>\n",
       "      <td>[SamWorthington, ZoeSaldana, SigourneyWeaver]</td>\n",
       "      <td>[JamesCameron]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>[Captain, Barbossa,, long, believed, to, be, d...</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drugabuse, exoticisland, eastindiatrad...</td>\n",
       "      <td>[JohnnyDepp, OrlandoBloom, KeiraKnightley]</td>\n",
       "      <td>[GoreVerbinski]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>[A, cryptic, message, from, Bond’s, past, send...</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, basedonnovel, secretagent, sequel, mi6, ...</td>\n",
       "      <td>[DanielCraig, ChristophWaltz, LéaSeydoux]</td>\n",
       "      <td>[SamMendes]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>[Following, the, death, of, District, Attorney...</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dccomics, crimefighter, terrorist, secretiden...</td>\n",
       "      <td>[ChristianBale, MichaelCaine, GaryOldman]</td>\n",
       "      <td>[ChristopherNolan]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>[John, Carter, is, a, war-weary,, former, mili...</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, ScienceFiction]</td>\n",
       "      <td>[basedonnovel, mars, medallion, spacetravel, p...</td>\n",
       "      <td>[TaylorKitsch, LynnCollins, SamanthaMorton]</td>\n",
       "      <td>[AndrewStanton]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                           overview  \\\n",
       "0     19995  [In, the, 22nd, century,, a, paraplegic, Marin...   \n",
       "1       285  [Captain, Barbossa,, long, believed, to, be, d...   \n",
       "2    206647  [A, cryptic, message, from, Bond’s, past, send...   \n",
       "3     49026  [Following, the, death, of, District, Attorney...   \n",
       "4     49529  [John, Carter, is, a, war-weary,, former, mili...   \n",
       "\n",
       "                                      title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "2                                   Spectre   \n",
       "3                     The Dark Knight Rises   \n",
       "4                               John Carter   \n",
       "\n",
       "                                         genres  \\\n",
       "0  [Action, Adventure, Fantasy, ScienceFiction]   \n",
       "1                  [Adventure, Fantasy, Action]   \n",
       "2                    [Action, Adventure, Crime]   \n",
       "3              [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, ScienceFiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [cultureclash, future, spacewar, spacecolony, ...   \n",
       "1  [ocean, drugabuse, exoticisland, eastindiatrad...   \n",
       "2  [spy, basedonnovel, secretagent, sequel, mi6, ...   \n",
       "3  [dccomics, crimefighter, terrorist, secretiden...   \n",
       "4  [basedonnovel, mars, medallion, spacetravel, p...   \n",
       "\n",
       "                                            cast                crew  \n",
       "0  [SamWorthington, ZoeSaldana, SigourneyWeaver]      [JamesCameron]  \n",
       "1     [JohnnyDepp, OrlandoBloom, KeiraKnightley]     [GoreVerbinski]  \n",
       "2      [DanielCraig, ChristophWaltz, LéaSeydoux]         [SamMendes]  \n",
       "3      [ChristianBale, MichaelCaine, GaryOldman]  [ChristopherNolan]  \n",
       "4    [TaylorKitsch, LynnCollins, SamanthaMorton]     [AndrewStanton]  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Making a single column that contains all text data and removing other columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['tags'] = movies['overview'] + movies['genres'] + movies['keywords'] + movies['cast'] + movies['crew']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>keywords</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>[In, the, 22nd, century,, a, paraplegic, Marin...</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[Action, Adventure, Fantasy, ScienceFiction]</td>\n",
       "      <td>[cultureclash, future, spacewar, spacecolony, ...</td>\n",
       "      <td>[SamWorthington, ZoeSaldana, SigourneyWeaver]</td>\n",
       "      <td>[JamesCameron]</td>\n",
       "      <td>[In, the, 22nd, century,, a, paraplegic, Marin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>[Captain, Barbossa,, long, believed, to, be, d...</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>[ocean, drugabuse, exoticisland, eastindiatrad...</td>\n",
       "      <td>[JohnnyDepp, OrlandoBloom, KeiraKnightley]</td>\n",
       "      <td>[GoreVerbinski]</td>\n",
       "      <td>[Captain, Barbossa,, long, believed, to, be, d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>[A, cryptic, message, from, Bond’s, past, send...</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[Action, Adventure, Crime]</td>\n",
       "      <td>[spy, basedonnovel, secretagent, sequel, mi6, ...</td>\n",
       "      <td>[DanielCraig, ChristophWaltz, LéaSeydoux]</td>\n",
       "      <td>[SamMendes]</td>\n",
       "      <td>[A, cryptic, message, from, Bond’s, past, send...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>[Following, the, death, of, District, Attorney...</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[Action, Crime, Drama, Thriller]</td>\n",
       "      <td>[dccomics, crimefighter, terrorist, secretiden...</td>\n",
       "      <td>[ChristianBale, MichaelCaine, GaryOldman]</td>\n",
       "      <td>[ChristopherNolan]</td>\n",
       "      <td>[Following, the, death, of, District, Attorney...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>[John, Carter, is, a, war-weary,, former, mili...</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[Action, Adventure, ScienceFiction]</td>\n",
       "      <td>[basedonnovel, mars, medallion, spacetravel, p...</td>\n",
       "      <td>[TaylorKitsch, LynnCollins, SamanthaMorton]</td>\n",
       "      <td>[AndrewStanton]</td>\n",
       "      <td>[John, Carter, is, a, war-weary,, former, mili...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                           overview  \\\n",
       "0     19995  [In, the, 22nd, century,, a, paraplegic, Marin...   \n",
       "1       285  [Captain, Barbossa,, long, believed, to, be, d...   \n",
       "2    206647  [A, cryptic, message, from, Bond’s, past, send...   \n",
       "3     49026  [Following, the, death, of, District, Attorney...   \n",
       "4     49529  [John, Carter, is, a, war-weary,, former, mili...   \n",
       "\n",
       "                                      title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "2                                   Spectre   \n",
       "3                     The Dark Knight Rises   \n",
       "4                               John Carter   \n",
       "\n",
       "                                         genres  \\\n",
       "0  [Action, Adventure, Fantasy, ScienceFiction]   \n",
       "1                  [Adventure, Fantasy, Action]   \n",
       "2                    [Action, Adventure, Crime]   \n",
       "3              [Action, Crime, Drama, Thriller]   \n",
       "4           [Action, Adventure, ScienceFiction]   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [cultureclash, future, spacewar, spacecolony, ...   \n",
       "1  [ocean, drugabuse, exoticisland, eastindiatrad...   \n",
       "2  [spy, basedonnovel, secretagent, sequel, mi6, ...   \n",
       "3  [dccomics, crimefighter, terrorist, secretiden...   \n",
       "4  [basedonnovel, mars, medallion, spacetravel, p...   \n",
       "\n",
       "                                            cast                crew  \\\n",
       "0  [SamWorthington, ZoeSaldana, SigourneyWeaver]      [JamesCameron]   \n",
       "1     [JohnnyDepp, OrlandoBloom, KeiraKnightley]     [GoreVerbinski]   \n",
       "2      [DanielCraig, ChristophWaltz, LéaSeydoux]         [SamMendes]   \n",
       "3      [ChristianBale, MichaelCaine, GaryOldman]  [ChristopherNolan]   \n",
       "4    [TaylorKitsch, LynnCollins, SamanthaMorton]     [AndrewStanton]   \n",
       "\n",
       "                                                tags  \n",
       "0  [In, the, 22nd, century,, a, paraplegic, Marin...  \n",
       "1  [Captain, Barbossa,, long, believed, to, be, d...  \n",
       "2  [A, cryptic, message, from, Bond’s, past, send...  \n",
       "3  [Following, the, death, of, District, Attorney...  \n",
       "4  [John, Carter, is, a, war-weary,, former, mili...  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df = movies[['movie_id','title','tags']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df['tags']= new_df['tags'].apply(lambda x:\" \".join(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                                tags  \n",
       "0  In the 22nd century, a paraplegic Marine is di...  \n",
       "1  Captain Barbossa, long believed to be dead, ha...  \n",
       "2  A cryptic message from Bond’s past sends him o...  \n",
       "3  Following the death of District Attorney Harve...  \n",
       "4  John Carter is a war-weary, former military ca...  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'In the 22nd century, a paraplegic Marine is dispatched to the moon Pandora on a unique mission, but becomes torn between following orders and protecting an alien civilization. Action Adventure Fantasy ScienceFiction cultureclash future spacewar spacecolony society spacetravel futuristic romance space alien tribe alienplanet cgi marine soldier battle loveaffair antiwar powerrelations mindandsoul 3d SamWorthington ZoeSaldana SigourneyWeaver JamesCameron'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df['tags'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df['tags']= new_df['tags'].apply(lambda x:x.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>in the 22nd century, a paraplegic marine is di...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>captain barbossa, long believed to be dead, ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>a cryptic message from bond’s past sends him o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>following the death of district attorney harve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>john carter is a war-weary, former military ca...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "4     49529                               John Carter   \n",
       "\n",
       "                                                tags  \n",
       "0  in the 22nd century, a paraplegic marine is di...  \n",
       "1  captain barbossa, long believed to be dead, ha...  \n",
       "2  a cryptic message from bond’s past sends him o...  \n",
       "3  following the death of district attorney harve...  \n",
       "4  john carter is a war-weary, former military ca...  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps = PorterStemmer()\n",
    "\n",
    "def stem(text):\n",
    "    y = []\n",
    "    \n",
    "    for i in text.split():\n",
    "        y.append(ps.stem(i))\n",
    "        \n",
    "    return \" \".join(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_df['tags'] = new_df['tags'].apply(stem)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       ...,\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv = CountVectorizer(max_features=5000,stop_words='english')\n",
    "\n",
    "vectors = cv.fit_transform(new_df['tags']).toarray()\n",
    "vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['000',\n",
       " '007',\n",
       " '10',\n",
       " '100',\n",
       " '11',\n",
       " '12',\n",
       " '13',\n",
       " '14',\n",
       " '15',\n",
       " '16',\n",
       " '17',\n",
       " '18',\n",
       " '18th',\n",
       " '18thcenturi',\n",
       " '19',\n",
       " '1910',\n",
       " '1920',\n",
       " '1930',\n",
       " '1940',\n",
       " '1944',\n",
       " '1950',\n",
       " '1950s',\n",
       " '1960',\n",
       " '1960s',\n",
       " '1970',\n",
       " '1970s',\n",
       " '1971',\n",
       " '1974',\n",
       " '1976',\n",
       " '1980',\n",
       " '1985',\n",
       " '1990',\n",
       " '1999',\n",
       " '19th',\n",
       " '19thcenturi',\n",
       " '20',\n",
       " '200',\n",
       " '2003',\n",
       " '2009',\n",
       " '20th',\n",
       " '21st',\n",
       " '23',\n",
       " '24',\n",
       " '25',\n",
       " '30',\n",
       " '300',\n",
       " '3d',\n",
       " '40',\n",
       " '50',\n",
       " '500',\n",
       " '60',\n",
       " '70',\n",
       " '80',\n",
       " 'aaron',\n",
       " 'aaroneckhart',\n",
       " 'abandon',\n",
       " 'abduct',\n",
       " 'abigailbreslin',\n",
       " 'abil',\n",
       " 'abl',\n",
       " 'aboard',\n",
       " 'abov',\n",
       " 'abus',\n",
       " 'academ',\n",
       " 'academi',\n",
       " 'accept',\n",
       " 'access',\n",
       " 'accid',\n",
       " 'accident',\n",
       " 'acclaim',\n",
       " 'accompani',\n",
       " 'accomplish',\n",
       " 'account',\n",
       " 'accus',\n",
       " 'ace',\n",
       " 'achiev',\n",
       " 'acquaint',\n",
       " 'act',\n",
       " 'action',\n",
       " 'actionhero',\n",
       " 'activ',\n",
       " 'activist',\n",
       " 'activities',\n",
       " 'actor',\n",
       " 'actress',\n",
       " 'actual',\n",
       " 'ad',\n",
       " 'adam',\n",
       " 'adamsandl',\n",
       " 'adamshankman',\n",
       " 'adapt',\n",
       " 'add',\n",
       " 'addict',\n",
       " 'adjust',\n",
       " 'admir',\n",
       " 'admit',\n",
       " 'adolesc',\n",
       " 'adopt',\n",
       " 'ador',\n",
       " 'adrienbrodi',\n",
       " 'adult',\n",
       " 'adultanim',\n",
       " 'adulteri',\n",
       " 'adulthood',\n",
       " 'advanc',\n",
       " 'adventur',\n",
       " 'adventure',\n",
       " 'adventures',\n",
       " 'advertis',\n",
       " 'advic',\n",
       " 'advis',\n",
       " 'affair',\n",
       " 'affect',\n",
       " 'afghanistan',\n",
       " 'africa',\n",
       " 'african',\n",
       " 'africanamerican',\n",
       " 'aftercreditssting',\n",
       " 'afterlif',\n",
       " 'aftermath',\n",
       " 'ag',\n",
       " 'age',\n",
       " 'agediffer',\n",
       " 'agenc',\n",
       " 'agency',\n",
       " 'agenda',\n",
       " 'agent',\n",
       " 'agents',\n",
       " 'aggress',\n",
       " 'ago',\n",
       " 'agre',\n",
       " 'ahead',\n",
       " 'aid',\n",
       " 'aidanquinn',\n",
       " 'ail',\n",
       " 'aim',\n",
       " 'air',\n",
       " 'airplan',\n",
       " 'airplanecrash',\n",
       " 'airport',\n",
       " 'aka',\n",
       " 'al',\n",
       " 'alabama',\n",
       " 'alan',\n",
       " 'alaska',\n",
       " 'albert',\n",
       " 'alcatraz',\n",
       " 'alcohol',\n",
       " 'alecbaldwin',\n",
       " 'alex',\n",
       " 'alexkendrick',\n",
       " 'alfredhitchcock',\n",
       " 'alfredmolina',\n",
       " 'ali',\n",
       " 'alic',\n",
       " 'alice',\n",
       " 'alien',\n",
       " 'alieninvas',\n",
       " 'alienlife',\n",
       " 'alienplanet',\n",
       " 'alik',\n",
       " 'aliv',\n",
       " 'alive',\n",
       " 'allen',\n",
       " 'alli',\n",
       " 'allianc',\n",
       " 'allow',\n",
       " 'alon',\n",
       " 'alongsid',\n",
       " 'alpacino',\n",
       " 'alpha',\n",
       " 'alreadi',\n",
       " 'alter',\n",
       " 'altern',\n",
       " 'alway',\n",
       " 'alyssa',\n",
       " 'alzheimer',\n",
       " 'amanda',\n",
       " 'amandapeet',\n",
       " 'amandaseyfri',\n",
       " 'amateur',\n",
       " 'amaz',\n",
       " 'amazon',\n",
       " 'ambassador',\n",
       " 'ambit',\n",
       " 'ambiti',\n",
       " 'ambul',\n",
       " 'ambush',\n",
       " 'america',\n",
       " 'american',\n",
       " 'americanabroad',\n",
       " 'americancivilwar',\n",
       " 'americanfootbal',\n",
       " 'americanfootballplay',\n",
       " 'amid',\n",
       " 'amidst',\n",
       " 'amnesia',\n",
       " 'amp',\n",
       " 'amsterdam',\n",
       " 'amus',\n",
       " 'amusementpark',\n",
       " 'amy',\n",
       " 'amyadam',\n",
       " 'amysmart',\n",
       " 'ana',\n",
       " 'anakin',\n",
       " 'analyst',\n",
       " 'anarchiccomedi',\n",
       " 'ancient',\n",
       " 'ancientrom',\n",
       " 'ancientworld',\n",
       " 'anderson',\n",
       " 'andi',\n",
       " 'andiemacdowel',\n",
       " 'andrew',\n",
       " 'android',\n",
       " 'andy',\n",
       " 'andygarcía',\n",
       " 'angel',\n",
       " 'angela',\n",
       " 'angelabassett',\n",
       " 'angeles',\n",
       " 'angelinajoli',\n",
       " 'anger',\n",
       " 'angle',\n",
       " 'angri',\n",
       " 'ani',\n",
       " 'anim',\n",
       " 'animalattack',\n",
       " 'animalhorror',\n",
       " 'animals',\n",
       " 'anjelicahuston',\n",
       " 'ann',\n",
       " 'anna',\n",
       " 'annafari',\n",
       " 'annakendrick',\n",
       " 'anne',\n",
       " 'annehathaway',\n",
       " 'annemoss',\n",
       " 'annetteben',\n",
       " 'anni',\n",
       " 'annie',\n",
       " 'anniversari',\n",
       " 'announc',\n",
       " 'annual',\n",
       " 'anonym',\n",
       " 'anoth',\n",
       " 'answer',\n",
       " 'ant',\n",
       " 'antholog',\n",
       " 'anthoni',\n",
       " 'anthonyanderson',\n",
       " 'anthonyhopkin',\n",
       " 'anthonymacki',\n",
       " 'anthropomorph',\n",
       " 'anti',\n",
       " 'antic',\n",
       " 'antihero',\n",
       " 'antiqu',\n",
       " 'antoinefuqua',\n",
       " 'antoniobandera',\n",
       " 'antonyelchin',\n",
       " 'anyon',\n",
       " 'anyth',\n",
       " 'apart',\n",
       " 'apartheid',\n",
       " 'apartment',\n",
       " 'ape',\n",
       " 'apocalyps',\n",
       " 'apocalypse',\n",
       " 'apocalypt',\n",
       " 'appar',\n",
       " 'appear',\n",
       " 'appl',\n",
       " 'apple',\n",
       " 'appoint',\n",
       " 'appreci',\n",
       " 'apprentic',\n",
       " 'approach',\n",
       " 'april',\n",
       " 'aquarium',\n",
       " 'arab',\n",
       " 'arch',\n",
       " 'archaeologist',\n",
       " 'archeolog',\n",
       " 'archer',\n",
       " 'architect',\n",
       " 'arctic',\n",
       " 'area',\n",
       " 'aren',\n",
       " 'arena',\n",
       " 'argument',\n",
       " 'aris',\n",
       " 'aristocrat',\n",
       " 'arm',\n",
       " 'armi',\n",
       " 'armor',\n",
       " 'armsdeal',\n",
       " 'army',\n",
       " 'arnold',\n",
       " 'arnoldschwarzenegg',\n",
       " 'arrang',\n",
       " 'arrangedmarriag',\n",
       " 'arrest',\n",
       " 'arriv',\n",
       " 'arrog',\n",
       " 'art',\n",
       " 'arthur',\n",
       " 'artifact',\n",
       " 'artifici',\n",
       " 'artificialintellig',\n",
       " 'artist',\n",
       " 'ash',\n",
       " 'ashley',\n",
       " 'ashleyjudd',\n",
       " 'ashtonkutch',\n",
       " 'asia',\n",
       " 'asian',\n",
       " 'asid',\n",
       " 'ask',\n",
       " 'aspect',\n",
       " 'aspir',\n",
       " 'assassin',\n",
       " 'assault',\n",
       " 'assembl',\n",
       " 'assign',\n",
       " 'assist',\n",
       " 'assistant',\n",
       " 'associ',\n",
       " 'assum',\n",
       " 'asteroid',\n",
       " 'astronaut',\n",
       " 'asylum',\n",
       " 'atheist',\n",
       " 'athlet',\n",
       " 'atom',\n",
       " 'atomicbomb',\n",
       " 'attack',\n",
       " 'attacks',\n",
       " 'attempt',\n",
       " 'attempttoescap',\n",
       " 'attend',\n",
       " 'attent',\n",
       " 'attic',\n",
       " 'attitud',\n",
       " 'attorney',\n",
       " 'attract',\n",
       " 'auction',\n",
       " 'audienc',\n",
       " 'audit',\n",
       " 'august',\n",
       " 'aunt',\n",
       " 'austin',\n",
       " 'australia',\n",
       " 'australian',\n",
       " 'author',\n",
       " 'autism',\n",
       " 'auto',\n",
       " 'automobilerac',\n",
       " 'aveng',\n",
       " 'averag',\n",
       " 'avoid',\n",
       " 'await',\n",
       " 'awak',\n",
       " 'awaken',\n",
       " 'awar',\n",
       " 'award',\n",
       " 'away',\n",
       " 'awkward',\n",
       " 'awri',\n",
       " 'awry',\n",
       " 'ax',\n",
       " 'babe',\n",
       " 'babi',\n",
       " 'baby',\n",
       " 'bachelor',\n",
       " 'backdrop',\n",
       " 'background',\n",
       " 'backpack',\n",
       " 'bad',\n",
       " 'bag',\n",
       " 'bahama',\n",
       " 'bail',\n",
       " 'balanc',\n",
       " 'ball',\n",
       " 'ballet',\n",
       " 'balloon',\n",
       " 'baltimor',\n",
       " 'ban',\n",
       " 'band',\n",
       " 'bandit',\n",
       " 'bangkok',\n",
       " 'banish',\n",
       " 'bank',\n",
       " 'banker',\n",
       " 'bankrobb',\n",
       " 'bankrobberi',\n",
       " 'bar',\n",
       " 'barbrastreisand',\n",
       " 'bare',\n",
       " 'bargain',\n",
       " 'barn',\n",
       " 'barney',\n",
       " 'baron',\n",
       " 'barri',\n",
       " 'barrylevinson',\n",
       " 'barrysonnenfeld',\n",
       " 'bas',\n",
       " 'base',\n",
       " 'basebal',\n",
       " 'basedoncomicbook',\n",
       " 'basedongraphicnovel',\n",
       " 'basedonnovel',\n",
       " 'basedonplay',\n",
       " 'basedonstagemus',\n",
       " 'basedontrueev',\n",
       " 'basedontruestori',\n",
       " 'basedontvseri',\n",
       " 'basedonvideogam',\n",
       " 'basedonyoungadultnovel',\n",
       " 'basement',\n",
       " 'basketbal',\n",
       " 'basketball',\n",
       " 'bat',\n",
       " 'batman',\n",
       " 'battl',\n",
       " 'battle',\n",
       " 'battlefield',\n",
       " 'bay',\n",
       " 'beach',\n",
       " 'beam',\n",
       " 'bear',\n",
       " 'beard',\n",
       " 'beast',\n",
       " 'beat',\n",
       " 'beauti',\n",
       " 'beautiful',\n",
       " 'beautifulwoman',\n",
       " 'beauty',\n",
       " 'becam',\n",
       " 'becaus',\n",
       " 'becki',\n",
       " 'becom',\n",
       " 'becominganadult',\n",
       " 'bed',\n",
       " 'bedroom',\n",
       " 'bee',\n",
       " 'beer',\n",
       " 'befor',\n",
       " 'befriend',\n",
       " 'began',\n",
       " 'begin',\n",
       " 'begins',\n",
       " 'behavior',\n",
       " 'belief',\n",
       " 'believ',\n",
       " 'bell',\n",
       " 'bella',\n",
       " 'belong',\n",
       " 'belov',\n",
       " 'ben',\n",
       " 'benaffleck',\n",
       " 'bend',\n",
       " 'beneath',\n",
       " 'benefit',\n",
       " 'benfost',\n",
       " 'beniciodeltoro',\n",
       " 'benjamin',\n",
       " 'benjaminbratt',\n",
       " 'benkingsley',\n",
       " 'bennett',\n",
       " 'benstil',\n",
       " 'bent',\n",
       " 'berlin',\n",
       " 'best',\n",
       " 'bestfriend',\n",
       " 'bestfriendsinlov',\n",
       " 'bet',\n",
       " 'beth',\n",
       " 'betray',\n",
       " 'bettemidl',\n",
       " 'better',\n",
       " 'betti',\n",
       " 'beverli',\n",
       " 'bibl',\n",
       " 'bid',\n",
       " 'big',\n",
       " 'bigger',\n",
       " 'biggest',\n",
       " 'bike',\n",
       " 'biker',\n",
       " 'bikini',\n",
       " 'billhad',\n",
       " 'billi',\n",
       " 'billionair',\n",
       " 'billmurray',\n",
       " 'billnighi',\n",
       " 'billpaxton',\n",
       " 'billpullman',\n",
       " 'billybobthornton',\n",
       " 'billycrudup',\n",
       " 'billycryst',\n",
       " 'biographi',\n",
       " 'biolog',\n",
       " 'bird',\n",
       " 'birth',\n",
       " 'birthday',\n",
       " 'bisexu',\n",
       " 'bishop',\n",
       " 'bit',\n",
       " 'bite',\n",
       " 'bitter',\n",
       " 'bizarr',\n",
       " 'black',\n",
       " 'blackmag',\n",
       " 'blackmail',\n",
       " 'blackpeopl',\n",
       " 'blacksmith',\n",
       " 'blade',\n",
       " 'blame',\n",
       " 'blend',\n",
       " 'blind',\n",
       " 'bliss',\n",
       " 'blizzard',\n",
       " 'block',\n",
       " 'blond',\n",
       " 'blood',\n",
       " 'bloodi',\n",
       " 'bloodsplatt',\n",
       " 'bloodthirsti',\n",
       " 'blow',\n",
       " 'blue',\n",
       " 'board',\n",
       " 'boardingschool',\n",
       " 'boat',\n",
       " 'bob',\n",
       " 'bobbi',\n",
       " 'bobbyfarrelli',\n",
       " 'bobhoskin',\n",
       " 'bodi',\n",
       " 'body',\n",
       " 'bodyguard',\n",
       " 'bold',\n",
       " 'bollywood',\n",
       " 'bomb',\n",
       " 'bond',\n",
       " 'bone',\n",
       " 'book',\n",
       " 'border',\n",
       " 'bore',\n",
       " 'boredom',\n",
       " 'born',\n",
       " 'boss',\n",
       " 'boston',\n",
       " 'botch',\n",
       " 'bound',\n",
       " 'boundari',\n",
       " 'bounti',\n",
       " 'bountyhunt',\n",
       " 'bout',\n",
       " 'box',\n",
       " 'boxer',\n",
       " 'boy',\n",
       " 'boyfriend',\n",
       " 'boys',\n",
       " 'bradleycoop',\n",
       " 'bradpitt',\n",
       " 'brain',\n",
       " 'brainwash',\n",
       " 'brand',\n",
       " 'brandon',\n",
       " 'brave',\n",
       " 'braveri',\n",
       " 'brazil',\n",
       " 'brazilian',\n",
       " 'break',\n",
       " 'breakdown',\n",
       " 'breakup',\n",
       " 'breast',\n",
       " 'breath',\n",
       " 'breed',\n",
       " 'brendanfras',\n",
       " 'brendangleeson',\n",
       " 'brent',\n",
       " 'brettratn',\n",
       " 'brian',\n",
       " 'briandepalma',\n",
       " 'bride',\n",
       " 'bridesmaid',\n",
       " 'bridg',\n",
       " 'brief',\n",
       " 'brielarson',\n",
       " 'brien',\n",
       " 'bright',\n",
       " 'brilliant',\n",
       " 'bring',\n",
       " 'brink',\n",
       " 'britain',\n",
       " 'british',\n",
       " 'britishsecretservic',\n",
       " 'brittanymurphi',\n",
       " 'broadcast',\n",
       " 'broadway',\n",
       " 'broke',\n",
       " 'broken',\n",
       " 'broker',\n",
       " 'bronx',\n",
       " 'brook',\n",
       " 'brooklyn',\n",
       " 'broom',\n",
       " 'brothel',\n",
       " 'brother',\n",
       " 'brotherbrotherrelationship',\n",
       " 'brothers',\n",
       " 'brothersisterrelationship',\n",
       " 'brought',\n",
       " 'brown',\n",
       " 'bruce',\n",
       " 'brucegreenwood',\n",
       " 'brucewilli',\n",
       " 'brutal',\n",
       " 'bryansing',\n",
       " 'bu',\n",
       " 'buck',\n",
       " 'bud',\n",
       " 'buddi',\n",
       " 'buddy',\n",
       " 'buddycomedi',\n",
       " 'buddycop',\n",
       " 'budget',\n",
       " 'build',\n",
       " 'building',\n",
       " 'built',\n",
       " 'bullet',\n",
       " 'bulli',\n",
       " 'bumbl',\n",
       " 'bunch',\n",
       " 'bunker',\n",
       " 'bunni',\n",
       " 'burglar',\n",
       " 'buri',\n",
       " 'burn',\n",
       " 'bush',\n",
       " 'busi',\n",
       " 'business',\n",
       " 'businessman',\n",
       " 'bust',\n",
       " 'butcher',\n",
       " 'butler',\n",
       " 'butt',\n",
       " 'button',\n",
       " 'buy',\n",
       " 'buzz',\n",
       " 'cabin',\n",
       " 'caesar',\n",
       " 'cage',\n",
       " 'cairo',\n",
       " 'cal',\n",
       " 'california',\n",
       " 'calvin',\n",
       " 'camcord',\n",
       " 'came',\n",
       " 'camera',\n",
       " 'cameraman',\n",
       " 'camerondiaz',\n",
       " 'camp',\n",
       " 'campaign',\n",
       " 'campbell',\n",
       " 'campu',\n",
       " 'canada',\n",
       " 'canadian',\n",
       " 'cancer',\n",
       " 'candi',\n",
       " 'candid',\n",
       " 'canin',\n",
       " 'cannib',\n",
       " 'canuxploit',\n",
       " 'capabl',\n",
       " 'caper',\n",
       " 'capit',\n",
       " 'capt',\n",
       " 'captain',\n",
       " 'captiv',\n",
       " 'captur',\n",
       " 'capture',\n",
       " 'car',\n",
       " 'caraccid',\n",
       " 'carchas',\n",
       " 'carcrash',\n",
       " 'card',\n",
       " 'care',\n",
       " 'career',\n",
       " 'carefre',\n",
       " 'caretak',\n",
       " 'careymulligan',\n",
       " 'caribbean',\n",
       " 'carjourney',\n",
       " 'carl',\n",
       " 'carlagugino',\n",
       " 'carmen',\n",
       " 'carol',\n",
       " 'carolina',\n",
       " 'carrac',\n",
       " 'carri',\n",
       " 'carrie',\n",
       " 'cartel',\n",
       " 'carter',\n",
       " 'cartoon',\n",
       " 'caryelw',\n",
       " 'case',\n",
       " 'caseyaffleck',\n",
       " 'cash',\n",
       " 'casino',\n",
       " 'cast',\n",
       " 'castl',\n",
       " 'cat',\n",
       " 'cataclysm',\n",
       " 'catastroph',\n",
       " 'catch',\n",
       " 'cateblanchett',\n",
       " 'catherinedeneuv',\n",
       " 'catherinekeen',\n",
       " 'catherinezeta',\n",
       " 'cathol',\n",
       " 'catholic',\n",
       " 'cattl',\n",
       " 'caught',\n",
       " 'caus',\n",
       " 'cavalri',\n",
       " 'cave',\n",
       " 'cavemen',\n",
       " 'celebr',\n",
       " 'celebration',\n",
       " 'cell',\n",
       " 'cellphon',\n",
       " 'cemeteri',\n",
       " 'center',\n",
       " 'centr',\n",
       " 'central',\n",
       " 'centuri',\n",
       " 'centuries',\n",
       " 'century',\n",
       " 'ceo',\n",
       " 'certain',\n",
       " 'chad',\n",
       " 'chain',\n",
       " 'chainsaw',\n",
       " 'challeng',\n",
       " 'chamber',\n",
       " 'champion',\n",
       " 'championship',\n",
       " 'chanc',\n",
       " 'chance',\n",
       " 'chang',\n",
       " 'change',\n",
       " 'changed',\n",
       " 'changes',\n",
       " 'channingtatum',\n",
       " 'chao',\n",
       " 'chaos',\n",
       " 'chaotic',\n",
       " 'chapter',\n",
       " 'charact',\n",
       " 'character',\n",
       " 'characters',\n",
       " 'charg',\n",
       " 'charismat',\n",
       " 'charl',\n",
       " 'charli',\n",
       " 'charlie',\n",
       " 'charliesheen',\n",
       " 'charlizetheron',\n",
       " 'charm',\n",
       " 'chart',\n",
       " 'chase',\n",
       " 'chauffeur',\n",
       " 'chazzpalminteri',\n",
       " 'cheat',\n",
       " 'check',\n",
       " 'cheerlead',\n",
       " 'chef',\n",
       " 'chemic',\n",
       " 'cher',\n",
       " 'chevychas',\n",
       " 'chicago',\n",
       " 'chicken',\n",
       " 'chief',\n",
       " 'child',\n",
       " 'childabus',\n",
       " 'childhero',\n",
       " 'childhood',\n",
       " 'childprodigi',\n",
       " 'children',\n",
       " 'chill',\n",
       " 'chimp',\n",
       " 'china',\n",
       " 'chines',\n",
       " 'chip',\n",
       " 'chipmunk',\n",
       " 'chiwetelejiofor',\n",
       " 'chloe',\n",
       " 'chloëgracemoretz',\n",
       " 'chloësevigni',\n",
       " 'chocol',\n",
       " 'choic',\n",
       " 'choice',\n",
       " 'choos',\n",
       " 'chosen',\n",
       " 'chowyun',\n",
       " 'chri',\n",
       " 'chriscolumbu',\n",
       " 'chriscoop',\n",
       " 'chrisevan',\n",
       " 'chrishemsworth',\n",
       " 'chrisklein',\n",
       " 'chrispin',\n",
       " 'chrisrock',\n",
       " 'christ',\n",
       " 'christian',\n",
       " 'christianbal',\n",
       " 'christianslat',\n",
       " 'christin',\n",
       " 'christinaappleg',\n",
       " 'christinaricci',\n",
       " 'christma',\n",
       " 'christmas',\n",
       " 'christmasparti',\n",
       " 'christmastre',\n",
       " 'christoph',\n",
       " 'christopherlambert',\n",
       " 'christopherlloyd',\n",
       " 'christophernolan',\n",
       " 'christopherplumm',\n",
       " 'christopherwalken',\n",
       " 'christophwaltz',\n",
       " 'chrisweitz',\n",
       " 'chronicl',\n",
       " 'chuck',\n",
       " 'church',\n",
       " 'cia',\n",
       " 'ciaránhind',\n",
       " 'cigarettesmok',\n",
       " 'cillianmurphi',\n",
       " 'cinema',\n",
       " 'circl',\n",
       " 'circu',\n",
       " 'circuit',\n",
       " 'circumst',\n",
       " 'citi',\n",
       " 'citizen',\n",
       " 'city',\n",
       " 'civil',\n",
       " 'civilian',\n",
       " 'civilwar',\n",
       " 'claim',\n",
       " 'clair',\n",
       " 'clairedan',\n",
       " 'claireforlani',\n",
       " 'clan',\n",
       " 'clark',\n",
       " 'clash',\n",
       " 'class',\n",
       " 'classdiffer',\n",
       " 'classic',\n",
       " 'classmat',\n",
       " 'classroom',\n",
       " 'claudevandamm',\n",
       " 'clay',\n",
       " 'clean',\n",
       " 'clear',\n",
       " 'clerk',\n",
       " 'clever',\n",
       " 'client',\n",
       " 'clients',\n",
       " 'cliff',\n",
       " 'climat',\n",
       " 'climb',\n",
       " 'clinteastwood',\n",
       " 'cliveowen',\n",
       " 'clock',\n",
       " 'clone',\n",
       " 'close',\n",
       " 'closer',\n",
       " 'cloud',\n",
       " 'clown',\n",
       " 'club',\n",
       " 'clue',\n",
       " 'clueless',\n",
       " 'clutch',\n",
       " 'coach',\n",
       " 'coast',\n",
       " 'cocain',\n",
       " 'code',\n",
       " 'coffin',\n",
       " 'cohen',\n",
       " 'col',\n",
       " 'cold',\n",
       " 'coldwar',\n",
       " 'cole',\n",
       " 'colin',\n",
       " 'colinfarrel',\n",
       " 'colinfirth',\n",
       " 'collaps',\n",
       " 'colleagu',\n",
       " 'collect',\n",
       " 'collector',\n",
       " 'colleg',\n",
       " 'college',\n",
       " 'collid',\n",
       " 'collis',\n",
       " 'colombia',\n",
       " 'colonel',\n",
       " 'coloni',\n",
       " 'color',\n",
       " 'colorado',\n",
       " 'coma',\n",
       " 'combat',\n",
       " 'combin',\n",
       " 'come',\n",
       " 'comeback',\n",
       " 'comed',\n",
       " 'comedi',\n",
       " 'comedian',\n",
       " 'comedy',\n",
       " 'comet',\n",
       " 'comfort',\n",
       " 'comic',\n",
       " 'coming',\n",
       " 'comingofag',\n",
       " 'comingout',\n",
       " 'command',\n",
       " 'commando',\n",
       " 'commerci',\n",
       " 'commiss',\n",
       " 'commit',\n",
       " 'common',\n",
       " 'commun',\n",
       " 'communist',\n",
       " 'community',\n",
       " 'compani',\n",
       " 'companion',\n",
       " 'company',\n",
       " 'compet',\n",
       " 'competit',\n",
       " 'competition',\n",
       " 'complet',\n",
       " 'complex',\n",
       " 'complic',\n",
       " 'compos',\n",
       " 'compuls',\n",
       " 'comput',\n",
       " 'computerviru',\n",
       " 'conan',\n",
       " 'concern',\n",
       " 'concert',\n",
       " 'concoct',\n",
       " 'condit',\n",
       " 'condition',\n",
       " 'conduct',\n",
       " 'confeder',\n",
       " 'confess',\n",
       " 'confid',\n",
       " 'confin',\n",
       " 'conflict',\n",
       " 'confront',\n",
       " 'confus',\n",
       " 'congress',\n",
       " 'conman',\n",
       " 'connect',\n",
       " 'connecticut',\n",
       " 'connel',\n",
       " 'connor',\n",
       " 'conquer',\n",
       " 'consequ',\n",
       " 'consequences',\n",
       " 'conserv',\n",
       " 'consid',\n",
       " 'conspir',\n",
       " 'conspiraci',\n",
       " 'conspiracy',\n",
       " 'constant',\n",
       " 'constantli',\n",
       " 'construct',\n",
       " 'consum',\n",
       " 'contact',\n",
       " 'contain',\n",
       " 'contemporari',\n",
       " 'contend',\n",
       " 'content',\n",
       " 'contest',\n",
       " 'continu',\n",
       " 'contract',\n",
       " 'contractor',\n",
       " 'control',\n",
       " 'controversi',\n",
       " 'convent',\n",
       " 'converg',\n",
       " 'convers',\n",
       " 'convict',\n",
       " 'convinc',\n",
       " ...]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.get_feature_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Comparing the data using cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity = cosine_similarity(vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 0.08346223, 0.0860309 , ..., 0.04499213, 0.        ,\n",
       "        0.        ],\n",
       "       [0.08346223, 1.        , 0.06063391, ..., 0.02378257, 0.        ,\n",
       "        0.02615329],\n",
       "       [0.0860309 , 0.06063391, 1.        , ..., 0.02451452, 0.        ,\n",
       "        0.        ],\n",
       "       ...,\n",
       "       [0.04499213, 0.02378257, 0.02451452, ..., 1.        , 0.03962144,\n",
       "        0.04229549],\n",
       "       [0.        , 0.        , 0.        , ..., 0.03962144, 1.        ,\n",
       "        0.08714204],\n",
       "       [0.        , 0.02615329, 0.        , ..., 0.04229549, 0.08714204,\n",
       "        1.        ]])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function to get similar movie suggestion on the  basis of model created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommend(movie):\n",
    "    movie_index = new_df[new_df['title']== movie].index[0]\n",
    "    distances = similarity[movie_index]\n",
    "    movies_list = sorted(list(enumerate(distances)),reverse=True,key=lambda x:x[1])[1:6]\n",
    "    \n",
    "    for i in movies_list:\n",
    "        print(new_df.iloc[i[0]].title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Texas Chainsaw Massacre 2\n",
      "Dwegons\n",
      "The Master of Disguise\n",
      "Karachi se Lahore\n",
      "Ri¢hie Ri¢h\n"
     ]
    }
   ],
   "source": [
    "recommend('Alien')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Storing our data into a database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(new_df.to_dict(),open('movies_dict.pkl','wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(similarity,open('similarity.pkl','wb'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}