Term frequecy - Inverse Document frequency.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings;warnings.filterwarnings('ignore')\n",
    "import collections\n",
    "import pandas as pd\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We are going to experiment tf-idf on 3 sets of texts & document similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_sets = {\n",
    "    'set-1':[\"india won 2011 world cup\",\n",
    "            \"virat is captain of indian cricket team\",\n",
    "            \"Music is an art form and cultural activity whose medium is sound\"\n",
    "           ],\n",
    "    'set-2':[\"indian cricket team won 2011 world cup\",\n",
    "            \"virat is captain of indian cricket team\",\n",
    "            \"Music is an art form and cultural activity whose medium is sound\"\n",
    "           ],\n",
    "    'set-3':[\"man bites dog\",\n",
    "             \"dog bites man\"\n",
    "           ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs1 = doc_sets['set-1']\n",
    "docs2 = doc_sets['set-2']\n",
    "docs3 = doc_sets['set-3']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs1 = list(map(lambda x: x.split(),docs1))\n",
    "docs2 = list(map(lambda x: x.split(),docs2))\n",
    "docs3 = list(map(lambda x: x.split(),docs3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs1 = collections.OrderedDict({i:doc for i,doc in enumerate(docs1)})\n",
    "docs2 = collections.OrderedDict({i:doc for i,doc in enumerate(docs2)})\n",
    "docs3 = collections.OrderedDict({i:doc for i,doc in enumerate(docs3)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k,v in docs1.items():\n",
    "    docs1[k] = [word for word in v if word not in STOP_WORDS]\n",
    "    \n",
    "for k,v in docs2.items():\n",
    "    docs2[k] = [word for word in v if word not in STOP_WORDS]\n",
    "    \n",
    "for k,v in docs3.items():\n",
    "    docs3[k] = [word for word in v if word not in STOP_WORDS]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc set-1 17\n",
      "Doc set-2 16\n",
      "Doc set-3 3\n"
     ]
    }
   ],
   "source": [
    "uniquewords1 = set()\n",
    "for doc in docs1.values():\n",
    "    uniquewords1.update(doc)\n",
    "print(\"Doc set-1\",len(uniquewords1))\n",
    "\n",
    "uniquewords2 = set()\n",
    "for doc in docs2.values():\n",
    "    uniquewords2.update(doc)\n",
    "print(\"Doc set-2\",len(uniquewords2))\n",
    "\n",
    "uniquewords3 = set()\n",
    "for doc in docs3.values():\n",
    "    uniquewords3.update(doc)\n",
    "print(\"Doc set-3\",len(uniquewords3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_of_words1=collections.OrderedDict()\n",
    "for k,words in docs1.items():\n",
    "    num_of_words1[k] = dict.fromkeys(uniquewords1,0)\n",
    "    for word in words:\n",
    "        num_of_words1[k][word]+=1\n",
    "        \n",
    "num_of_words2=collections.OrderedDict()\n",
    "for k,words in docs2.items():\n",
    "    num_of_words2[k] = dict.fromkeys(uniquewords2,0)\n",
    "    for word in words:\n",
    "        num_of_words2[k][word]+=1\n",
    "        \n",
    "num_of_words3=collections.OrderedDict()\n",
    "for k,words in docs3.items():\n",
    "    num_of_words3[k] = dict.fromkeys(uniquewords3,0)\n",
    "    for word in words:\n",
    "        num_of_words3[k][word]+=1\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Term Frequency(TF):\n",
    "\n",
    "    number of times each term occurs in each document\n",
    "    \n",
    "    weight of a term that occurs in a document is proportional to the term frequency\n",
    "    \n",
    "    tf(t,d) = count of t in d / number of terms in d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_tf(num_of_words,docs):\n",
    "    tf=collections.OrderedDict()\n",
    "    for k,doc in num_of_words.items():\n",
    "        tf[k]={}\n",
    "        doc_len = len(docs[k])\n",
    "        for word,count in doc.items():\n",
    "            tf[k][word]=count/doc_len\n",
    "    return tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf1 = compute_tf(num_of_words1,docs1)\n",
    "tf2 = compute_tf(num_of_words2,docs2)\n",
    "tf3 = compute_tf(num_of_words3,docs3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document Frequency(DF):\n",
    "\n",
    "    Document Frequency is the count of occurrences of term t in the document set N.\n",
    "    \n",
    "    df(t) = occurences of t in documents\n",
    "    \n",
    "## Inverse Document Frequency(IDF):   \n",
    "\n",
    "    While computing TF, all the terms are considered equally important. However, it is known that certain terms \"is, of, that\", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing IDF, an inverse document frequency factor is incorporated which diminishes the weight of terms that occur very frequently in the document set and increases the weight of terms that occur rarely.\n",
    "    \n",
    "    IDF is the inverse of the document frequency which measures the informativeness of term t.\n",
    "    \n",
    "    idf(t) = N/df(t)\n",
    "    \n",
    "    Now there are few other problems with the IDF , in case of a large corpus,say 100,000,000 , the IDF value explodes , to avoid the effect we take the log of idf .\n",
    "    \n",
    "    idf(t) = log( N/ df(t) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_idf(uniquewords,num_of_words):\n",
    "    N = len(num_of_words)\n",
    "    idf = dict.fromkeys(uniquewords,0)\n",
    "    for k,doc in num_of_words.items():\n",
    "        for word,count in doc.items():\n",
    "            if count>0:\n",
    "                idf[word]+=1\n",
    "    for word,val in idf.items():\n",
    "        idf[word] = math.log(N/float(val+1))\n",
    "    return idf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "idf1 = compute_idf(uniquewords1,num_of_words1)\n",
    "idf2 = compute_idf(uniquewords2,num_of_words2)\n",
    "idf3 = compute_idf(uniquewords3,num_of_words3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Term Frequency - Inverse Document Frequency:\n",
    "\n",
    "    tf-idf is a the right measure to evaluate how important a word is to a document in a collection or corpus.\n",
    "    \n",
    "    tf-idf(t,d) = tf(t,d) * idf(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_tfidf(tf,idf):\n",
    "    tfidf=collections.OrderedDict()\n",
    "    for k,doc in tf.items():\n",
    "        tfidf[k]={}\n",
    "        for word,term_freq in doc.items():\n",
    "            tfidf[k][word] = term_freq*idf[word]\n",
    "    return tfidf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf1 = compute_tfidf(tf1,idf1)\n",
    "tfidf2 = compute_tfidf(tf2,idf2)\n",
    "tfidf3 = compute_tfidf(tf3,idf3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doc Set 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>2011</th>\n",
       "      <th>Music</th>\n",
       "      <th>activity</th>\n",
       "      <th>art</th>\n",
       "      <th>captain</th>\n",
       "      <th>cricket</th>\n",
       "      <th>cultural</th>\n",
       "      <th>cup</th>\n",
       "      <th>form</th>\n",
       "      <th>india</th>\n",
       "      <th>indian</th>\n",
       "      <th>medium</th>\n",
       "      <th>sound</th>\n",
       "      <th>team</th>\n",
       "      <th>virat</th>\n",
       "      <th>won</th>\n",
       "      <th>world</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.081093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       2011     Music  activity       art   captain   cricket  cultural  \\\n",
       "0  0.081093  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000   \n",
       "1  0.000000  0.000000  0.000000  0.000000  0.081093  0.081093  0.000000   \n",
       "2  0.000000  0.057924  0.057924  0.057924  0.000000  0.000000  0.057924   \n",
       "\n",
       "        cup      form     india    indian    medium     sound      team  \\\n",
       "0  0.081093  0.000000  0.081093  0.000000  0.000000  0.000000  0.000000   \n",
       "1  0.000000  0.000000  0.000000  0.081093  0.000000  0.000000  0.081093   \n",
       "2  0.000000  0.057924  0.000000  0.000000  0.057924  0.057924  0.000000   \n",
       "\n",
       "      virat       won     world  \n",
       "0  0.000000  0.081093  0.081093  \n",
       "1  0.081093  0.000000  0.000000  \n",
       "2  0.000000  0.000000  0.000000  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_df1 = pd.DataFrame(tfidf1).T\n",
    "tfidf_df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = tfidf_df1.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def document_sim(A):\n",
    "    doc_dot_mat = np.matmul(A,A.T)\n",
    "    mod = np.sqrt(np.square(A).sum(1))\n",
    "    res = doc_dot_mat / np.matmul(mod.reshape(-1,1),mod.reshape(1,-1))\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_sim1 = document_sim(tfidf_df1.to_numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc-1 vs Doc-2 = 0.0\n",
      "Doc-1 vs Doc-3 = 0.0\n",
      "Doc-2 vs Doc-3 = 0.0\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(docs1)):\n",
    "    for j in range(i+1,len(docs1)):\n",
    "        print(\"Doc-{} vs Doc-{} = {}\".format(i+1,j+1,doc_sim1[i,j]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Since, there are no common words in all 3 different texts, the similarity between them is 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doc Set 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>2011</th>\n",
       "      <th>Music</th>\n",
       "      <th>activity</th>\n",
       "      <th>art</th>\n",
       "      <th>captain</th>\n",
       "      <th>cricket</th>\n",
       "      <th>cultural</th>\n",
       "      <th>cup</th>\n",
       "      <th>form</th>\n",
       "      <th>indian</th>\n",
       "      <th>medium</th>\n",
       "      <th>sound</th>\n",
       "      <th>team</th>\n",
       "      <th>virat</th>\n",
       "      <th>won</th>\n",
       "      <th>world</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.057924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.081093</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.057924</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       2011     Music  activity       art   captain  cricket  cultural  \\\n",
       "0  0.057924  0.000000  0.000000  0.000000  0.000000      0.0  0.000000   \n",
       "1  0.000000  0.000000  0.000000  0.000000  0.081093      0.0  0.000000   \n",
       "2  0.000000  0.057924  0.057924  0.057924  0.000000      0.0  0.057924   \n",
       "\n",
       "        cup      form  indian    medium     sound  team     virat       won  \\\n",
       "0  0.057924  0.000000     0.0  0.000000  0.000000   0.0  0.000000  0.057924   \n",
       "1  0.000000  0.000000     0.0  0.000000  0.000000   0.0  0.081093  0.000000   \n",
       "2  0.000000  0.057924     0.0  0.057924  0.057924   0.0  0.000000  0.000000   \n",
       "\n",
       "      world  \n",
       "0  0.057924  \n",
       "1  0.000000  \n",
       "2  0.000000  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_df2 = pd.DataFrame(tfidf2).T\n",
    "tfidf_df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = tfidf_df2.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def document_sim(A):\n",
    "    doc_dot_mat = np.matmul(A,A.T)\n",
    "    mod = np.sqrt(np.square(A).sum(1))\n",
    "    res = doc_dot_mat / np.matmul(mod.reshape(-1,1),mod.reshape(1,-1))\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_sim2 = document_sim(tfidf_df2.to_numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc-1 vs Doc-2 = 0.0\n",
      "Doc-1 vs Doc-3 = 0.0\n",
      "Doc-2 vs Doc-3 = 0.0\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(docs2)):\n",
    "    for j in range(i+1,len(docs2)):\n",
    "        print(\"Doc-{} vs Doc-{} = {}\".format(i+1,j+1,doc_sim2[i,j]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Document 1 and Document 2 has common words \"indian, cricket, team\" - it has simalirity 0.125. But other combinations has no common words, so similarity is 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doc Set 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bites</th>\n",
       "      <th>dog</th>\n",
       "      <th>man</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.135155</td>\n",
       "      <td>-0.135155</td>\n",
       "      <td>-0.135155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.135155</td>\n",
       "      <td>-0.135155</td>\n",
       "      <td>-0.135155</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      bites       dog       man\n",
       "0 -0.135155 -0.135155 -0.135155\n",
       "1 -0.135155 -0.135155 -0.135155"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_df3 = pd.DataFrame(tfidf3).T\n",
    "tfidf_df3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = tfidf_df3.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def document_sim(A):\n",
    "    doc_dot_mat = np.matmul(A,A.T)\n",
    "    mod = np.sqrt(np.square(A).sum(1))\n",
    "    res = doc_dot_mat / np.matmul(mod.reshape(-1,1),mod.reshape(1,-1))\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_sim3 = document_sim(tfidf_df3.to_numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc-1 vs Doc-2 = 1.0\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(docs3)):\n",
    "    for j in range(i+1,len(docs3)):\n",
    "        print(\"Doc-{} vs Doc-{} = {}\".format(i+1,j+1,doc_sim3[i,j]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    Document 1 and Document 2 has all words in common ,it has simalirity 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference:\n",
    "    \n",
    "    (i) word's occurence in more number of documents inversely proportional to tf-idf score\n",
    "    \n",
    "    (ii) if word in a document is occured more number of times, term frequency will be high (i.e., word occured 5 times out of 10 words in a document, 5/10 = 0.5 whereas other words occured 1 time, 1/10=0.1)\n",
    "    \n",
    "    (iii) following point (ii), if word present in very few documents, word will have high tfidf score. For instance,\n",
    "    \n",
    "        Total documents = 200\n",
    "        \n",
    "                    (A)                                            (B)\n",
    "        \n",
    "        word - occured documents = 10                 word - occured documents = 190\n",
    "        IDF = math.log(200/10) = 2.9957               IDF = math.log(200/190) = 0.0512\n",
    "        \n",
    "        Assume document of length 10 and word occured only one time, thus tf will be 0.1\n",
    "        tf = 1/10 = 0.1\n",
    "        TFIDF = 0.1 * 2.9957 = 0.2995                 TFIDF = 0.1 * 0.0512 = 0.0051\n",
    "        \n",
    "        See, (A) has high tfidf score than (B)\n",
    "        \n",
    "    (iv) if word present in very few documents and occured more times in a document, word will have very high tfidf score. For instance,\n",
    "    \n",
    "        Total documents = 200\n",
    "        \n",
    "                    (A)                                          (B)\n",
    "        \n",
    "        word - occured documents = 5                  word - occured documents = 180\n",
    "        IDF = math.log(200/5) = 3.6888                IDF = math.log(200/190) = 0.1053\n",
    "        \n",
    "        Assume document of length 10 and word occured 7 times, thus tf will be 0.7\n",
    "        tf = 7/10 = 0.7\n",
    "        TFIDF = 0.7 * 3.6888 = 2.5822                 TFIDF = 0.7 * 0.1053 = 0.0737\n",
    "        \n",
    "        See, (A) has very high tfidf score than (B) because of its high term frequency."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Disadvantage:\n",
    "\n",
    "    (i) From doc set 1:\n",
    "\n",
    "        Doc 1: india won 2011 world cup\n",
    "\n",
    "        Dco 2: virat is captain of indian cricket team\n",
    "\n",
    "        Both these docs are related, but tf-idf cannot capture document similarity between them, because exact terms are not present in both.\n",
    "        \n",
    "        As a result, fails to capture words similarity. i.e., india & indian\n",
    "        \n",
    "    (ii) From doc set 3:\n",
    "    \n",
    "        Doc 1: man bites dog\n",
    "        \n",
    "        Dco 2: dog bites man\n",
    "        \n",
    "        Both docs are semantically different, but tfidf vector for both the docs are same, hence simialrity between them turns out to be 1.\n",
    "        \n",
    "        As a result, fails to capture semantics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advantage:\n",
    "\n",
    "    (i) simple algorithm for matching query words relavant to document words\n",
    "    \n",
    "    (ii) easy to compute"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}