IR_govt_documents.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IR for Governmemt related Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "* Government documents and TREC eval files are present in zip format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "# Put all your imports here\n",
    "from whoosh import index, writing\n",
    "from whoosh.fields import Schema, TEXT, KEYWORD, ID, STORED\n",
    "from whoosh.analysis import *\n",
    "from whoosh.qparser import QueryParser\n",
    "import os.path\n",
    "from pathlib import Path\n",
    "import tempfile\n",
    "import subprocess\n",
    "import nltk\n",
    "from nltk.stem import *\n",
    "from whoosh import qparser\n",
    "from whoosh import scoring "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_DIR = \"government\"\n",
    "\n",
    "# Put other path constants here\n",
    "\n",
    "DOCUMENTS_DIR = os.path.join(DATA_DIR, \"documents\")\n",
    "TOPIC_FILE = os.path.join(DATA_DIR, \"gov.topics\")\n",
    "QRELS_FILE = os.path.join(DATA_DIR, \"gov.qrels\")\n",
    "\n",
    "\n",
    "TREC_EVAL = os.path.join(\"trec_eval\", \"trec_eval.exe\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choosing the Best Evaluative Measure for this task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Theres is no particular best evaluative measure for information retrival (Search Engine) tasks. Choice of evaluation metric should be based on the Information need for that task. (Consider whether it is a Recall specific task or Precision based task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The choosen measure for the given goverenment data collection  is - P5 (precision, in the first five documents returned)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reasons for choosing P5:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the topic descriptions it is clear that users are looking for general information about different topics impliying that this is not a recall specific task - but required relevant information should be retrived in top results hence I selected \"P5 - Precision in the first five documents returned\" as an appropriate measure for this case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document Preprocessing & Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Whoosh library: **\n",
    "Whoosh is a fast, pure Python search engine library. It helps us to index free-form or structured text and then quickly find matching documents based on simple or complex search criteria. Also, it has Pluggable scoring algorithm (including BM25F), text analysis, storage, posting format, etc.\n",
    "https://whoosh.readthedocs.io/en/latest/intro.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Schema:\n",
    "\n",
    "In whoosh, structure the retrieval system by defining a storage schema. Each document can have multiple fields, such as title, content, url, date, etc. Firstly, we need to create a schema for our corpus to specify these fields of documents in an index.\n",
    "\n",
    "The schema is the set of all possible fields in a document. Each individual document might only use a subset of the available fields in the schema.\n",
    "\n",
    "note: without a schema, the query parser in Whoosh will not process the text in the user query (i.e., cannot do phrase searching)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining Schema for the index -  Vanilla Whoosh (Base case Schema)\n",
    "Base_Schema = Schema(file_path = ID(stored=True),\n",
    "                  file_content = TEXT(analyzer = RegexTokenizer()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Index documents:\n",
    "After creating our schema, we will index each document in the corpus. 1. Indexed fields must be passed a unicode value.\n",
    "Opening a writer locks the index for writing. In a multi-threaded or multi-process environment, opening a writer may raise an error if a writer is already open. Advanced writer object \"whoosh.writing.AsyncWriter\" and \"whoosh.writing.BufferedWriter\" can solve this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def createIndex(schema):\n",
    "    # Generate a temporary directory for the index\n",
    "    indexDir = tempfile.mkdtemp()\n",
    "    # creating and returning the index\n",
    "    return index.create_in(indexDir, schema)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the index at the path INDEX_DIR based on the Base_Schema\n",
    "Base_Index = createIndex(Base_Schema)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function for indexing the files\n",
    "def addFilesToIndex(indexObj, fileList):\n",
    "    # open writer\n",
    "    writer = writing.BufferedWriter(indexObj, period=None, limit=4000)\n",
    "    \n",
    "    ## Used BufferedWriter because it solves the problem of multiple openings of a writer(problem of locking the index for writing)\n",
    "    ## Also, using a BufferedWriter is much faster than opening and committing a writer for each document when adding multiple docs\n",
    "    try:\n",
    "        # Opening and reading each file/documents\n",
    "        for docNum, filePath in enumerate(fileList):\n",
    "            with open(filePath, \"r\", encoding=\"utf-8\") as f:\n",
    "                fileContent = f.read()\n",
    "                # Adding documents to the index\n",
    "                writer.add_document(file_path = filePath,\n",
    "                                    file_content = fileContent)\n",
    "    \n",
    "                if (docNum+1) % 1000 == 0:\n",
    "                    print(\"already indexed:\", docNum+1)\n",
    "        print(\"done indexing.\")\n",
    "\n",
    "    finally:\n",
    "        # close the writer to save the uncommited changes\n",
    "        writer.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Building the list of files for indexing\n",
    "filesToIndex = [str(filePath) for filePath in Path(DOCUMENTS_DIR).glob(\"**/*\") if filePath.is_file()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "already indexed: 1000\n",
      "already indexed: 2000\n",
      "already indexed: 3000\n",
      "already indexed: 4000\n",
      "done indexing.\n"
     ]
    }
   ],
   "source": [
    "#calling addFilesToIndex function for indexing the files\n",
    "addFilesToIndex(Base_Index, filesToIndex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query parser: \n",
    "\n",
    "After indexing the documents, we can write down the query and convert the query string into query object by the query parser.\n",
    "\n",
    "Create a whoosh.qparser.QueryParser object, pass it the name of the default field to search and the schema of the index you’ll be searching.\n",
    "\n",
    "Query parser is built on modular plug-ins. For example, qparser.WildcardPlugin, which is already in the default plug-in list of parser, gives the parser the ability to search for wildcards. Some frequently used plug-ins are shown in the following code.\n",
    "\n",
    "You can use the plugins argument when creating the object to override the default list of plug-ins, use add_plugin() and/or remove_plugin_class() to change the plug-ins included in the parser.\n",
    "\n",
    "Here is the list of available plug-ins:http://whoosh.readthedocs.io/en/latest/api/qparser.html#plug-ins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining a Base_QueryParser for the field \"file_content\" in the index\n",
    "Base_QueryParser = QueryParser(\"file_content\", schema=Base_Index.schema)\n",
    "## creating a Base_Searcher for query seraching in file_contents\n",
    "Base_Searcher = Base_Index.searcher()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 mining gold silver coal\n",
      "2 juvenile delinquency\n",
      "4 wireless communications\n",
      "6 physical therapists\n",
      "7 cotton industry\n",
      "9 genealogy searches\n",
      "10 Physical Fitness\n",
      "14 Agricultural biotechnology\n",
      "16 Emergency and disaster preparedness assistance\n",
      "18 Shipwrecks\n",
      "19 Cybercrime, internet fraud, and cyber fraud\n",
      "22 Veteran's Benefits\n",
      "24 Air Bag Safety\n",
      "26 Nuclear power plants\n",
      "28 Early Childhood Education\n"
     ]
    }
   ],
   "source": [
    "#printing all 15 topics from the topic files\n",
    "with open(TOPIC_FILE, \"r\") as tf:\n",
    "    topics = tf.read().splitlines()\n",
    "for topic in topics:\n",
    "    topic_id, topic_phrase = tuple(topic.split(\" \", 1))\n",
    "    print(topic_id, topic_phrase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "INDEX_Q2 = Base_Index # Replace None with your index for Q2\n",
    "QP_Q2 = Base_QueryParser # Replace None with your query parser for Q2\n",
    "SEARCHER_Q2 = Base_Searcher # Replace None with your searcher for Q2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def trecEval(topicFile, qrelsFile, queryParser, searcher):\n",
    "    # Load topic file - a list of topics(search phrases) used for evalutation\n",
    "    with open(topicFile, \"r\") as tf:\n",
    "        topics = tf.read().splitlines()\n",
    "\n",
    "    # create an temporary output file to write the results\n",
    "    tempOutputFile = tempfile.mkstemp()[1]\n",
    "    with open(tempOutputFile, \"w\") as outputTRECFile:\n",
    "        # for each evaluated topic:\n",
    "        # parsing the query and recording the results in the tempOutputFile in TREC_EVAL format\n",
    "        for topic in topics:\n",
    "            topic_id, topic_phrase = tuple(topic.split(\" \", 1))\n",
    "            topicQuery = queryParser.parse(topic_phrase)\n",
    "            topicResults = searcher.search(topicQuery, limit=None)\n",
    "            for (docnum, result) in enumerate(topicResults):\n",
    "                score = topicResults.score(docnum)\n",
    "                outputTRECFile.write(\"%s Q0 %s %d %lf test\\n\" % (topic_id, os.path.basename(result[\"file_path\"]), docnum, score))\n",
    "#                 print((\"%s Q0 %s %d %lf test\\n\" % (topic_id, os.path.basename(result[\"file_path\"]), docnum, score)))\n",
    "    \n",
    "    result = subprocess.run([TREC_EVAL, '-q', qrelsFile, tempOutputFile], stdout=subprocess.PIPE)\n",
    "    print(result.stdout.decode())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the Retrival System:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_ret               \t1\t1\n",
      "num_rel               \t1\t5\n",
      "num_rel_ret           \t1\t0\n",
      "map                   \t1\t0.0000\n",
      "Rprec                 \t1\t0.0000\n",
      "bpref                 \t1\t0.0000\n",
      "recip_rank            \t1\t0.0000\n",
      "iprec_at_recall_0.00  \t1\t0.0000\n",
      "iprec_at_recall_0.10  \t1\t0.0000\n",
      "iprec_at_recall_0.20  \t1\t0.0000\n",
      "iprec_at_recall_0.30  \t1\t0.0000\n",
      "iprec_at_recall_0.40  \t1\t0.0000\n",
      "iprec_at_recall_0.50  \t1\t0.0000\n",
      "iprec_at_recall_0.60  \t1\t0.0000\n",
      "iprec_at_recall_0.70  \t1\t0.0000\n",
      "iprec_at_recall_0.80  \t1\t0.0000\n",
      "iprec_at_recall_0.90  \t1\t0.0000\n",
      "iprec_at_recall_1.00  \t1\t0.0000\n",
      "P_5                   \t1\t0.0000\n",
      "P_10                  \t1\t0.0000\n",
      "P_15                  \t1\t0.0000\n",
      "P_20                  \t1\t0.0000\n",
      "P_30                  \t1\t0.0000\n",
      "P_100                 \t1\t0.0000\n",
      "P_200                 \t1\t0.0000\n",
      "P_500                 \t1\t0.0000\n",
      "P_1000                \t1\t0.0000\n",
      "num_ret               \t10\t16\n",
      "num_rel               \t10\t1\n",
      "num_rel_ret           \t10\t1\n",
      "map                   \t10\t0.1667\n",
      "Rprec                 \t10\t0.0000\n",
      "bpref                 \t10\t0.0000\n",
      "recip_rank            \t10\t0.1667\n",
      "iprec_at_recall_0.00  \t10\t0.1667\n",
      "iprec_at_recall_0.10  \t10\t0.1667\n",
      "iprec_at_recall_0.20  \t10\t0.1667\n",
      "iprec_at_recall_0.30  \t10\t0.1667\n",
      "iprec_at_recall_0.40  \t10\t0.1667\n",
      "iprec_at_recall_0.50  \t10\t0.1667\n",
      "iprec_at_recall_0.60  \t10\t0.1667\n",
      "iprec_at_recall_0.70  \t10\t0.1667\n",
      "iprec_at_recall_0.80  \t10\t0.1667\n",
      "iprec_at_recall_0.90  \t10\t0.1667\n",
      "iprec_at_recall_1.00  \t10\t0.1667\n",
      "P_5                   \t10\t0.0000\n",
      "P_10                  \t10\t0.1000\n",
      "P_15                  \t10\t0.0667\n",
      "P_20                  \t10\t0.0500\n",
      "P_30                  \t10\t0.0333\n",
      "P_100                 \t10\t0.0100\n",
      "P_200                 \t10\t0.0050\n",
      "P_500                 \t10\t0.0020\n",
      "P_1000                \t10\t0.0010\n",
      "num_ret               \t14\t7\n",
      "num_rel               \t14\t1\n",
      "num_rel_ret           \t14\t1\n",
      "map                   \t14\t0.2500\n",
      "Rprec                 \t14\t0.0000\n",
      "bpref                 \t14\t0.0000\n",
      "recip_rank            \t14\t0.2500\n",
      "iprec_at_recall_0.00  \t14\t0.2500\n",
      "iprec_at_recall_0.10  \t14\t0.2500\n",
      "iprec_at_recall_0.20  \t14\t0.2500\n",
      "iprec_at_recall_0.30  \t14\t0.2500\n",
      "iprec_at_recall_0.40  \t14\t0.2500\n",
      "iprec_at_recall_0.50  \t14\t0.2500\n",
      "iprec_at_recall_0.60  \t14\t0.2500\n",
      "iprec_at_recall_0.70  \t14\t0.2500\n",
      "iprec_at_recall_0.80  \t14\t0.2500\n",
      "iprec_at_recall_0.90  \t14\t0.2500\n",
      "iprec_at_recall_1.00  \t14\t0.2500\n",
      "P_5                   \t14\t0.2000\n",
      "P_10                  \t14\t0.1000\n",
      "P_15                  \t14\t0.0667\n",
      "P_20                  \t14\t0.0500\n",
      "P_30                  \t14\t0.0333\n",
      "P_100                 \t14\t0.0100\n",
      "P_200                 \t14\t0.0050\n",
      "P_500                 \t14\t0.0020\n",
      "P_1000                \t14\t0.0010\n",
      "num_ret               \t16\t7\n",
      "num_rel               \t16\t7\n",
      "num_rel_ret           \t16\t0\n",
      "map                   \t16\t0.0000\n",
      "Rprec                 \t16\t0.0000\n",
      "bpref                 \t16\t0.0000\n",
      "recip_rank            \t16\t0.0000\n",
      "iprec_at_recall_0.00  \t16\t0.0000\n",
      "iprec_at_recall_0.10  \t16\t0.0000\n",
      "iprec_at_recall_0.20  \t16\t0.0000\n",
      "iprec_at_recall_0.30  \t16\t0.0000\n",
      "iprec_at_recall_0.40  \t16\t0.0000\n",
      "iprec_at_recall_0.50  \t16\t0.0000\n",
      "iprec_at_recall_0.60  \t16\t0.0000\n",
      "iprec_at_recall_0.70  \t16\t0.0000\n",
      "iprec_at_recall_0.80  \t16\t0.0000\n",
      "iprec_at_recall_0.90  \t16\t0.0000\n",
      "iprec_at_recall_1.00  \t16\t0.0000\n",
      "P_5                   \t16\t0.0000\n",
      "P_10                  \t16\t0.0000\n",
      "P_15                  \t16\t0.0000\n",
      "P_20                  \t16\t0.0000\n",
      "P_30                  \t16\t0.0000\n",
      "P_100                 \t16\t0.0000\n",
      "P_200                 \t16\t0.0000\n",
      "P_500                 \t16\t0.0000\n",
      "P_1000                \t16\t0.0000\n",
      "num_ret               \t18\t17\n",
      "num_rel               \t18\t1\n",
      "num_rel_ret           \t18\t1\n",
      "map                   \t18\t1.0000\n",
      "Rprec                 \t18\t1.0000\n",
      "bpref                 \t18\t1.0000\n",
      "recip_rank            \t18\t1.0000\n",
      "iprec_at_recall_0.00  \t18\t1.0000\n",
      "iprec_at_recall_0.10  \t18\t1.0000\n",
      "iprec_at_recall_0.20  \t18\t1.0000\n",
      "iprec_at_recall_0.30  \t18\t1.0000\n",
      "iprec_at_recall_0.40  \t18\t1.0000\n",
      "iprec_at_recall_0.50  \t18\t1.0000\n",
      "iprec_at_recall_0.60  \t18\t1.0000\n",
      "iprec_at_recall_0.70  \t18\t1.0000\n",
      "iprec_at_recall_0.80  \t18\t1.0000\n",
      "iprec_at_recall_0.90  \t18\t1.0000\n",
      "iprec_at_recall_1.00  \t18\t1.0000\n",
      "P_5                   \t18\t0.2000\n",
      "P_10                  \t18\t0.1000\n",
      "P_15                  \t18\t0.0667\n",
      "P_20                  \t18\t0.0500\n",
      "P_30                  \t18\t0.0333\n",
      "P_100                 \t18\t0.0100\n",
      "P_200                 \t18\t0.0050\n",
      "P_500                 \t18\t0.0020\n",
      "P_1000                \t18\t0.0010\n",
      "num_ret               \t2\t6\n",
      "num_rel               \t2\t2\n",
      "num_rel_ret           \t2\t0\n",
      "map                   \t2\t0.0000\n",
      "Rprec                 \t2\t0.0000\n",
      "bpref                 \t2\t0.0000\n",
      "recip_rank            \t2\t0.0000\n",
      "iprec_at_recall_0.00  \t2\t0.0000\n",
      "iprec_at_recall_0.10  \t2\t0.0000\n",
      "iprec_at_recall_0.20  \t2\t0.0000\n",
      "iprec_at_recall_0.30  \t2\t0.0000\n",
      "iprec_at_recall_0.40  \t2\t0.0000\n",
      "iprec_at_recall_0.50  \t2\t0.0000\n",
      "iprec_at_recall_0.60  \t2\t0.0000\n",
      "iprec_at_recall_0.70  \t2\t0.0000\n",
      "iprec_at_recall_0.80  \t2\t0.0000\n",
      "iprec_at_recall_0.90  \t2\t0.0000\n",
      "iprec_at_recall_1.00  \t2\t0.0000\n",
      "P_5                   \t2\t0.0000\n",
      "P_10                  \t2\t0.0000\n",
      "P_15                  \t2\t0.0000\n",
      "P_20                  \t2\t0.0000\n",
      "P_30                  \t2\t0.0000\n",
      "P_100                 \t2\t0.0000\n",
      "P_200                 \t2\t0.0000\n",
      "P_500                 \t2\t0.0000\n",
      "P_1000                \t2\t0.0000\n",
      "num_ret               \t22\t17\n",
      "num_rel               \t22\t1\n",
      "num_rel_ret           \t22\t1\n",
      "map                   \t22\t0.2000\n",
      "Rprec                 \t22\t0.0000\n",
      "bpref                 \t22\t0.0000\n",
      "recip_rank            \t22\t0.2000\n",
      "iprec_at_recall_0.00  \t22\t0.2000\n",
      "iprec_at_recall_0.10  \t22\t0.2000\n",
      "iprec_at_recall_0.20  \t22\t0.2000\n",
      "iprec_at_recall_0.30  \t22\t0.2000\n",
      "iprec_at_recall_0.40  \t22\t0.2000\n",
      "iprec_at_recall_0.50  \t22\t0.2000\n",
      "iprec_at_recall_0.60  \t22\t0.2000\n",
      "iprec_at_recall_0.70  \t22\t0.2000\n",
      "iprec_at_recall_0.80  \t22\t0.2000\n",
      "iprec_at_recall_0.90  \t22\t0.2000\n",
      "iprec_at_recall_1.00  \t22\t0.2000\n",
      "P_5                   \t22\t0.2000\n",
      "P_10                  \t22\t0.1000\n",
      "P_15                  \t22\t0.0667\n",
      "P_20                  \t22\t0.0500\n",
      "P_30                  \t22\t0.0333\n",
      "P_100                 \t22\t0.0100\n",
      "P_200                 \t22\t0.0050\n",
      "P_500                 \t22\t0.0020\n",
      "P_1000                \t22\t0.0010\n",
      "num_ret               \t24\t7\n",
      "num_rel               \t24\t1\n",
      "num_rel_ret           \t24\t1\n",
      "map                   \t24\t1.0000\n",
      "Rprec                 \t24\t1.0000\n",
      "bpref                 \t24\t1.0000\n",
      "recip_rank            \t24\t1.0000\n",
      "iprec_at_recall_0.00  \t24\t1.0000\n",
      "iprec_at_recall_0.10  \t24\t1.0000\n",
      "iprec_at_recall_0.20  \t24\t1.0000\n",
      "iprec_at_recall_0.30  \t24\t1.0000\n",
      "iprec_at_recall_0.40  \t24\t1.0000\n",
      "iprec_at_recall_0.50  \t24\t1.0000\n",
      "iprec_at_recall_0.60  \t24\t1.0000\n",
      "iprec_at_recall_0.70  \t24\t1.0000\n",
      "iprec_at_recall_0.80  \t24\t1.0000\n",
      "iprec_at_recall_0.90  \t24\t1.0000\n",
      "iprec_at_recall_1.00  \t24\t1.0000\n",
      "P_5                   \t24\t0.2000\n",
      "P_10                  \t24\t0.1000\n",
      "P_15                  \t24\t0.0667\n",
      "P_20                  \t24\t0.0500\n",
      "P_30                  \t24\t0.0333\n",
      "P_100                 \t24\t0.0100\n",
      "P_200                 \t24\t0.0050\n",
      "P_500                 \t24\t0.0020\n",
      "P_1000                \t24\t0.0010\n",
      "num_ret               \t26\t25\n",
      "num_rel               \t26\t3\n",
      "num_rel_ret           \t26\t1\n",
      "map                   \t26\t0.1111\n",
      "Rprec                 \t26\t0.3333\n",
      "bpref                 \t26\t0.1111\n",
      "recip_rank            \t26\t0.3333\n",
      "iprec_at_recall_0.00  \t26\t0.3333\n",
      "iprec_at_recall_0.10  \t26\t0.3333\n",
      "iprec_at_recall_0.20  \t26\t0.3333\n",
      "iprec_at_recall_0.30  \t26\t0.3333\n",
      "iprec_at_recall_0.40  \t26\t0.0000\n",
      "iprec_at_recall_0.50  \t26\t0.0000\n",
      "iprec_at_recall_0.60  \t26\t0.0000\n",
      "iprec_at_recall_0.70  \t26\t0.0000\n",
      "iprec_at_recall_0.80  \t26\t0.0000\n",
      "iprec_at_recall_0.90  \t26\t0.0000\n",
      "iprec_at_recall_1.00  \t26\t0.0000\n",
      "P_5                   \t26\t0.2000\n",
      "P_10                  \t26\t0.1000\n",
      "P_15                  \t26\t0.0667\n",
      "P_20                  \t26\t0.0500\n",
      "P_30                  \t26\t0.0333\n",
      "P_100                 \t26\t0.0100\n",
      "P_200                 \t26\t0.0050\n",
      "P_500                 \t26\t0.0020\n",
      "P_1000                \t26\t0.0010\n",
      "num_ret               \t28\t22\n",
      "num_rel               \t28\t2\n",
      "num_rel_ret           \t28\t0\n",
      "map                   \t28\t0.0000\n",
      "Rprec                 \t28\t0.0000\n",
      "bpref                 \t28\t0.0000\n",
      "recip_rank            \t28\t0.0000\n",
      "iprec_at_recall_0.00  \t28\t0.0000\n",
      "iprec_at_recall_0.10  \t28\t0.0000\n",
      "iprec_at_recall_0.20  \t28\t0.0000\n",
      "iprec_at_recall_0.30  \t28\t0.0000\n",
      "iprec_at_recall_0.40  \t28\t0.0000\n",
      "iprec_at_recall_0.50  \t28\t0.0000\n",
      "iprec_at_recall_0.60  \t28\t0.0000\n",
      "iprec_at_recall_0.70  \t28\t0.0000\n",
      "iprec_at_recall_0.80  \t28\t0.0000\n",
      "iprec_at_recall_0.90  \t28\t0.0000\n",
      "iprec_at_recall_1.00  \t28\t0.0000\n",
      "P_5                   \t28\t0.0000\n",
      "P_10                  \t28\t0.0000\n",
      "P_15                  \t28\t0.0000\n",
      "P_20                  \t28\t0.0000\n",
      "P_30                  \t28\t0.0000\n",
      "P_100                 \t28\t0.0000\n",
      "P_200                 \t28\t0.0000\n",
      "P_500                 \t28\t0.0000\n",
      "P_1000                \t28\t0.0000\n",
      "num_ret               \t4\t12\n",
      "num_rel               \t4\t4\n",
      "num_rel_ret           \t4\t1\n",
      "map                   \t4\t0.0312\n",
      "Rprec                 \t4\t0.0000\n",
      "bpref                 \t4\t0.0000\n",
      "recip_rank            \t4\t0.1250\n",
      "iprec_at_recall_0.00  \t4\t0.1250\n",
      "iprec_at_recall_0.10  \t4\t0.1250\n",
      "iprec_at_recall_0.20  \t4\t0.1250\n",
      "iprec_at_recall_0.30  \t4\t0.0000\n",
      "iprec_at_recall_0.40  \t4\t0.0000\n",
      "iprec_at_recall_0.50  \t4\t0.0000\n",
      "iprec_at_recall_0.60  \t4\t0.0000\n",
      "iprec_at_recall_0.70  \t4\t0.0000\n",
      "iprec_at_recall_0.80  \t4\t0.0000\n",
      "iprec_at_recall_0.90  \t4\t0.0000\n",
      "iprec_at_recall_1.00  \t4\t0.0000\n",
      "P_5                   \t4\t0.0000\n",
      "P_10                  \t4\t0.1000\n",
      "P_15                  \t4\t0.0667\n",
      "P_20                  \t4\t0.0500\n",
      "P_30                  \t4\t0.0333\n",
      "P_100                 \t4\t0.0100\n",
      "P_200                 \t4\t0.0050\n",
      "P_500                 \t4\t0.0020\n",
      "P_1000                \t4\t0.0010\n",
      "num_ret               \t6\t5\n",
      "num_rel               \t6\t1\n",
      "num_rel_ret           \t6\t0\n",
      "map                   \t6\t0.0000\n",
      "Rprec                 \t6\t0.0000\n",
      "bpref                 \t6\t0.0000\n",
      "recip_rank            \t6\t0.0000\n",
      "iprec_at_recall_0.00  \t6\t0.0000\n",
      "iprec_at_recall_0.10  \t6\t0.0000\n",
      "iprec_at_recall_0.20  \t6\t0.0000\n",
      "iprec_at_recall_0.30  \t6\t0.0000\n",
      "iprec_at_recall_0.40  \t6\t0.0000\n",
      "iprec_at_recall_0.50  \t6\t0.0000\n",
      "iprec_at_recall_0.60  \t6\t0.0000\n",
      "iprec_at_recall_0.70  \t6\t0.0000\n",
      "iprec_at_recall_0.80  \t6\t0.0000\n",
      "iprec_at_recall_0.90  \t6\t0.0000\n",
      "iprec_at_recall_1.00  \t6\t0.0000\n",
      "P_5                   \t6\t0.0000\n",
      "P_10                  \t6\t0.0000\n",
      "P_15                  \t6\t0.0000\n",
      "P_20                  \t6\t0.0000\n",
      "P_30                  \t6\t0.0000\n",
      "P_100                 \t6\t0.0000\n",
      "P_200                 \t6\t0.0000\n",
      "P_500                 \t6\t0.0000\n",
      "P_1000                \t6\t0.0000\n",
      "num_ret               \t7\t7\n",
      "num_rel               \t7\t3\n",
      "num_rel_ret           \t7\t0\n",
      "map                   \t7\t0.0000\n",
      "Rprec                 \t7\t0.0000\n",
      "bpref                 \t7\t0.0000\n",
      "recip_rank            \t7\t0.0000\n",
      "iprec_at_recall_0.00  \t7\t0.0000\n",
      "iprec_at_recall_0.10  \t7\t0.0000\n",
      "iprec_at_recall_0.20  \t7\t0.0000\n",
      "iprec_at_recall_0.30  \t7\t0.0000\n",
      "iprec_at_recall_0.40  \t7\t0.0000\n",
      "iprec_at_recall_0.50  \t7\t0.0000\n",
      "iprec_at_recall_0.60  \t7\t0.0000\n",
      "iprec_at_recall_0.70  \t7\t0.0000\n",
      "iprec_at_recall_0.80  \t7\t0.0000\n",
      "iprec_at_recall_0.90  \t7\t0.0000\n",
      "iprec_at_recall_1.00  \t7\t0.0000\n",
      "P_5                   \t7\t0.0000\n",
      "P_10                  \t7\t0.0000\n",
      "P_15                  \t7\t0.0000\n",
      "P_20                  \t7\t0.0000\n",
      "P_30                  \t7\t0.0000\n",
      "P_100                 \t7\t0.0000\n",
      "P_200                 \t7\t0.0000\n",
      "P_500                 \t7\t0.0000\n",
      "P_1000                \t7\t0.0000\n",
      "num_ret               \t9\t2\n",
      "num_rel               \t9\t1\n",
      "num_rel_ret           \t9\t0\n",
      "map                   \t9\t0.0000\n",
      "Rprec                 \t9\t0.0000\n",
      "bpref                 \t9\t0.0000\n",
      "recip_rank            \t9\t0.0000\n",
      "iprec_at_recall_0.00  \t9\t0.0000\n",
      "iprec_at_recall_0.10  \t9\t0.0000\n",
      "iprec_at_recall_0.20  \t9\t0.0000\n",
      "iprec_at_recall_0.30  \t9\t0.0000\n",
      "iprec_at_recall_0.40  \t9\t0.0000\n",
      "iprec_at_recall_0.50  \t9\t0.0000\n",
      "iprec_at_recall_0.60  \t9\t0.0000\n",
      "iprec_at_recall_0.70  \t9\t0.0000\n",
      "iprec_at_recall_0.80  \t9\t0.0000\n",
      "iprec_at_recall_0.90  \t9\t0.0000\n",
      "iprec_at_recall_1.00  \t9\t0.0000\n",
      "P_5                   \t9\t0.0000\n",
      "P_10                  \t9\t0.0000\n",
      "P_15                  \t9\t0.0000\n",
      "P_20                  \t9\t0.0000\n",
      "P_30                  \t9\t0.0000\n",
      "P_100                 \t9\t0.0000\n",
      "P_200                 \t9\t0.0000\n",
      "P_500                 \t9\t0.0000\n",
      "P_1000                \t9\t0.0000\n",
      "runid                 \tall\ttest\n",
      "num_q                 \tall\t14\n",
      "num_ret               \tall\t151\n",
      "num_rel               \tall\t33\n",
      "num_rel_ret           \tall\t7\n",
      "map                   \tall\t0.1971\n",
      "gm_map                \tall\t0.0015\n",
      "Rprec                 \tall\t0.1667\n",
      "bpref                 \tall\t0.1508\n",
      "recip_rank            \tall\t0.2196\n",
      "iprec_at_recall_0.00  \tall\t0.2196\n",
      "iprec_at_recall_0.10  \tall\t0.2196\n",
      "iprec_at_recall_0.20  \tall\t0.2196\n",
      "iprec_at_recall_0.30  \tall\t0.2107\n",
      "iprec_at_recall_0.40  \tall\t0.1869\n",
      "iprec_at_recall_0.50  \tall\t0.1869\n",
      "iprec_at_recall_0.60  \tall\t0.1869\n",
      "iprec_at_recall_0.70  \tall\t0.1869\n",
      "iprec_at_recall_0.80  \tall\t0.1869\n",
      "iprec_at_recall_0.90  \tall\t0.1869\n",
      "iprec_at_recall_1.00  \tall\t0.1869\n",
      "P_5                   \tall\t0.0714\n",
      "P_10                  \tall\t0.0500\n",
      "P_15                  \tall\t0.0333\n",
      "P_20                  \tall\t0.0250\n",
      "P_30                  \tall\t0.0167\n",
      "P_100                 \tall\t0.0050\n",
      "P_200                 \tall\t0.0025\n",
      "P_500                 \tall\t0.0010\n",
      "P_1000                \tall\t0.0005\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#evaluatons of the returned results using TREC_eval\n",
    "trecEval(TOPIC_FILE, QRELS_FILE, QP_Q2, SEARCHER_Q2) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results for Base system:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "P5 (Precision after 5 docs retrieved) for the baseline Whoosh Query system is: 0.0714"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Performed Well:  \n",
    "For topic ID's: [14, 18, 22, 24, 26] it performed well because it retrived relevant documents in the first 5 results even with the baseline system.\n",
    "\n",
    "#### Performed Bad:  \n",
    "For topic ID's: [19] it performed very badly because it didnt even retrive any document. Also, for remanining topic Id's  [1,10,16,2,28,4,6,7,9] it performed badly beacuse there were no relevant docs in the first five results.\n",
    "\n",
    "#### Overall Perfomance:  \n",
    "Overall only for 5 out of 15 queries we got required relevant information in the first five retrived documents for each topic which shows that there is so much scope for improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to Improve the Search perfomance:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To improve the search, first try to identify cases where our system is performing well and where it is not performing - By idetifying particular documents like these we can decide on what changes and improvements need to be made."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Query number 2 (\"juvenile delinquency\") in the given quries list - Although our system retrived 6 documents all of them were False positives. We have 2 relevant docs in the given collections for this query, but they were false negatives in our base whoosh retrival system.\n",
    "\n",
    "#### False Positive:\n",
    "\n",
    "Document no: Q0 G00-22-3396139 is retrived and highly ranked because it having both \"juvenile\" AND \"delinquency\" terms once occured in the document.But if we check the relevant documents for this query one of them (G00-08-1145623) has only \"delinquency\" term occuring multiple times but no \"juvenile\" term. Hence instead of \"AND\" in the query parser using \"OR\" might improve the results.\n",
    "\n",
    "#### False negative:\n",
    "\n",
    "Document no: G00-37-1427392 is not retrived (False Negative) because the words \"Juvenile\" are in the upper case format and also word \"Delinquency\" is present as \"delinquent\". So by converting everything to lower case and also by using a stemmer to shorten the words we can make these kind of documents True Positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# docs with 'juvenile' 19\n",
      "# docs with 'Juvenile' 42\n",
      "# docs with 'delinquency' 6\n",
      "# docs with 'Delinquency' 12\n",
      "# docs with 'Delinquent' 7\n"
     ]
    }
   ],
   "source": [
    "# defining a reader object on the index\n",
    "myReader = Base_Index.reader()\n",
    "print(\"# docs with 'juvenile'\", myReader.doc_frequency(\"file_content\", \"juvenile\"))\n",
    "print(\"# docs with 'Juvenile'\", myReader.doc_frequency(\"file_content\", \"Juvenile\"))\n",
    "print(\"# docs with 'delinquency'\", myReader.doc_frequency(\"file_content\", \"delinquency\"))\n",
    "print(\"# docs with 'Delinquency'\", myReader.doc_frequency(\"file_content\", \"Delinquency\"))\n",
    "print(\"# docs with 'Delinquent'\", myReader.doc_frequency(\"file_content\", \"delinquent\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        delinqu         delinqu     delinquency     delinquency\n",
      "        delinqu         delinqu      Delinquent      Delinquent\n",
      "            saf          safeti          safety          safety\n",
      "            saf            safe            safe            safe\n"
     ]
    }
   ],
   "source": [
    "## choosing from different types of Stemmers and lemmatizers vailable: \n",
    "## We can choose any of the stemming & Lemmatizers based on the system we are working\n",
    "listWords = [\"delinquency\", \"Delinquent\", \"safety\", \"safe\"]\n",
    "for word in listWords:\n",
    "    print(\"%15s %15s %15s %15s\" % (LancasterStemmer().stem(word),\n",
    "                                   SnowballStemmer(\"english\").stem(word),\n",
    "                                   WordNetLemmatizer().lemmatize(word),\n",
    "                                   WordNetLemmatizer().lemmatize(word, 'v')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suggested Improvements: lower case, Stemming, \"OR\" query parser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Using NLTK in Whoosh:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This filter will run for both the index and the query\n",
    "from whoosh.analysis import Filter\n",
    "class CustomFilter(Filter):\n",
    "    is_morph = True\n",
    "    def __init__(self, filterFunc, *args, **kwargs):\n",
    "        self.customFilter = filterFunc\n",
    "        self.args = args\n",
    "        self.kwargs = kwargs\n",
    "    def __eq__(self):\n",
    "        return (other\n",
    "                and self.__class__ is other.__class__)\n",
    "    def __call__(self, tokens):\n",
    "        for t in tokens:\n",
    "            if t.mode == 'query': # if called by query parser\n",
    "                t.text = self.customFilter(t.text, *self.args, **self.kwargs)\n",
    "                yield t\n",
    "            else: # == 'index' if called by indexer\n",
    "                t.text = self.customFilter(t.text, *self.args, **self.kwargs)\n",
    "                yield t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "Improved_analyzer = RegexTokenizer() | LowercaseFilter()  | StopFilter() | CustomFilter(LancasterStemmer().stem)\n",
    "# [token.text for token in myFilter1(\"We are going to do Text Analysis with whoosh.analysis\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining improved Schema for the index after converting to lowercase and stemming\n",
    "Improved_Schema = Schema(file_path = ID(stored=True),\n",
    "                  file_content = TEXT(analyzer=Improved_analyzer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "Improved_Index = createIndex(Improved_Schema)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "already indexed: 1000\n",
      "already indexed: 2000\n",
      "already indexed: 3000\n",
      "already indexed: 4000\n",
      "done indexing.\n"
     ]
    }
   ],
   "source": [
    "addFilesToIndex(Improved_Index, filesToIndex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "Improved_OR_QueryParser = QueryParser(\"file_content\", schema=Improved_Index.schema, group=qparser.OrGroup)\n",
    "## query seraching in file_contents\n",
    "Improved_Searcher = Improved_Index.searcher()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "INDEX_Q3 = Improved_Index # Replace None with your index for Q3\n",
    "QP_Q3 = Improved_OR_QueryParser # Replace None with your query parser for Q3\n",
    "SEARCHER_Q3 = Improved_Searcher # Replace None with your searcher for Q3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_ret               \t1\t471\n",
      "num_rel               \t1\t5\n",
      "num_rel_ret           \t1\t5\n",
      "map                   \t1\t0.0625\n",
      "Rprec                 \t1\t0.0000\n",
      "bpref                 \t1\t0.0000\n",
      "recip_rank            \t1\t0.0500\n",
      "iprec_at_recall_0.00  \t1\t0.0968\n",
      "iprec_at_recall_0.10  \t1\t0.0968\n",
      "iprec_at_recall_0.20  \t1\t0.0968\n",
      "iprec_at_recall_0.30  \t1\t0.0968\n",
      "iprec_at_recall_0.40  \t1\t0.0968\n",
      "iprec_at_recall_0.50  \t1\t0.0968\n",
      "iprec_at_recall_0.60  \t1\t0.0968\n",
      "iprec_at_recall_0.70  \t1\t0.0430\n",
      "iprec_at_recall_0.80  \t1\t0.0430\n",
      "iprec_at_recall_0.90  \t1\t0.0427\n",
      "iprec_at_recall_1.00  \t1\t0.0427\n",
      "P_5                   \t1\t0.0000\n",
      "P_10                  \t1\t0.0000\n",
      "P_15                  \t1\t0.0000\n",
      "P_20                  \t1\t0.0500\n",
      "P_30                  \t1\t0.0667\n",
      "P_100                 \t1\t0.0400\n",
      "P_200                 \t1\t0.0250\n",
      "P_500                 \t1\t0.0100\n",
      "P_1000                \t1\t0.0050\n",
      "num_ret               \t10\t498\n",
      "num_rel               \t10\t1\n",
      "num_rel_ret           \t10\t1\n",
      "map                   \t10\t0.2500\n",
      "Rprec                 \t10\t0.0000\n",
      "bpref                 \t10\t0.0000\n",
      "recip_rank            \t10\t0.2500\n",
      "iprec_at_recall_0.00  \t10\t0.2500\n",
      "iprec_at_recall_0.10  \t10\t0.2500\n",
      "iprec_at_recall_0.20  \t10\t0.2500\n",
      "iprec_at_recall_0.30  \t10\t0.2500\n",
      "iprec_at_recall_0.40  \t10\t0.2500\n",
      "iprec_at_recall_0.50  \t10\t0.2500\n",
      "iprec_at_recall_0.60  \t10\t0.2500\n",
      "iprec_at_recall_0.70  \t10\t0.2500\n",
      "iprec_at_recall_0.80  \t10\t0.2500\n",
      "iprec_at_recall_0.90  \t10\t0.2500\n",
      "iprec_at_recall_1.00  \t10\t0.2500\n",
      "P_5                   \t10\t0.2000\n",
      "P_10                  \t10\t0.1000\n",
      "P_15                  \t10\t0.0667\n",
      "P_20                  \t10\t0.0500\n",
      "P_30                  \t10\t0.0333\n",
      "P_100                 \t10\t0.0100\n",
      "P_200                 \t10\t0.0050\n",
      "P_500                 \t10\t0.0020\n",
      "P_1000                \t10\t0.0010\n",
      "num_ret               \t14\t308\n",
      "num_rel               \t14\t1\n",
      "num_rel_ret           \t14\t1\n",
      "map                   \t14\t1.0000\n",
      "Rprec                 \t14\t1.0000\n",
      "bpref                 \t14\t1.0000\n",
      "recip_rank            \t14\t1.0000\n",
      "iprec_at_recall_0.00  \t14\t1.0000\n",
      "iprec_at_recall_0.10  \t14\t1.0000\n",
      "iprec_at_recall_0.20  \t14\t1.0000\n",
      "iprec_at_recall_0.30  \t14\t1.0000\n",
      "iprec_at_recall_0.40  \t14\t1.0000\n",
      "iprec_at_recall_0.50  \t14\t1.0000\n",
      "iprec_at_recall_0.60  \t14\t1.0000\n",
      "iprec_at_recall_0.70  \t14\t1.0000\n",
      "iprec_at_recall_0.80  \t14\t1.0000\n",
      "iprec_at_recall_0.90  \t14\t1.0000\n",
      "iprec_at_recall_1.00  \t14\t1.0000\n",
      "P_5                   \t14\t0.2000\n",
      "P_10                  \t14\t0.1000\n",
      "P_15                  \t14\t0.0667\n",
      "P_20                  \t14\t0.0500\n",
      "P_30                  \t14\t0.0333\n",
      "P_100                 \t14\t0.0100\n",
      "P_200                 \t14\t0.0050\n",
      "P_500                 \t14\t0.0020\n",
      "P_1000                \t14\t0.0010\n",
      "num_ret               \t16\t1111\n",
      "num_rel               \t16\t7\n",
      "num_rel_ret           \t16\t7\n",
      "map                   \t16\t0.1558\n",
      "Rprec                 \t16\t0.0000\n",
      "bpref                 \t16\t0.0000\n",
      "recip_rank            \t16\t0.0833\n",
      "iprec_at_recall_0.00  \t16\t0.2000\n",
      "iprec_at_recall_0.10  \t16\t0.2000\n",
      "iprec_at_recall_0.20  \t16\t0.2000\n",
      "iprec_at_recall_0.30  \t16\t0.2000\n",
      "iprec_at_recall_0.40  \t16\t0.2000\n",
      "iprec_at_recall_0.50  \t16\t0.1818\n",
      "iprec_at_recall_0.60  \t16\t0.1613\n",
      "iprec_at_recall_0.70  \t16\t0.1613\n",
      "iprec_at_recall_0.80  \t16\t0.1579\n",
      "iprec_at_recall_0.90  \t16\t0.1522\n",
      "iprec_at_recall_1.00  \t16\t0.1522\n",
      "P_5                   \t16\t0.0000\n",
      "P_10                  \t16\t0.0000\n",
      "P_15                  \t16\t0.2000\n",
      "P_20                  \t16\t0.1500\n",
      "P_30                  \t16\t0.1333\n",
      "P_100                 \t16\t0.0700\n",
      "P_200                 \t16\t0.0350\n",
      "P_500                 \t16\t0.0140\n",
      "P_1000                \t16\t0.0070\n",
      "num_ret               \t18\t32\n",
      "num_rel               \t18\t1\n",
      "num_rel_ret           \t18\t1\n",
      "map                   \t18\t1.0000\n",
      "Rprec                 \t18\t1.0000\n",
      "bpref                 \t18\t1.0000\n",
      "recip_rank            \t18\t1.0000\n",
      "iprec_at_recall_0.00  \t18\t1.0000\n",
      "iprec_at_recall_0.10  \t18\t1.0000\n",
      "iprec_at_recall_0.20  \t18\t1.0000\n",
      "iprec_at_recall_0.30  \t18\t1.0000\n",
      "iprec_at_recall_0.40  \t18\t1.0000\n",
      "iprec_at_recall_0.50  \t18\t1.0000\n",
      "iprec_at_recall_0.60  \t18\t1.0000\n",
      "iprec_at_recall_0.70  \t18\t1.0000\n",
      "iprec_at_recall_0.80  \t18\t1.0000\n",
      "iprec_at_recall_0.90  \t18\t1.0000\n",
      "iprec_at_recall_1.00  \t18\t1.0000\n",
      "P_5                   \t18\t0.2000\n",
      "P_10                  \t18\t0.1000\n",
      "P_15                  \t18\t0.0667\n",
      "P_20                  \t18\t0.0500\n",
      "P_30                  \t18\t0.0333\n",
      "P_100                 \t18\t0.0100\n",
      "P_200                 \t18\t0.0050\n",
      "P_500                 \t18\t0.0020\n",
      "P_1000                \t18\t0.0010\n",
      "num_ret               \t19\t627\n",
      "num_rel               \t19\t2\n",
      "num_rel_ret           \t19\t1\n",
      "map                   \t19\t0.5000\n",
      "Rprec                 \t19\t0.5000\n",
      "bpref                 \t19\t0.5000\n",
      "recip_rank            \t19\t1.0000\n",
      "iprec_at_recall_0.00  \t19\t1.0000\n",
      "iprec_at_recall_0.10  \t19\t1.0000\n",
      "iprec_at_recall_0.20  \t19\t1.0000\n",
      "iprec_at_recall_0.30  \t19\t1.0000\n",
      "iprec_at_recall_0.40  \t19\t1.0000\n",
      "iprec_at_recall_0.50  \t19\t1.0000\n",
      "iprec_at_recall_0.60  \t19\t0.0000\n",
      "iprec_at_recall_0.70  \t19\t0.0000\n",
      "iprec_at_recall_0.80  \t19\t0.0000\n",
      "iprec_at_recall_0.90  \t19\t0.0000\n",
      "iprec_at_recall_1.00  \t19\t0.0000\n",
      "P_5                   \t19\t0.2000\n",
      "P_10                  \t19\t0.1000\n",
      "P_15                  \t19\t0.0667\n",
      "P_20                  \t19\t0.0500\n",
      "P_30                  \t19\t0.0333\n",
      "P_100                 \t19\t0.0100\n",
      "P_200                 \t19\t0.0050\n",
      "P_500                 \t19\t0.0020\n",
      "P_1000                \t19\t0.0010\n",
      "num_ret               \t2\t59\n",
      "num_rel               \t2\t2\n",
      "num_rel_ret           \t2\t2\n",
      "map                   \t2\t0.5357\n",
      "Rprec                 \t2\t0.5000\n",
      "bpref                 \t2\t0.5000\n",
      "recip_rank            \t2\t1.0000\n",
      "iprec_at_recall_0.00  \t2\t1.0000\n",
      "iprec_at_recall_0.10  \t2\t1.0000\n",
      "iprec_at_recall_0.20  \t2\t1.0000\n",
      "iprec_at_recall_0.30  \t2\t1.0000\n",
      "iprec_at_recall_0.40  \t2\t1.0000\n",
      "iprec_at_recall_0.50  \t2\t1.0000\n",
      "iprec_at_recall_0.60  \t2\t0.0714\n",
      "iprec_at_recall_0.70  \t2\t0.0714\n",
      "iprec_at_recall_0.80  \t2\t0.0714\n",
      "iprec_at_recall_0.90  \t2\t0.0714\n",
      "iprec_at_recall_1.00  \t2\t0.0714\n",
      "P_5                   \t2\t0.2000\n",
      "P_10                  \t2\t0.1000\n",
      "P_15                  \t2\t0.0667\n",
      "P_20                  \t2\t0.0500\n",
      "P_30                  \t2\t0.0667\n",
      "P_100                 \t2\t0.0200\n",
      "P_200                 \t2\t0.0100\n",
      "P_500                 \t2\t0.0040\n",
      "P_1000                \t2\t0.0020\n",
      "num_ret               \t22\t467\n",
      "num_rel               \t22\t1\n",
      "num_rel_ret           \t22\t1\n",
      "map                   \t22\t0.0357\n",
      "Rprec                 \t22\t0.0000\n",
      "bpref                 \t22\t0.0000\n",
      "recip_rank            \t22\t0.0357\n",
      "iprec_at_recall_0.00  \t22\t0.0357\n",
      "iprec_at_recall_0.10  \t22\t0.0357\n",
      "iprec_at_recall_0.20  \t22\t0.0357\n",
      "iprec_at_recall_0.30  \t22\t0.0357\n",
      "iprec_at_recall_0.40  \t22\t0.0357\n",
      "iprec_at_recall_0.50  \t22\t0.0357\n",
      "iprec_at_recall_0.60  \t22\t0.0357\n",
      "iprec_at_recall_0.70  \t22\t0.0357\n",
      "iprec_at_recall_0.80  \t22\t0.0357\n",
      "iprec_at_recall_0.90  \t22\t0.0357\n",
      "iprec_at_recall_1.00  \t22\t0.0357\n",
      "P_5                   \t22\t0.0000\n",
      "P_10                  \t22\t0.0000\n",
      "P_15                  \t22\t0.0000\n",
      "P_20                  \t22\t0.0000\n",
      "P_30                  \t22\t0.0333\n",
      "P_100                 \t22\t0.0100\n",
      "P_200                 \t22\t0.0050\n",
      "P_500                 \t22\t0.0020\n",
      "P_1000                \t22\t0.0010\n",
      "num_ret               \t24\t770\n",
      "num_rel               \t24\t1\n",
      "num_rel_ret           \t24\t1\n",
      "map                   \t24\t1.0000\n",
      "Rprec                 \t24\t1.0000\n",
      "bpref                 \t24\t1.0000\n",
      "recip_rank            \t24\t1.0000\n",
      "iprec_at_recall_0.00  \t24\t1.0000\n",
      "iprec_at_recall_0.10  \t24\t1.0000\n",
      "iprec_at_recall_0.20  \t24\t1.0000\n",
      "iprec_at_recall_0.30  \t24\t1.0000\n",
      "iprec_at_recall_0.40  \t24\t1.0000\n",
      "iprec_at_recall_0.50  \t24\t1.0000\n",
      "iprec_at_recall_0.60  \t24\t1.0000\n",
      "iprec_at_recall_0.70  \t24\t1.0000\n",
      "iprec_at_recall_0.80  \t24\t1.0000\n",
      "iprec_at_recall_0.90  \t24\t1.0000\n",
      "iprec_at_recall_1.00  \t24\t1.0000\n",
      "P_5                   \t24\t0.2000\n",
      "P_10                  \t24\t0.1000\n",
      "P_15                  \t24\t0.0667\n",
      "P_20                  \t24\t0.0500\n",
      "P_30                  \t24\t0.0333\n",
      "P_100                 \t24\t0.0100\n",
      "P_200                 \t24\t0.0050\n",
      "P_500                 \t24\t0.0020\n",
      "P_1000                \t24\t0.0010\n",
      "num_ret               \t26\t584\n",
      "num_rel               \t26\t3\n",
      "num_rel_ret           \t26\t3\n",
      "map                   \t26\t0.1074\n",
      "Rprec                 \t26\t0.0000\n",
      "bpref                 \t26\t0.0000\n",
      "recip_rank            \t26\t0.1667\n",
      "iprec_at_recall_0.00  \t26\t0.1667\n",
      "iprec_at_recall_0.10  \t26\t0.1667\n",
      "iprec_at_recall_0.20  \t26\t0.1667\n",
      "iprec_at_recall_0.30  \t26\t0.1667\n",
      "iprec_at_recall_0.40  \t26\t0.0909\n",
      "iprec_at_recall_0.50  \t26\t0.0909\n",
      "iprec_at_recall_0.60  \t26\t0.0909\n",
      "iprec_at_recall_0.70  \t26\t0.0909\n",
      "iprec_at_recall_0.80  \t26\t0.0909\n",
      "iprec_at_recall_0.90  \t26\t0.0909\n",
      "iprec_at_recall_1.00  \t26\t0.0909\n",
      "P_5                   \t26\t0.0000\n",
      "P_10                  \t26\t0.1000\n",
      "P_15                  \t26\t0.0667\n",
      "P_20                  \t26\t0.0500\n",
      "P_30                  \t26\t0.0333\n",
      "P_100                 \t26\t0.0300\n",
      "P_200                 \t26\t0.0150\n",
      "P_500                 \t26\t0.0060\n",
      "P_1000                \t26\t0.0030\n",
      "num_ret               \t28\t1259\n",
      "num_rel               \t28\t2\n",
      "num_rel_ret           \t28\t2\n",
      "map                   \t28\t0.2262\n",
      "Rprec                 \t28\t0.0000\n",
      "bpref                 \t28\t0.0000\n",
      "recip_rank            \t28\t0.1667\n",
      "iprec_at_recall_0.00  \t28\t0.2857\n",
      "iprec_at_recall_0.10  \t28\t0.2857\n",
      "iprec_at_recall_0.20  \t28\t0.2857\n",
      "iprec_at_recall_0.30  \t28\t0.2857\n",
      "iprec_at_recall_0.40  \t28\t0.2857\n",
      "iprec_at_recall_0.50  \t28\t0.2857\n",
      "iprec_at_recall_0.60  \t28\t0.2857\n",
      "iprec_at_recall_0.70  \t28\t0.2857\n",
      "iprec_at_recall_0.80  \t28\t0.2857\n",
      "iprec_at_recall_0.90  \t28\t0.2857\n",
      "iprec_at_recall_1.00  \t28\t0.2857\n",
      "P_5                   \t28\t0.0000\n",
      "P_10                  \t28\t0.2000\n",
      "P_15                  \t28\t0.1333\n",
      "P_20                  \t28\t0.1000\n",
      "P_30                  \t28\t0.0667\n",
      "P_100                 \t28\t0.0200\n",
      "P_200                 \t28\t0.0100\n",
      "P_500                 \t28\t0.0040\n",
      "P_1000                \t28\t0.0020\n",
      "num_ret               \t4\t900\n",
      "num_rel               \t4\t4\n",
      "num_rel_ret           \t4\t4\n",
      "map                   \t4\t0.5596\n",
      "Rprec                 \t4\t0.5000\n",
      "bpref                 \t4\t0.5000\n",
      "recip_rank            \t4\t1.0000\n",
      "iprec_at_recall_0.00  \t4\t1.0000\n",
      "iprec_at_recall_0.10  \t4\t1.0000\n",
      "iprec_at_recall_0.20  \t4\t1.0000\n",
      "iprec_at_recall_0.30  \t4\t1.0000\n",
      "iprec_at_recall_0.40  \t4\t1.0000\n",
      "iprec_at_recall_0.50  \t4\t1.0000\n",
      "iprec_at_recall_0.60  \t4\t0.1304\n",
      "iprec_at_recall_0.70  \t4\t0.1304\n",
      "iprec_at_recall_0.80  \t4\t0.1081\n",
      "iprec_at_recall_0.90  \t4\t0.1081\n",
      "iprec_at_recall_1.00  \t4\t0.1081\n",
      "P_5                   \t4\t0.4000\n",
      "P_10                  \t4\t0.2000\n",
      "P_15                  \t4\t0.1333\n",
      "P_20                  \t4\t0.1000\n",
      "P_30                  \t4\t0.1000\n",
      "P_100                 \t4\t0.0400\n",
      "P_200                 \t4\t0.0200\n",
      "P_500                 \t4\t0.0080\n",
      "P_1000                \t4\t0.0040\n",
      "num_ret               \t6\t449\n",
      "num_rel               \t6\t1\n",
      "num_rel_ret           \t6\t1\n",
      "map                   \t6\t0.0270\n",
      "Rprec                 \t6\t0.0000\n",
      "bpref                 \t6\t0.0000\n",
      "recip_rank            \t6\t0.0270\n",
      "iprec_at_recall_0.00  \t6\t0.0270\n",
      "iprec_at_recall_0.10  \t6\t0.0270\n",
      "iprec_at_recall_0.20  \t6\t0.0270\n",
      "iprec_at_recall_0.30  \t6\t0.0270\n",
      "iprec_at_recall_0.40  \t6\t0.0270\n",
      "iprec_at_recall_0.50  \t6\t0.0270\n",
      "iprec_at_recall_0.60  \t6\t0.0270\n",
      "iprec_at_recall_0.70  \t6\t0.0270\n",
      "iprec_at_recall_0.80  \t6\t0.0270\n",
      "iprec_at_recall_0.90  \t6\t0.0270\n",
      "iprec_at_recall_1.00  \t6\t0.0270\n",
      "P_5                   \t6\t0.0000\n",
      "P_10                  \t6\t0.0000\n",
      "P_15                  \t6\t0.0000\n",
      "P_20                  \t6\t0.0000\n",
      "P_30                  \t6\t0.0000\n",
      "P_100                 \t6\t0.0100\n",
      "P_200                 \t6\t0.0050\n",
      "P_500                 \t6\t0.0020\n",
      "P_1000                \t6\t0.0010\n",
      "num_ret               \t7\t416\n",
      "num_rel               \t7\t3\n",
      "num_rel_ret           \t7\t2\n",
      "map                   \t7\t0.1778\n",
      "Rprec                 \t7\t0.0000\n",
      "bpref                 \t7\t0.0000\n",
      "recip_rank            \t7\t0.2000\n",
      "iprec_at_recall_0.00  \t7\t0.3333\n",
      "iprec_at_recall_0.10  \t7\t0.3333\n",
      "iprec_at_recall_0.20  \t7\t0.3333\n",
      "iprec_at_recall_0.30  \t7\t0.3333\n",
      "iprec_at_recall_0.40  \t7\t0.3333\n",
      "iprec_at_recall_0.50  \t7\t0.3333\n",
      "iprec_at_recall_0.60  \t7\t0.3333\n",
      "iprec_at_recall_0.70  \t7\t0.3333\n",
      "iprec_at_recall_0.80  \t7\t0.0000\n",
      "iprec_at_recall_0.90  \t7\t0.0000\n",
      "iprec_at_recall_1.00  \t7\t0.0000\n",
      "P_5                   \t7\t0.2000\n",
      "P_10                  \t7\t0.2000\n",
      "P_15                  \t7\t0.1333\n",
      "P_20                  \t7\t0.1000\n",
      "P_30                  \t7\t0.0667\n",
      "P_100                 \t7\t0.0200\n",
      "P_200                 \t7\t0.0100\n",
      "P_500                 \t7\t0.0040\n",
      "P_1000                \t7\t0.0020\n",
      "num_ret               \t9\t1825\n",
      "num_rel               \t9\t1\n",
      "num_rel_ret           \t9\t1\n",
      "map                   \t9\t0.0556\n",
      "Rprec                 \t9\t0.0000\n",
      "bpref                 \t9\t0.0000\n",
      "recip_rank            \t9\t0.0556\n",
      "iprec_at_recall_0.00  \t9\t0.0556\n",
      "iprec_at_recall_0.10  \t9\t0.0556\n",
      "iprec_at_recall_0.20  \t9\t0.0556\n",
      "iprec_at_recall_0.30  \t9\t0.0556\n",
      "iprec_at_recall_0.40  \t9\t0.0556\n",
      "iprec_at_recall_0.50  \t9\t0.0556\n",
      "iprec_at_recall_0.60  \t9\t0.0556\n",
      "iprec_at_recall_0.70  \t9\t0.0556\n",
      "iprec_at_recall_0.80  \t9\t0.0556\n",
      "iprec_at_recall_0.90  \t9\t0.0556\n",
      "iprec_at_recall_1.00  \t9\t0.0556\n",
      "P_5                   \t9\t0.0000\n",
      "P_10                  \t9\t0.0000\n",
      "P_15                  \t9\t0.0000\n",
      "P_20                  \t9\t0.0500\n",
      "P_30                  \t9\t0.0333\n",
      "P_100                 \t9\t0.0100\n",
      "P_200                 \t9\t0.0050\n",
      "P_500                 \t9\t0.0020\n",
      "P_1000                \t9\t0.0010\n",
      "runid                 \tall\ttest\n",
      "num_q                 \tall\t15\n",
      "num_ret               \tall\t9776\n",
      "num_rel               \tall\t35\n",
      "num_rel_ret           \tall\t33\n",
      "map                   \tall\t0.3795\n",
      "gm_map                \tall\t0.2130\n",
      "Rprec                 \tall\t0.3000\n",
      "bpref                 \tall\t0.3000\n",
      "recip_rank            \tall\t0.4690\n",
      "iprec_at_recall_0.00  \tall\t0.4967\n",
      "iprec_at_recall_0.10  \tall\t0.4967\n",
      "iprec_at_recall_0.20  \tall\t0.4967\n",
      "iprec_at_recall_0.30  \tall\t0.4967\n",
      "iprec_at_recall_0.40  \tall\t0.4917\n",
      "iprec_at_recall_0.50  \tall\t0.4905\n",
      "iprec_at_recall_0.60  \tall\t0.3025\n",
      "iprec_at_recall_0.70  \tall\t0.2990\n",
      "iprec_at_recall_0.80  \tall\t0.2750\n",
      "iprec_at_recall_0.90  \tall\t0.2746\n",
      "iprec_at_recall_1.00  \tall\t0.2746\n",
      "P_5                   \tall\t0.1200\n",
      "P_10                  \tall\t0.0867\n",
      "P_15                  \tall\t0.0711\n",
      "P_20                  \tall\t0.0600\n",
      "P_30                  \tall\t0.0511\n",
      "P_100                 \tall\t0.0213\n",
      "P_200                 \tall\t0.0110\n",
      "P_500                 \tall\t0.0044\n",
      "P_1000                \tall\t0.0022\n",
      "\n"
     ]
    }
   ],
   "source": [
    "trecEval(TOPIC_FILE, QRELS_FILE, QP_Q3, SEARCHER_Q3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Perfomance Evaliation after the changes:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Improvements made: \n",
    "Converting everything to lower case, removing stopwords (Since they are not relevant in our case, removing them can make code more efficient for running), using LancasterStemmer for stemming the words. \n",
    "\n",
    "#### Perfomance:\n",
    "Perfomance improved overall because for 8 out of 15 queries we are getting required relevant information in the first five retrived documents. But queries 22 (\"Veteran's Benefits\") and 26 (\"Nuclear power plants\") which retrived relevant documents in first five results in Base whoosh system are not retriving now  [since the query terms are very general in nature and because we are using \"OR\" query parser and we also did stemming they are retriving the documents with the words with more term frequencies]\n",
    "for query 2: we are getting the relevant documents in first five documents now. (False negative became True Positive) and (False positive is ranked lower now)\n",
    "\n",
    "(Overall p5 measure improved from  0.0714 to 0.12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall after making modifications, for 8 out of 15 queries we are getting required relevant information in the first five retrived documents for each topic which was only 5 before making modifications. This means that making the given modifications is a good idea for this case, but still there is scope for improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring Techniques:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How will different scoring techniques affect the perfomance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Query number 28 (\"Early Childhood Education\") in the given queries list - Although our system retrived both the 2 relevant documents in the results but none of them were ranked in first five results. \n",
    "\n",
    "#### False Positive:\n",
    "\n",
    "Document no: G00-77-3295130  is retrived and highly ranked because it is relatively shorter document (with relatively more or equal matching words) when compared with other two relevant documents. Generally as document size increases overall score decreases at higher values of b (considering similar number of matching words). As default b value is 0.75 in the BM25 scoring system, may be lowering the b values might reduce the impact of document size on the score.\n",
    "\n",
    "\n",
    "#### False negative:\n",
    "\n",
    "Document no: Although document G00-54-2576117 is retrived, it is not highly ranked because the size of the document is relatively larger compared to others. In the BM25 scoring equation at higher values of b and slightly lower values of K1, Document size greatly impacts the score. Hence reducing the impact of document size might increase the score of this document making it True Positive.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first, define a Schema for the index\n",
    "Improved_Schema = Schema(file_path = ID(stored=True),\n",
    "                  file_content = TEXT(analyzer=Improved_analyzer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "Improved_Index = createIndex(Improved_Schema)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "already indexed: 1000\n",
      "already indexed: 2000\n",
      "already indexed: 3000\n",
      "already indexed: 4000\n",
      "done indexing.\n"
     ]
    }
   ],
   "source": [
    "addFilesToIndex(Improved_Index, filesToIndex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "Improved_OR_QueryParser = QueryParser(\"file_content\", schema=Improved_Index.schema, group=qparser.OrGroup)\n",
    "## Tuning the BM25 parameters as discussed above\n",
    "Tuned_Searcher = Improved_Index.searcher(weighting=scoring.BM25F(B=0.3, K1=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "INDEX_Q4 = Improved_Index # Replace None with your index for Q4\n",
    "QP_Q4 = Improved_OR_QueryParser # Replace None with your query parser for Q4\n",
    "SEARCHER_Q4 = Tuned_Searcher # Replace None with your searcher for Q4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_ret               \t1\t471\n",
      "num_rel               \t1\t5\n",
      "num_rel_ret           \t1\t5\n",
      "map                   \t1\t0.0611\n",
      "Rprec                 \t1\t0.0000\n",
      "bpref                 \t1\t0.0000\n",
      "recip_rank            \t1\t0.0556\n",
      "iprec_at_recall_0.00  \t1\t0.0909\n",
      "iprec_at_recall_0.10  \t1\t0.0909\n",
      "iprec_at_recall_0.20  \t1\t0.0909\n",
      "iprec_at_recall_0.30  \t1\t0.0909\n",
      "iprec_at_recall_0.40  \t1\t0.0909\n",
      "iprec_at_recall_0.50  \t1\t0.0909\n",
      "iprec_at_recall_0.60  \t1\t0.0909\n",
      "iprec_at_recall_0.70  \t1\t0.0494\n",
      "iprec_at_recall_0.80  \t1\t0.0494\n",
      "iprec_at_recall_0.90  \t1\t0.0385\n",
      "iprec_at_recall_1.00  \t1\t0.0385\n",
      "P_5                   \t1\t0.0000\n",
      "P_10                  \t1\t0.0000\n",
      "P_15                  \t1\t0.0000\n",
      "P_20                  \t1\t0.0500\n",
      "P_30                  \t1\t0.0667\n",
      "P_100                 \t1\t0.0400\n",
      "P_200                 \t1\t0.0250\n",
      "P_500                 \t1\t0.0100\n",
      "P_1000                \t1\t0.0050\n",
      "num_ret               \t10\t498\n",
      "num_rel               \t10\t1\n",
      "num_rel_ret           \t10\t1\n",
      "map                   \t10\t0.2500\n",
      "Rprec                 \t10\t0.0000\n",
      "bpref                 \t10\t0.0000\n",
      "recip_rank            \t10\t0.2500\n",
      "iprec_at_recall_0.00  \t10\t0.2500\n",
      "iprec_at_recall_0.10  \t10\t0.2500\n",
      "iprec_at_recall_0.20  \t10\t0.2500\n",
      "iprec_at_recall_0.30  \t10\t0.2500\n",
      "iprec_at_recall_0.40  \t10\t0.2500\n",
      "iprec_at_recall_0.50  \t10\t0.2500\n",
      "iprec_at_recall_0.60  \t10\t0.2500\n",
      "iprec_at_recall_0.70  \t10\t0.2500\n",
      "iprec_at_recall_0.80  \t10\t0.2500\n",
      "iprec_at_recall_0.90  \t10\t0.2500\n",
      "iprec_at_recall_1.00  \t10\t0.2500\n",
      "P_5                   \t10\t0.2000\n",
      "P_10                  \t10\t0.1000\n",
      "P_15                  \t10\t0.0667\n",
      "P_20                  \t10\t0.0500\n",
      "P_30                  \t10\t0.0333\n",
      "P_100                 \t10\t0.0100\n",
      "P_200                 \t10\t0.0050\n",
      "P_500                 \t10\t0.0020\n",
      "P_1000                \t10\t0.0010\n",
      "num_ret               \t14\t308\n",
      "num_rel               \t14\t1\n",
      "num_rel_ret           \t14\t1\n",
      "map                   \t14\t1.0000\n",
      "Rprec                 \t14\t1.0000\n",
      "bpref                 \t14\t1.0000\n",
      "recip_rank            \t14\t1.0000\n",
      "iprec_at_recall_0.00  \t14\t1.0000\n",
      "iprec_at_recall_0.10  \t14\t1.0000\n",
      "iprec_at_recall_0.20  \t14\t1.0000\n",
      "iprec_at_recall_0.30  \t14\t1.0000\n",
      "iprec_at_recall_0.40  \t14\t1.0000\n",
      "iprec_at_recall_0.50  \t14\t1.0000\n",
      "iprec_at_recall_0.60  \t14\t1.0000\n",
      "iprec_at_recall_0.70  \t14\t1.0000\n",
      "iprec_at_recall_0.80  \t14\t1.0000\n",
      "iprec_at_recall_0.90  \t14\t1.0000\n",
      "iprec_at_recall_1.00  \t14\t1.0000\n",
      "P_5                   \t14\t0.2000\n",
      "P_10                  \t14\t0.1000\n",
      "P_15                  \t14\t0.0667\n",
      "P_20                  \t14\t0.0500\n",
      "P_30                  \t14\t0.0333\n",
      "P_100                 \t14\t0.0100\n",
      "P_200                 \t14\t0.0050\n",
      "P_500                 \t14\t0.0020\n",
      "P_1000                \t14\t0.0010\n",
      "num_ret               \t16\t1111\n",
      "num_rel               \t16\t7\n",
      "num_rel_ret           \t16\t7\n",
      "map                   \t16\t0.1608\n",
      "Rprec                 \t16\t0.1429\n",
      "bpref                 \t16\t0.0612\n",
      "recip_rank            \t16\t0.2000\n",
      "iprec_at_recall_0.00  \t16\t0.2143\n",
      "iprec_at_recall_0.10  \t16\t0.2143\n",
      "iprec_at_recall_0.20  \t16\t0.2143\n",
      "iprec_at_recall_0.30  \t16\t0.2143\n",
      "iprec_at_recall_0.40  \t16\t0.2143\n",
      "iprec_at_recall_0.50  \t16\t0.1905\n",
      "iprec_at_recall_0.60  \t16\t0.1724\n",
      "iprec_at_recall_0.70  \t16\t0.1724\n",
      "iprec_at_recall_0.80  \t16\t0.0984\n",
      "iprec_at_recall_0.90  \t16\t0.0959\n",
      "iprec_at_recall_1.00  \t16\t0.0959\n",
      "P_5                   \t16\t0.2000\n",
      "P_10                  \t16\t0.1000\n",
      "P_15                  \t16\t0.2000\n",
      "P_20                  \t16\t0.1500\n",
      "P_30                  \t16\t0.1667\n",
      "P_100                 \t16\t0.0700\n",
      "P_200                 \t16\t0.0350\n",
      "P_500                 \t16\t0.0140\n",
      "P_1000                \t16\t0.0070\n",
      "num_ret               \t18\t32\n",
      "num_rel               \t18\t1\n",
      "num_rel_ret           \t18\t1\n",
      "map                   \t18\t1.0000\n",
      "Rprec                 \t18\t1.0000\n",
      "bpref                 \t18\t1.0000\n",
      "recip_rank            \t18\t1.0000\n",
      "iprec_at_recall_0.00  \t18\t1.0000\n",
      "iprec_at_recall_0.10  \t18\t1.0000\n",
      "iprec_at_recall_0.20  \t18\t1.0000\n",
      "iprec_at_recall_0.30  \t18\t1.0000\n",
      "iprec_at_recall_0.40  \t18\t1.0000\n",
      "iprec_at_recall_0.50  \t18\t1.0000\n",
      "iprec_at_recall_0.60  \t18\t1.0000\n",
      "iprec_at_recall_0.70  \t18\t1.0000\n",
      "iprec_at_recall_0.80  \t18\t1.0000\n",
      "iprec_at_recall_0.90  \t18\t1.0000\n",
      "iprec_at_recall_1.00  \t18\t1.0000\n",
      "P_5                   \t18\t0.2000\n",
      "P_10                  \t18\t0.1000\n",
      "P_15                  \t18\t0.0667\n",
      "P_20                  \t18\t0.0500\n",
      "P_30                  \t18\t0.0333\n",
      "P_100                 \t18\t0.0100\n",
      "P_200                 \t18\t0.0050\n",
      "P_500                 \t18\t0.0020\n",
      "P_1000                \t18\t0.0010\n",
      "num_ret               \t19\t627\n",
      "num_rel               \t19\t2\n",
      "num_rel_ret           \t19\t1\n",
      "map                   \t19\t0.1667\n",
      "Rprec                 \t19\t0.0000\n",
      "bpref                 \t19\t0.0000\n",
      "recip_rank            \t19\t0.3333\n",
      "iprec_at_recall_0.00  \t19\t0.3333\n",
      "iprec_at_recall_0.10  \t19\t0.3333\n",
      "iprec_at_recall_0.20  \t19\t0.3333\n",
      "iprec_at_recall_0.30  \t19\t0.3333\n",
      "iprec_at_recall_0.40  \t19\t0.3333\n",
      "iprec_at_recall_0.50  \t19\t0.3333\n",
      "iprec_at_recall_0.60  \t19\t0.0000\n",
      "iprec_at_recall_0.70  \t19\t0.0000\n",
      "iprec_at_recall_0.80  \t19\t0.0000\n",
      "iprec_at_recall_0.90  \t19\t0.0000\n",
      "iprec_at_recall_1.00  \t19\t0.0000\n",
      "P_5                   \t19\t0.2000\n",
      "P_10                  \t19\t0.1000\n",
      "P_15                  \t19\t0.0667\n",
      "P_20                  \t19\t0.0500\n",
      "P_30                  \t19\t0.0333\n",
      "P_100                 \t19\t0.0100\n",
      "P_200                 \t19\t0.0050\n",
      "P_500                 \t19\t0.0020\n",
      "P_1000                \t19\t0.0010\n",
      "num_ret               \t2\t59\n",
      "num_rel               \t2\t2\n",
      "num_rel_ret           \t2\t2\n",
      "map                   \t2\t0.5270\n",
      "Rprec                 \t2\t0.5000\n",
      "bpref                 \t2\t0.5000\n",
      "recip_rank            \t2\t1.0000\n",
      "iprec_at_recall_0.00  \t2\t1.0000\n",
      "iprec_at_recall_0.10  \t2\t1.0000\n",
      "iprec_at_recall_0.20  \t2\t1.0000\n",
      "iprec_at_recall_0.30  \t2\t1.0000\n",
      "iprec_at_recall_0.40  \t2\t1.0000\n",
      "iprec_at_recall_0.50  \t2\t1.0000\n",
      "iprec_at_recall_0.60  \t2\t0.0541\n",
      "iprec_at_recall_0.70  \t2\t0.0541\n",
      "iprec_at_recall_0.80  \t2\t0.0541\n",
      "iprec_at_recall_0.90  \t2\t0.0541\n",
      "iprec_at_recall_1.00  \t2\t0.0541\n",
      "P_5                   \t2\t0.2000\n",
      "P_10                  \t2\t0.1000\n",
      "P_15                  \t2\t0.0667\n",
      "P_20                  \t2\t0.0500\n",
      "P_30                  \t2\t0.0333\n",
      "P_100                 \t2\t0.0200\n",
      "P_200                 \t2\t0.0100\n",
      "P_500                 \t2\t0.0040\n",
      "P_1000                \t2\t0.0020\n",
      "num_ret               \t22\t467\n",
      "num_rel               \t22\t1\n",
      "num_rel_ret           \t22\t1\n",
      "map                   \t22\t0.0476\n",
      "Rprec                 \t22\t0.0000\n",
      "bpref                 \t22\t0.0000\n",
      "recip_rank            \t22\t0.0476\n",
      "iprec_at_recall_0.00  \t22\t0.0476\n",
      "iprec_at_recall_0.10  \t22\t0.0476\n",
      "iprec_at_recall_0.20  \t22\t0.0476\n",
      "iprec_at_recall_0.30  \t22\t0.0476\n",
      "iprec_at_recall_0.40  \t22\t0.0476\n",
      "iprec_at_recall_0.50  \t22\t0.0476\n",
      "iprec_at_recall_0.60  \t22\t0.0476\n",
      "iprec_at_recall_0.70  \t22\t0.0476\n",
      "iprec_at_recall_0.80  \t22\t0.0476\n",
      "iprec_at_recall_0.90  \t22\t0.0476\n",
      "iprec_at_recall_1.00  \t22\t0.0476\n",
      "P_5                   \t22\t0.0000\n",
      "P_10                  \t22\t0.0000\n",
      "P_15                  \t22\t0.0000\n",
      "P_20                  \t22\t0.0000\n",
      "P_30                  \t22\t0.0333\n",
      "P_100                 \t22\t0.0100\n",
      "P_200                 \t22\t0.0050\n",
      "P_500                 \t22\t0.0020\n",
      "P_1000                \t22\t0.0010\n",
      "num_ret               \t24\t770\n",
      "num_rel               \t24\t1\n",
      "num_rel_ret           \t24\t1\n",
      "map                   \t24\t0.5000\n",
      "Rprec                 \t24\t0.0000\n",
      "bpref                 \t24\t0.0000\n",
      "recip_rank            \t24\t0.5000\n",
      "iprec_at_recall_0.00  \t24\t0.5000\n",
      "iprec_at_recall_0.10  \t24\t0.5000\n",
      "iprec_at_recall_0.20  \t24\t0.5000\n",
      "iprec_at_recall_0.30  \t24\t0.5000\n",
      "iprec_at_recall_0.40  \t24\t0.5000\n",
      "iprec_at_recall_0.50  \t24\t0.5000\n",
      "iprec_at_recall_0.60  \t24\t0.5000\n",
      "iprec_at_recall_0.70  \t24\t0.5000\n",
      "iprec_at_recall_0.80  \t24\t0.5000\n",
      "iprec_at_recall_0.90  \t24\t0.5000\n",
      "iprec_at_recall_1.00  \t24\t0.5000\n",
      "P_5                   \t24\t0.2000\n",
      "P_10                  \t24\t0.1000\n",
      "P_15                  \t24\t0.0667\n",
      "P_20                  \t24\t0.0500\n",
      "P_30                  \t24\t0.0333\n",
      "P_100                 \t24\t0.0100\n",
      "P_200                 \t24\t0.0050\n",
      "P_500                 \t24\t0.0020\n",
      "P_1000                \t24\t0.0010\n",
      "num_ret               \t26\t584\n",
      "num_rel               \t26\t3\n",
      "num_rel_ret           \t26\t3\n",
      "map                   \t26\t0.0826\n",
      "Rprec                 \t26\t0.0000\n",
      "bpref                 \t26\t0.0000\n",
      "recip_rank            \t26\t0.1111\n",
      "iprec_at_recall_0.00  \t26\t0.1111\n",
      "iprec_at_recall_0.10  \t26\t0.1111\n",
      "iprec_at_recall_0.20  \t26\t0.1111\n",
      "iprec_at_recall_0.30  \t26\t0.1111\n",
      "iprec_at_recall_0.40  \t26\t0.0741\n",
      "iprec_at_recall_0.50  \t26\t0.0741\n",
      "iprec_at_recall_0.60  \t26\t0.0741\n",
      "iprec_at_recall_0.70  \t26\t0.0741\n",
      "iprec_at_recall_0.80  \t26\t0.0625\n",
      "iprec_at_recall_0.90  \t26\t0.0625\n",
      "iprec_at_recall_1.00  \t26\t0.0625\n",
      "P_5                   \t26\t0.0000\n",
      "P_10                  \t26\t0.1000\n",
      "P_15                  \t26\t0.0667\n",
      "P_20                  \t26\t0.0500\n",
      "P_30                  \t26\t0.0667\n",
      "P_100                 \t26\t0.0300\n",
      "P_200                 \t26\t0.0150\n",
      "P_500                 \t26\t0.0060\n",
      "P_1000                \t26\t0.0030\n",
      "num_ret               \t28\t1259\n",
      "num_rel               \t28\t2\n",
      "num_rel_ret           \t28\t2\n",
      "map                   \t28\t0.1833\n",
      "Rprec                 \t28\t0.0000\n",
      "bpref                 \t28\t0.0000\n",
      "recip_rank            \t28\t0.2000\n",
      "iprec_at_recall_0.00  \t28\t0.2000\n",
      "iprec_at_recall_0.10  \t28\t0.2000\n",
      "iprec_at_recall_0.20  \t28\t0.2000\n",
      "iprec_at_recall_0.30  \t28\t0.2000\n",
      "iprec_at_recall_0.40  \t28\t0.2000\n",
      "iprec_at_recall_0.50  \t28\t0.2000\n",
      "iprec_at_recall_0.60  \t28\t0.1667\n",
      "iprec_at_recall_0.70  \t28\t0.1667\n",
      "iprec_at_recall_0.80  \t28\t0.1667\n",
      "iprec_at_recall_0.90  \t28\t0.1667\n",
      "iprec_at_recall_1.00  \t28\t0.1667\n",
      "P_5                   \t28\t0.2000\n",
      "P_10                  \t28\t0.1000\n",
      "P_15                  \t28\t0.1333\n",
      "P_20                  \t28\t0.1000\n",
      "P_30                  \t28\t0.0667\n",
      "P_100                 \t28\t0.0200\n",
      "P_200                 \t28\t0.0100\n",
      "P_500                 \t28\t0.0040\n",
      "P_1000                \t28\t0.0020\n",
      "num_ret               \t4\t900\n",
      "num_rel               \t4\t4\n",
      "num_rel_ret           \t4\t4\n",
      "map                   \t4\t0.5402\n",
      "Rprec                 \t4\t0.5000\n",
      "bpref                 \t4\t0.5000\n",
      "recip_rank            \t4\t1.0000\n",
      "iprec_at_recall_0.00  \t4\t1.0000\n",
      "iprec_at_recall_0.10  \t4\t1.0000\n",
      "iprec_at_recall_0.20  \t4\t1.0000\n",
      "iprec_at_recall_0.30  \t4\t1.0000\n",
      "iprec_at_recall_0.40  \t4\t1.0000\n",
      "iprec_at_recall_0.50  \t4\t1.0000\n",
      "iprec_at_recall_0.60  \t4\t0.1200\n",
      "iprec_at_recall_0.70  \t4\t0.1200\n",
      "iprec_at_recall_0.80  \t4\t0.0408\n",
      "iprec_at_recall_0.90  \t4\t0.0408\n",
      "iprec_at_recall_1.00  \t4\t0.0408\n",
      "P_5                   \t4\t0.4000\n",
      "P_10                  \t4\t0.2000\n",
      "P_15                  \t4\t0.1333\n",
      "P_20                  \t4\t0.1000\n",
      "P_30                  \t4\t0.1000\n",
      "P_100                 \t4\t0.0400\n",
      "P_200                 \t4\t0.0200\n",
      "P_500                 \t4\t0.0080\n",
      "P_1000                \t4\t0.0040\n",
      "num_ret               \t6\t449\n",
      "num_rel               \t6\t1\n",
      "num_rel_ret           \t6\t1\n",
      "map                   \t6\t0.0256\n",
      "Rprec                 \t6\t0.0000\n",
      "bpref                 \t6\t0.0000\n",
      "recip_rank            \t6\t0.0256\n",
      "iprec_at_recall_0.00  \t6\t0.0256\n",
      "iprec_at_recall_0.10  \t6\t0.0256\n",
      "iprec_at_recall_0.20  \t6\t0.0256\n",
      "iprec_at_recall_0.30  \t6\t0.0256\n",
      "iprec_at_recall_0.40  \t6\t0.0256\n",
      "iprec_at_recall_0.50  \t6\t0.0256\n",
      "iprec_at_recall_0.60  \t6\t0.0256\n",
      "iprec_at_recall_0.70  \t6\t0.0256\n",
      "iprec_at_recall_0.80  \t6\t0.0256\n",
      "iprec_at_recall_0.90  \t6\t0.0256\n",
      "iprec_at_recall_1.00  \t6\t0.0256\n",
      "P_5                   \t6\t0.0000\n",
      "P_10                  \t6\t0.0000\n",
      "P_15                  \t6\t0.0000\n",
      "P_20                  \t6\t0.0000\n",
      "P_30                  \t6\t0.0000\n",
      "P_100                 \t6\t0.0100\n",
      "P_200                 \t6\t0.0050\n",
      "P_500                 \t6\t0.0020\n",
      "P_1000                \t6\t0.0010\n",
      "num_ret               \t7\t416\n",
      "num_rel               \t7\t3\n",
      "num_rel_ret           \t7\t2\n",
      "map                   \t7\t0.2167\n",
      "Rprec                 \t7\t0.0000\n",
      "bpref                 \t7\t0.0000\n",
      "recip_rank            \t7\t0.2500\n",
      "iprec_at_recall_0.00  \t7\t0.4000\n",
      "iprec_at_recall_0.10  \t7\t0.4000\n",
      "iprec_at_recall_0.20  \t7\t0.4000\n",
      "iprec_at_recall_0.30  \t7\t0.4000\n",
      "iprec_at_recall_0.40  \t7\t0.4000\n",
      "iprec_at_recall_0.50  \t7\t0.4000\n",
      "iprec_at_recall_0.60  \t7\t0.4000\n",
      "iprec_at_recall_0.70  \t7\t0.4000\n",
      "iprec_at_recall_0.80  \t7\t0.0000\n",
      "iprec_at_recall_0.90  \t7\t0.0000\n",
      "iprec_at_recall_1.00  \t7\t0.0000\n",
      "P_5                   \t7\t0.4000\n",
      "P_10                  \t7\t0.2000\n",
      "P_15                  \t7\t0.1333\n",
      "P_20                  \t7\t0.1000\n",
      "P_30                  \t7\t0.0667\n",
      "P_100                 \t7\t0.0200\n",
      "P_200                 \t7\t0.0100\n",
      "P_500                 \t7\t0.0040\n",
      "P_1000                \t7\t0.0020\n",
      "num_ret               \t9\t1825\n",
      "num_rel               \t9\t1\n",
      "num_rel_ret           \t9\t1\n",
      "map                   \t9\t0.3333\n",
      "Rprec                 \t9\t0.0000\n",
      "bpref                 \t9\t0.0000\n",
      "recip_rank            \t9\t0.3333\n",
      "iprec_at_recall_0.00  \t9\t0.3333\n",
      "iprec_at_recall_0.10  \t9\t0.3333\n",
      "iprec_at_recall_0.20  \t9\t0.3333\n",
      "iprec_at_recall_0.30  \t9\t0.3333\n",
      "iprec_at_recall_0.40  \t9\t0.3333\n",
      "iprec_at_recall_0.50  \t9\t0.3333\n",
      "iprec_at_recall_0.60  \t9\t0.3333\n",
      "iprec_at_recall_0.70  \t9\t0.3333\n",
      "iprec_at_recall_0.80  \t9\t0.3333\n",
      "iprec_at_recall_0.90  \t9\t0.3333\n",
      "iprec_at_recall_1.00  \t9\t0.3333\n",
      "P_5                   \t9\t0.2000\n",
      "P_10                  \t9\t0.1000\n",
      "P_15                  \t9\t0.0667\n",
      "P_20                  \t9\t0.0500\n",
      "P_30                  \t9\t0.0333\n",
      "P_100                 \t9\t0.0100\n",
      "P_200                 \t9\t0.0050\n",
      "P_500                 \t9\t0.0020\n",
      "P_1000                \t9\t0.0010\n",
      "runid                 \tall\ttest\n",
      "num_q                 \tall\t15\n",
      "num_ret               \tall\t9776\n",
      "num_rel               \tall\t35\n",
      "num_rel_ret           \tall\t33\n",
      "map                   \tall\t0.3397\n",
      "gm_map                \tall\t0.2119\n",
      "Rprec                 \tall\t0.2095\n",
      "bpref                 \tall\t0.2041\n",
      "recip_rank            \tall\t0.4204\n",
      "iprec_at_recall_0.00  \tall\t0.4337\n",
      "iprec_at_recall_0.10  \tall\t0.4337\n",
      "iprec_at_recall_0.20  \tall\t0.4337\n",
      "iprec_at_recall_0.30  \tall\t0.4337\n",
      "iprec_at_recall_0.40  \tall\t0.4313\n",
      "iprec_at_recall_0.50  \tall\t0.4297\n",
      "iprec_at_recall_0.60  \tall\t0.2823\n",
      "iprec_at_recall_0.70  \tall\t0.2795\n",
      "iprec_at_recall_0.80  \tall\t0.2419\n",
      "iprec_at_recall_0.90  \tall\t0.2410\n",
      "iprec_at_recall_1.00  \tall\t0.2410\n",
      "P_5                   \tall\t0.1733\n",
      "P_10                  \tall\t0.0933\n",
      "P_15                  \tall\t0.0756\n",
      "P_20                  \tall\t0.0600\n",
      "P_30                  \tall\t0.0533\n",
      "P_100                 \tall\t0.0213\n",
      "P_200                 \tall\t0.0110\n",
      "P_500                 \tall\t0.0044\n",
      "P_1000                \tall\t0.0022\n",
      "\n"
     ]
    }
   ],
   "source": [
    "trecEval(TOPIC_FILE, QRELS_FILE, QP_Q4, SEARCHER_Q4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation after Scoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Improvements\n",
    "\n",
    "Reduced the value of b from defalut 0.75 to 0.3 and increased the value of K1 from default 1.2 to 3. Reasons for doing this is to reduce the impact of document size on the overall scoring. \n",
    "Heuristics: For selecting the range of values for b and K1 experimental results and heuristics from the research papers are used.  Source: elastic.co - Blog (Artilce: considerations for picking b and k values)\n",
    "\n",
    "\n",
    "#### Perfomance:\n",
    "Perfomance improved overall because for 11 out of 15 queries we are getting required relevant information in the first five retrived documents. previously only 8 out of 15 quiries got relevant docs in top 5 results. (p5 measure improved from  0.12 to 0.1733) \n",
    "\n",
    "for query 28: we are getting the relevant document in first five documents now. (False negative became True Positive) and (False positive is ranked lower now as expected)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall after making modifications and tuning BM25 parameters, For 11 out of 15 queries we are getting required relevant information in the first five retrived documents. And also overall p5 measure improved from  0.12 to 0.1733 (1.44 times improvement). This means that tuning the BM25 scoring parameters is a good idea for this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}