Data preparation-Terrier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using BERT (and other transformer methods) for IR - Data preparation\n",
    "\n",
    "This notebook covers the basic on how to implement a nice pipeline for training and running inference over a IR dataset.\n",
    "We wil use Anserini, with PySerini, to index and retrieve documents over the MsMarco TREC 2019 DL dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Dependencies installation\n",
    "First, let's install wh§at we need. I highly recommend using something like Conda to manage your environment!\n",
    "\n",
    "We are using Python 3.7 and Cuda 10.1 (If you are using another version, check how to install Pytorch on https://pytorch.org/get-started/locally/#start-locally)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-20T10:41:39.481644Z",
     "start_time": "2020-05-20T10:41:19.353630Z"
    },
    "code_folding": [],
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "\n",
      "==> WARNING: A newer version of conda exists. <==\n",
      "  current version: 4.8.2\n",
      "  latest version: 4.8.3\n",
      "\n",
      "Please update conda by running\n",
      "\n",
      "    $ conda update -n base conda\n",
      "\n",
      "\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/arthur/miniconda3/envs/bert4IR\n",
      "\n",
      "  added / updated specs:\n",
      "    - cudatoolkit=10.1\n",
      "    - pytorch\n",
      "    - torchvision\n",
      "\n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "  ca-certificates                     2019.11.28-hecc5488_0 --> 2020.4.5.1-hecc5488_0\n",
      "  certifi                         2019.11.28-py37hc8dfbb8_1 --> 2020.4.5.1-py37hc8dfbb8_0\n",
      "  openssl                                 1.1.1f-h516909a_0 --> 1.1.1g-h516909a_0\n",
      "\n",
      "\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "Requirement already satisfied: tokenizer in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (2.0.5)\n",
      "Requirement already satisfied: transformers in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (2.7.0)\n",
      "Requirement already satisfied: filelock in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from transformers) (3.0.12)\n",
      "Requirement already satisfied: tqdm>=4.27 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from transformers) (4.44.1)\n",
      "Requirement already satisfied: requests in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from transformers) (2.23.0)\n",
      "Requirement already satisfied: sacremoses in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from transformers) (0.0.38)\n",
      "Requirement already satisfied: numpy in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from transformers) (1.18.1)\n",
      "Requirement already satisfied: sentencepiece in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from transformers) (0.1.85)\n",
      "Requirement already satisfied: tokenizers==0.5.2 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from transformers) (0.5.2)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from transformers) (2020.2.20)\n",
      "Requirement already satisfied: boto3 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from transformers) (1.12.34)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from requests->transformers) (1.25.8)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from requests->transformers) (2.9)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from requests->transformers) (2020.4.5.1)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from requests->transformers) (3.0.4)\n",
      "Requirement already satisfied: joblib in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from sacremoses->transformers) (0.14.1)\n",
      "Requirement already satisfied: six in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from sacremoses->transformers) (1.14.0)\n",
      "Requirement already satisfied: click in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from sacremoses->transformers) (7.1.1)\n",
      "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from boto3->transformers) (0.3.3)\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from boto3->transformers) (0.9.5)\n",
      "Requirement already satisfied: botocore<1.16.0,>=1.15.34 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from boto3->transformers) (1.15.34)\n",
      "Requirement already satisfied: docutils<0.16,>=0.10 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.34->boto3->transformers) (0.15.2)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.34->boto3->transformers) (2.8.1)\n"
     ]
    }
   ],
   "source": [
    "#Pytorch\n",
    "! conda install -y pytorch torchvision cudatoolkit=10.1 -c pytorch\n",
    "# 🤗 tokenizer (this gives us A HUGE boost on performance. Tokenizing is the slowest part of the process)\n",
    "! pip install tokenizer\n",
    "# 🤗 Transformer\n",
    "! pip install transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-02T00:43:25.717718Z",
     "start_time": "2020-04-02T00:43:25.599407Z"
    }
   },
   "source": [
    "### Terrier Instalation\n",
    "Terrier should be easier to install/use than Anserini (i.e. No need of Java 11). \n",
    "\n",
    "- If you have Java installed on your machine, make sure that `JAVA_HOME` is set properly.\n",
    "- If you want to be sure, check [SDKMAN!](https://sdkman.io/) to install a cleaner version of Java, in the version you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-20T10:42:59.727294Z",
     "start_time": "2020-05-20T10:42:53.006723Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting python-terrier\n",
      "  Cloning https://github.com/terrier-org/pyterrier.git to /tmp/pip-install-6o31_ypx/python-terrier\n",
      "  Running command git clone -q https://github.com/terrier-org/pyterrier.git /tmp/pip-install-6o31_ypx/python-terrier\n",
      "Requirement already satisfied, skipping upgrade: pyjnius~=1.3.0 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from python-terrier) (1.3.0)\n",
      "Requirement already satisfied, skipping upgrade: numpy in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from python-terrier) (1.18.1)\n",
      "Requirement already satisfied, skipping upgrade: pandas in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from python-terrier) (1.0.3)\n",
      "Requirement already satisfied, skipping upgrade: wget in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from python-terrier) (3.2)\n",
      "Requirement already satisfied, skipping upgrade: pytrec_eval in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from python-terrier) (0.4)\n",
      "Requirement already satisfied, skipping upgrade: tqdm in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from python-terrier) (4.44.1)\n",
      "Requirement already satisfied, skipping upgrade: matchpy in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from python-terrier) (0.5.1)\n",
      "Requirement already satisfied, skipping upgrade: deprecation in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from python-terrier) (2.1.0)\n",
      "Requirement already satisfied, skipping upgrade: cython in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from pyjnius~=1.3.0->python-terrier) (0.29.16)\n",
      "Requirement already satisfied, skipping upgrade: six>=1.7.0 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from pyjnius~=1.3.0->python-terrier) (1.14.0)\n",
      "Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from pandas->python-terrier) (2019.3)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2.6.1 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from pandas->python-terrier) (2.8.1)\n",
      "Requirement already satisfied, skipping upgrade: multiset<3.0,>=2.0 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from matchpy->python-terrier) (2.1.1)\n",
      "Requirement already satisfied, skipping upgrade: hopcroftkarp<2.0,>=1.2 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from matchpy->python-terrier) (1.2.5)\n",
      "Requirement already satisfied, skipping upgrade: packaging in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from deprecation->python-terrier) (20.4)\n",
      "Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /ssd/home/arthur/miniconda3/envs/bert4IR/lib/python3.7/site-packages (from packaging->deprecation->python-terrier) (2.4.7)\n",
      "Building wheels for collected packages: python-terrier\n",
      "  Building wheel for python-terrier (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for python-terrier: filename=python_terrier-0.1.3-py3-none-any.whl size=28344 sha256=a335fc9d1a5585f9a4572108fc7446fd0e801ee2a436b7f452c401b0e7a3fcbf\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-4x7swyea/wheels/61/12/f7/d3c3d17f72ab9ad1c5d510a0d6bd1612023e01fa0e07f01059\n",
      "Successfully built python-terrier\n",
      "Installing collected packages: python-terrier\n",
      "  Attempting uninstall: python-terrier\n",
      "    Found existing installation: python-terrier 0.1.3\n",
      "    Uninstalling python-terrier-0.1.3:\n",
      "      Successfully uninstalled python-terrier-0.1.3\n",
      "Successfully installed python-terrier-0.1.3\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ[\"JAVA_HOME\"] = \"/home/arthur/.sdkman/candidates/java/8.0.242-open\" # Make sure this points to the right place for your Java Home!\n",
    "!pip install --upgrade git+https://github.com/terrier-org/pyterrier.git#egg=python-terrier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local variables\n",
    "These variables are local to you, and should be eddited accordingly. thinks like path to download the dataset are all set here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-20T10:43:04.905672Z",
     "start_time": "2020-05-20T10:43:04.898662Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "data_home = \"/ssd2/arthur/MsMarcoTREC/\"  # Where you want to store the docs\n",
    "n_threads = 32  # Number of threads to use. Make sure you have more than the number here!\n",
    "\n",
    "\n",
    "def get_path(x):\n",
    "    return os.path.join(data_home, x)\n",
    "\n",
    "\n",
    "if not os.path.isdir(data_home):\n",
    "    os.makedirs(data_home)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download data\n",
    "We are using the MsMarco TREC 2019 dataset. We should download everything here.\n",
    "\n",
    "If you are running this from the DeepIR machine from WIS, we already have everything there. Ask Arthur where this is and `ln -s` to your path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-20T10:43:06.262538Z",
     "start_time": "2020-05-20T10:43:06.244120Z"
    }
   },
   "outputs": [],
   "source": [
    "from urllib import request\n",
    "import gzip\n",
    "import shutil\n",
    "\n",
    "download_path = \"https://msmarco.blob.core.windows.net/msmarcoranking/\"  # default MsMarco path for downloading data\n",
    "# It sucks to need documents in both .trec and .tsv, but it's easier this way, believe me.\n",
    "files_to_get = [\n",
    "    \"docs/msmarco-docs.trec\",  #docs in trec format\n",
    "    \"docs/msmarco-docs.tsv\",  # docs in tsv format\n",
    "    \"queries/msmarco-doctrain-queries.tsv\",  # train queries\n",
    "    \"qrels/msmarco-doctrain-qrels.tsv\",  # train qrels\n",
    "    \"queries/msmarco-docdev-queries.tsv\",  # dev queries\n",
    "    \"qrels/msmarco-docdev-qrels.tsv\",  # dev qrels\n",
    "    \"queries/msmarco-test2019-queries.tsv\",  # test queries\n",
    "    \"qrels/2019qrels-docs.txt\"  # test qrels\n",
    "]\n",
    "for file in files_to_get:\n",
    "    local_file_path = get_path(file)\n",
    "    if not os.path.isfile(local_file_path):\n",
    "        print(\n",
    "            f\"File {file.split('/')[-1]} not found. Downloading it from the Web\"\n",
    "        )\n",
    "        url_to_fetch_from = download_path + file.split(\"/\")[1] + \".gz\"\n",
    "        # qrels for test comes from NIST, not from Microsoft. Also, no need to uncompress\n",
    "        if file == \"qrels/2019qrels-docs.txt\":\n",
    "            url_to_fetch_from = \"https://trec.nist.gov/data/deep/2019qrels-docs.txt\"\n",
    "            request.urlretrieve(url_to_fetch_from, local_file_path)\n",
    "            continue\n",
    "        # Create dir if it doesn't exist\n",
    "        if not os.path.isdir(\"/\".join(local_file_path.split(\"/\")[:-1])):\n",
    "            os.makedirs(\"/\".join(local_file_path.split(\"/\")[:-1]))\n",
    "        try:\n",
    "            request.urlretrieve(url_to_fetch_from, local_file_path + \".gz\")\n",
    "        except:\n",
    "            print(\n",
    "                f\"Could not fetch {file} from {url_to_fetch_from}. Make sure that's the right URL!\"\n",
    "            )\n",
    "            continue\n",
    "        #Uncompress file. Not needed, but easier. (you could use the gzip lib to open the files...)\n",
    "        with gzip.open(local_file_path + \".gz\", 'rb') as f_in, open(local_file_path, 'wb') as outf:\n",
    "            print(f\"Extracting file {file}\")\n",
    "            shutil.copyfileobj(f_in, outf)\n",
    "            os.remove(local_file_path + \".gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Terrier Index\n",
    "This may take a while... (~1h in our experiments)\n",
    "\n",
    "- You will not receive any feedback on the output while the indexing is running. You may chack the progress by running `ls -lah` on the index folder and check if the files are increasing in size.\n",
    "- Alternatively, run the script manually (don't forget to set `JAVA_HOME`) and have some feedback on the terminal. As a sanity check, the index must contain 3,213,835 documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-20T11:35:18.272359Z",
     "start_time": "2020-05-20T10:44:08.545386Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IndexingType.CLASSIC\n",
      "IndexingType.CLASSIC\n"
     ]
    }
   ],
   "source": [
    "import pyterrier as pt\n",
    "try:\n",
    "    pt.init(mem=16384)\n",
    "except:\n",
    "    continue\n",
    "index_path = get_path(\"terrier-index\")\n",
    "import shutil\n",
    "shutil.rmtree(index_path, ignore_errors=True)\n",
    "indexer = pt.TRECCollectionIndexer(index_path)\n",
    "# index_properies = {\"block.indexing\":\"true\", \"invertedfile.lexiconscanner\":\"pointers\"}\n",
    "index = indexer.index(get_path(\"docs/msmarco-docs.trec\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-20T14:16:43.028239Z",
     "start_time": "2020-05-20T14:16:42.976441Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collection statistics:\n",
      "number of indexed documents: 3213835\n",
      "size of vocabulary: 16168096\n",
      "number of tokens: 2204592607\n",
      "number of pointers: 905088837\n",
      "number of fields: 0\n",
      "field names: []\n",
      "blocks: false\n"
     ]
    }
   ],
   "source": [
    "indexer.getIndexStats()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7 (Bert4IR)",
   "language": "python",
   "name": "myenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}