spacy-inception_ner_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training the NER pipeline component"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this project is to update the Named Entity Recognition (NER) pipeline component of the language model we've used in a previous project, [POS Tagging, Syntactic Dependency Parsing and NER](https://github.com/j-n-t/natural_language_processing/blob/master/POS%20Tagging%20and%20NER.ipynb), and to improve its performance.\n",
    "\n",
    "In order to do that, we'll use **spaCy** and the [**INCEpTION annotation tool**](https://inception-project.github.io/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Perform initial imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "from spacy.util import minibatch, compounding\n",
    "from spacy.pipeline import EntityRecognizer\n",
    "\n",
    "from spacy import displacy\n",
    "\n",
    "from mediawiki import MediaWiki\n",
    "\n",
    "import os\n",
    "import json\n",
    "\n",
    "from IPython.core.display import HTML\n",
    "\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our previous project, we've seen that the NER pipeline component was not able to associate the proper entity label to the token **Azores, an archipelago of nine islands in the North Atlantic Ocean**.\n",
    "\n",
    "To correct this, we'll load the **wikipedia page about Azores and use that text to update our NER pipeline component**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load wikipedia page about Azores\n",
    "\n",
    "wikipedia = MediaWiki()\n",
    "azores = wikipedia.page('Azores - Wikipedia')\n",
    "# Azores - Wikipedia corresponds to the title tag of the source code for this page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Explore data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://en.wikipedia.org/wiki/Azores'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# url of the wikipedia page about Azores\n",
    "\n",
    "azores.url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The Azores ( ə-ZORZ, also US:  AY-zorz; Portuguese: Açores [ɐˈsoɾɨʃ]), officially the Autonomous Region of the Azores (Região Autónoma dos Açores), is one of the two autonomous regions of Portugal (along with Madeira). It is an archipelago composed of nine volcanic islands in the Macaronesia region of the North Atlantic Ocean, about 1,360 km (850 mi) west of continental Portugal, about 1,500 km (930 mi) west of Lisbon, in continental Portugal, about 1,500 km (930 mi) northwest of Morocco, and about 2,980 km (1,850 mi) southeast of Newfoundland, Canada.'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fist 2 sentences (out of a maximum of 10) of the wikipedia page about Azores \n",
    "\n",
    "wikipedia.summary(\"Azores - Wikipedia\", sentences=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# doc with all the content of the wikipedia page about Azores\n",
    "\n",
    "doc = nlp(azores.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "The Azores ( ə-ZORZ, also US:  AY-zorz; Portuguese: Açores [ɐˈsoɾɨʃ]), officially the Autonomous Region of the Azores (Região Autónoma dos Açores), is one of the two autonomous regions of Portugal (along with Madeira)."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first sentence of the doc container\n",
    "\n",
    "list(doc.sents)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "513"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of sentences in the doc container\n",
    "\n",
    "len(list(doc.sents))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The wikipedia page about Azores has 513 sentences. To accomplish our goal, we'll use the **first 100 sentences** and test the updated NER pipeline component to see if it behaves as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Save data to a text file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select first 100 sentences\n",
    "\n",
    "azores_100sent = list(doc.sents)[0:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Beginning in 1868, Portugal issued its stamps overprinted with \"AÇORES\" for use in the islands.,\n",
       " Between 1892 and 1906, it also issued separate stamps for the three administrative districts of the time.,\n",
       " During the 18th and 19th centuries, Graciosa was host to many prominent figures, including Chateaubriand, the French writer who passed through upon his escape to America during the French revolution;,\n",
       " Almeida Garrett, the Portuguese poet who visited an uncle and wrote some poetry while there; and Prince Albert of Monaco, the 19th century oceanographer who led several expeditions in the waters of the Azores.,\n",
       " He arrived on his yacht Hirondelle, and visited the furna da caldeira, the noted hot springs grotto.]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# last 5 sentences\n",
    "\n",
    "azores_100sent[-5:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save azores_100sent to a txt file\n",
    "\n",
    "# output directory\n",
    "output_dir = './ner_training/text'\n",
    "\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "\n",
    "with open(output_dir+'/azores_100sent.txt', 'w', encoding='utf8') as f:\n",
    "    for sentence in azores_100sent:\n",
    "        f.write(sentence.text+'\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Annotate the text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now use the INCEpTION annotation tool together with spaCy to perform the annotation of the text we have just saved.\n",
    "\n",
    "In order to do this, we will need to **set an external recommender for INCEpTION**. Detailed explanations can be found [here](https://inception-project.github.io/example-projects/external-recommender/) and [here](https://github.com/inception-project/external-recommender-spacy)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After launching **INCEpTION** and the **external recommender based on spaCy**, we need to do the following with INCEpTION:\n",
    "\n",
    "* create new project - set name and description\n",
    "* in the 'Documents' tab, import our txt file\n",
    "* in the 'Recommenders' tab, create a new recommender - set name, layer (Named entity), feature (value), tool (Remote classifier) and remote URL (http://localhost:5000/ner)\n",
    "* in the 'Tagsets' tab, select 'Named Entity tags', and in 'Tagset Details' change the language to 'en' and select the option 'Annotators may add new tags'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now return to the main menu by clicking on the top left corner (INCEpTION). We click on 'Annotation' and open the document.\n",
    "\n",
    "It's now time to start annotating! We select 'Named entity' from the 'Layer' dropdown menu on the right side of the screen and once we double click on a word and select the corresponding value, the external recommendations will appear highlighted in blue above the tokens.\n",
    "\n",
    "We can now accept or reject the suggested annotations and make new annotations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./ner_training/images/azores_inception.jpg\" title=\"Annotating with INCEpTION\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After finishing the annotation process, we can **export the document and save it in the [CoNLL 2002](https://www.clips.uantwerpen.be/conll2002/ner/) format**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Convert the file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to **convert this file to spaCy's json format** as documented [here](https://spacy.io/api/cli#convert) and [here](https://github.com/explosion/spaCy/tree/master/examples/training/ner_example_data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've created an `out` folder to store the converted files and ran the following command on the command line: `python -m spacy convert azores_100sent.conll out -c ner -s -n 10 -b en`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this .json file, we could now **train our NER pipeline component** from the command line following [these guidelines](https://spacy.io/api/cli#train)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Get entity offsets and prepare list of training examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we could also get the entity offsets from this file and prepare a list of training examples. In order to do this, I've created the function `train_examples_converter` to do this conversion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After copying the.json file to our project's folder, we can now load the file and pass it to the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load json file\n",
    "\n",
    "json_dir = './ner_training/annot/out'\n",
    "\n",
    "with open(json_dir+'/azores_100sent.json', encoding='utf8') as json_file:\n",
    "    train_examples_json = json.load(json_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tokens': [{'orth': 'The', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'Azores', 'tag': '-', 'ner': 'U-LOC'},\n",
       "  {'orth': '(', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'ə-ZORZ', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': ',', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'also', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'US', 'tag': '-', 'ner': 'U-GPE'},\n",
       "  {'orth': ':', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'AY-zorz', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': ';', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'Portuguese', 'tag': '-', 'ner': 'U-NORP'},\n",
       "  {'orth': ':', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'Açores', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': '[', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'ɐˈsoɾɨʃ', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': ']', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': ')', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': ',', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'officially', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'the', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'Autonomous', 'tag': '-', 'ner': 'B-LOC'},\n",
       "  {'orth': 'Region', 'tag': '-', 'ner': 'I-LOC'},\n",
       "  {'orth': 'of', 'tag': '-', 'ner': 'I-LOC'},\n",
       "  {'orth': 'the', 'tag': '-', 'ner': 'I-LOC'},\n",
       "  {'orth': 'Azores', 'tag': '-', 'ner': 'L-LOC'},\n",
       "  {'orth': '(', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'Região', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'Autónoma', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'dos', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'Açores', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': ')', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': ',', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'is', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'one', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'of', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'the', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'two', 'tag': '-', 'ner': 'U-CARDINAL'},\n",
       "  {'orth': 'autonomous', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'regions', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'of', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'Portugal', 'tag': '-', 'ner': 'U-GPE'},\n",
       "  {'orth': '(', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'along', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'with', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': 'Madeira', 'tag': '-', 'ner': 'U-LOC'},\n",
       "  {'orth': ')', 'tag': '-', 'ner': 'O'},\n",
       "  {'orth': '.', 'tag': '-', 'ner': 'O'}]}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check file\n",
    "\n",
    "train_examples_json[0]['paragraphs'][0]['sentences'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file follows the [BILUO scheme](https://spacy.io/api/annotation#biluo) for encoding entity annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_examples_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We grouped every 10 sentences into a document and so we have 8 documents (even though we selected the first 100 sentences, after importing them to INCEpTION we were left with 75 sentences due to different segmentation rules).\n",
    "\n",
    "Let's get our list of training examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_examples_converter(json):\n",
    "    \"\"\"Returns the converted list of training examples.\n",
    "    \n",
    "    Every element of the list is a tuple consisting of two values:\n",
    "    the text string and a dictionary for the annotations of the corresponding entities. \n",
    "\n",
    "    Args:\n",
    "        json (list): List of training examples in json format to be converted.\n",
    "\n",
    "    Returns:\n",
    "        list: List of converted training examples.\n",
    "    \"\"\"\n",
    "\n",
    "    train_examples = []\n",
    "\n",
    "    for doc_i in range(len(json)): #looping through each document\n",
    "\n",
    "        for sent_i in range(len(json[doc_i]['paragraphs'][0]['sentences'])): #looping through each sentence\n",
    "\n",
    "            text=[]\n",
    "            tags=[]\n",
    "            entities=[]\n",
    "\n",
    "            for token in json[doc_i]['paragraphs'][0]['sentences'][sent_i]['tokens']: #looping through each token\n",
    "\n",
    "                text.append(token['orth'])\n",
    "                tags.append(token['ner'])\n",
    "                if token['ner'] != 'O':\n",
    "                    start=len(' '.join(text))-len(text[-1])\n",
    "                    end=len(' '.join(text))\n",
    "                    if token['ner'].startswith('U'):\n",
    "                        entities.append((start, end, token['ner'][2:]))\n",
    "                    elif token['ner'].startswith('B'):\n",
    "                        start_multi = start\n",
    "                    elif token['ner'].startswith('L'):\n",
    "                        end_multi = end\n",
    "                        entities.append((start_multi, end_multi, token['ner'][2:]))\n",
    "\n",
    "            train_examples.append((' '.join(text), {'entities':entities}))\n",
    "            \n",
    "    return train_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"alert alert-success\">Everything seems to be working just fine!</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_examples = train_examples_converter(train_examples_json)\n",
    "\n",
    "assert train_examples[0][0][4:10] == 'Azores', \"This LOC entity should be Azores.\"\n",
    "assert train_examples[0][0][27:29] == 'US', \"This GPE entity should be US.\"\n",
    "assert train_examples[1][0][62:73] == 'Macaronesia', \"This LOC entity should be Macaronesia.\"\n",
    "assert train_examples[1][0][226:234] == 'Portugal', \"This GPE entity should be Portugal.\"\n",
    "\n",
    "HTML('<div class=\"alert alert-success\">Everything seems to be working just fine!</div>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('The Azores ( ə-ZORZ , also US : AY-zorz ; Portuguese : Açores [ ɐˈsoɾɨʃ ] ) , officially the Autonomous Region of the Azores ( Região Autónoma dos Açores ) , is one of the two autonomous regions of Portugal ( along with Madeira ) .',\n",
       " {'entities': [(4, 10, 'LOC'),\n",
       "   (27, 29, 'GPE'),\n",
       "   (42, 52, 'NORP'),\n",
       "   (93, 124, 'LOC'),\n",
       "   (172, 175, 'CARDINAL'),\n",
       "   (198, 206, 'GPE'),\n",
       "   (220, 227, 'LOC')]})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first sentence and corresponding entities\n",
    "\n",
    "train_examples[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this list, we can now **update our NER pipeline component**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Update the NER pipeline component"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting the training of the NER pipeline component, it is recommended to disable all other pipeline components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tagger', 'parser', 'ner']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pipeline components\n",
    "\n",
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to **disable the tagger and the parser**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('tagger', <spacy.pipeline.pipes.Tagger at 0x1cbb70ab828>),\n",
       " ('parser', <spacy.pipeline.pipes.DependencyParser at 0x1cbb9d5e888>)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.disable_pipes(['tagger', 'parser'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check if this worked as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ner']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only the NER pipeline component is active and we can now train it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create output directory for the ner pipeline component\n",
    "\n",
    "if not os.path.exists('./ner_training/ner_pipe'):\n",
    "    os.makedirs('./ner_training/ner_pipe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will follow [these guidelines](https://spacy.io/usage/training#ner) for the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "optimizer = nlp.entity.create_optimizer()\n",
    "\n",
    "for i in range(10):\n",
    "    random.shuffle(train_examples)\n",
    "    batch_size = compounding(2.0, 4.0, 1.05)\n",
    "    batches = minibatch(train_examples, size=batch_size)\n",
    "    for batch in batches:\n",
    "        sentences, annotations = zip(*batch)\n",
    "        #print(i, len(sentences), annotations)\n",
    "        nlp.update(sentences, annotations, drop=0.2, sgd=optimizer)\n",
    "        \n",
    "ner = nlp.get_pipe('ner')\n",
    "ner.to_disk('./ner_training/ner_pipe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by **creating an optimizer**. The `nlp.entity.create_optimizer()` method allows us to update the weights of our model without losing information about the existing set of entity types.\n",
    "\n",
    "Next, we **iterate over our training examples** creating **random batches of different sizes** with the help of the `minibatch()` and the `compounding()` methods.\n",
    "\n",
    "We can then **update our model** using the `nlp.update()` method. I've set a **dropout rate** of 0.2 to avoid memorization of the data.\n",
    "\n",
    "Finally, we **get the component from the pipeline** and **save it to disk**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Evaluate the NER pipeline component"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's now time to evaluate our updated NER pipeline component.\n",
    "\n",
    "We'll start by **loading our language model disabling the default NER component**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_sm', disable=['ner'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now **create a new pipeline instance** and then **load the data into it using the component we have saved before**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<spacy.pipeline.pipes.EntityRecognizer at 0x1cbc0ebebe8>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ner = EntityRecognizer(nlp.vocab)\n",
    "ner.from_disk('./ner_training/ner_pipe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we **add this component to our pipeline**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp.add_pipe(ner, 'azores_ner')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check our pipeline components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tagger', 'parser', 'azores_ner']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We finally have everything we need to test our new NER pipeline component.\n",
    "\n",
    "We can go back to our example from [POS Tagging, Syntactic Dependency Parsing and NER](https://github.com/j-n-t/natural_language_processing/blob/master/POS%20Tagging%20and%20NER.ipynb) and see if everything works as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(\"if that between America and Europe is ample, will that between the Continent and the Azores, or Madeira, or the Canaries, or Ireland, be sufficient?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "America: GPE - Countries, cities, states\n",
      "Europe: LOC - Non-GPE locations, mountain ranges, bodies of water\n",
      "Continent: LOC - Non-GPE locations, mountain ranges, bodies of water\n",
      "Azores: LOC - Non-GPE locations, mountain ranges, bodies of water\n",
      "Madeira: LOC - Non-GPE locations, mountain ranges, bodies of water\n",
      "Canaries: LOC - Non-GPE locations, mountain ranges, bodies of water\n",
      "Ireland: GPE - Countries, cities, states\n"
     ]
    }
   ],
   "source": [
    "for ent in doc.ents:\n",
    "    print(ent.text+': '+ent.label_+' - '+str(spacy.explain(ent.label_)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the **updated NER pipeline component can now correctly identify Azores**.\n",
    "\n",
    "Not only that, but since our example sentences also included references to Madeira, both archipelagos are now correctly identified!\n",
    "\n",
    "Let's check this in a more visual way to finish this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">if that between \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    America\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #ff9561; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Europe\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">LOC</span>\n",
       "</mark>\n",
       " is ample, will that between the \n",
       "<mark class=\"entity\" style=\"background: #ff9561; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Continent\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">LOC</span>\n",
       "</mark>\n",
       " and the \n",
       "<mark class=\"entity\" style=\"background: #ff9561; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Azores\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">LOC</span>\n",
       "</mark>\n",
       ", or \n",
       "<mark class=\"entity\" style=\"background: #ff9561; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Madeira\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">LOC</span>\n",
       "</mark>\n",
       ", or the \n",
       "<mark class=\"entity\" style=\"background: #ff9561; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Canaries\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">LOC</span>\n",
       "</mark>\n",
       ", or \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Ireland\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ", be sufficient?</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displacy.render(doc, style='ent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In conclusion, we were able to update our NER pipeline component in a way that it correctly identifies the token Azores and the entity associated with it.\n",
    "\n",
    "In order to do this, we have used the **Wikipedia text about Azores** and **annotated it with INCEpTION, using spaCy as an external recommender**. With this approach, we were able to include training examples containing the token Azores, but also other entities of numerous types that were already correctly recognized by the model. This helped us to prevent the problem of the model \"forgetting\" how to annotate the latter entities.\n",
    "\n",
    "After **converting the training examples into the desired format**, we were then able to use them to **retrain our model and update the NER pipeline component**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}