Trove Aboriginal Advocate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Named Entity Recognition in the Trove Aboriginal Advocate\n",
    "\n",
    "This data source is a XML format dump from the NLA Trove archive of one title - the Aboriginal Advocate.  The data is in the form of a large XML file contianing 3497 articles from this title.  The goal here is to run a named entity recognition process over the documents to extract names of interest. \n",
    "\n",
    "As with other notebooks in this project we will use the SpaCy language processing library to extract names from the text.  The first step is to define a reader for the XML data, this is done in the module [trovereader.py](trovereader.py) which is then imported here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install -q -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "import csv\n",
    "from urllib.request import urlopen\n",
    "import os\n",
    "import geocoder\n",
    "import pandas as pd\n",
    "import networkx as nx\n",
    "import matplotlib.pylab as plt\n",
    "\n",
    "%load_ext autoreload\n",
    "# import these local modules and reload them if they change\n",
    "%aimport trovereader\n",
    "%aimport utils\n",
    "%autoreload 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the source XML filename - shared publicly on cloudstor\n",
    "xmlurl = \"https://cloudstor.aarnet.edu.au/plus/s/WEAzWn7kqORh8yD/download?path=%2F&files=nla.obj-573721295_Aborginies_Advocate.xml\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download the spacy model we need\n",
    "model = 'en_core_web_lg'\n",
    "#spacy.cli.download(model)\n",
    "nlp = spacy.load(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell uses the trove XML parser to read the separate document records in the XML file and run the NER system over these. The resulting entities are collected into a list of dictionaries which is then converted to a Pandas DataFrame.   We collect all entities that are found and for each one store a bit of context - the entity plus two tokens either side of it.  \n",
    "\n",
    "Since finding entities can take some time, we write out the result to a CSV file.  We first check if the CSV file already exists and if it does, we read the entities from the file rather than recomputing them.  To force the NER process to run, set the variable `force` to `True` at the top of the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with urlopen(xmlurl) as xmlfile:\n",
    "    df = trovereader.trove_to_dataframe(xmlfile)\n",
    "\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set to True if you want to recompute the entities even if the\n",
    "# spreadsheet is present, False to read from the spreadsheet if it is there\n",
    "force_ner = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "csvname = \"data/Aboriginies_Advocate_Entities.csv\"\n",
    "\n",
    "if not force_ner and os.path.exists(csvname):\n",
    "    entities = pd.read_csv(csvname)\n",
    "else:\n",
    "    entities = utils.apply_ner(df, textcol='description', ident='identifier', context=5)\n",
    "    entities.head()\n",
    "    entities.to_csv(csvname, index=False)\n",
    "    \n",
    "entities.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having extracted the entities we can now explore what we have found. Here we look at the locations (GPE) and oganisations (ORG) and see what the most frequent 30 entities are in each case. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "locations = entities[entities.type == \"GPE\"]\n",
    "locations.entity.value_counts()[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "orgs = entities[entities.type == \"ORG\"]\n",
    "orgs.entity.value_counts()[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mention_graph = nx.DiGraph()\n",
    "mention_graph = nx.from_pandas_edgelist(entities, source='id', target='entity', edge_attr=True, create_using=mention_graph)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mention_graph.number_of_nodes(), mention_graph.number_of_edges()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = []\n",
    "for name, group in locations.groupby('id'):\n",
    "    locs = group.entity\n",
    "    for place1 in group.entity:\n",
    "        for place2 in group.entity:\n",
    "            if not place1 == place2:\n",
    "                result.append({'loc1': place1, 'loc2': place2, 'doc': name})\n",
    "\n",
    "colloc = pd.DataFrame(result).sort_values(['loc1','loc2'])\n",
    "colloc.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "collocg = colloc.groupby(['loc1', 'loc2']).count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\n",
    "    'loc1': collocg.index.get_level_values('loc1'), \n",
    "    'loc2': collocg.index.get_level_values('loc2'), \n",
    "    'count': collocg.doc,\n",
    "    'weight': [1/c for c in collocg.doc]   # weight is inverse of count\n",
    "})\n",
    "df.index = range(df.shape[0])  # reset index to integers\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select those collocations that occur more than 10 times\n",
    "df10 = df[df['count'] > 10]\n",
    "df10.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cg = nx.from_pandas_edgelist(df10, source='loc1', target='loc2', edge_attr=True, create_using=nx.DiGraph())\n",
    "cg.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "node = \"Kingswood\"\n",
    "nodes = cg.neighbors(node)\n",
    "nodes = list(nodes)\n",
    "nodes.append(node)\n",
    "subg = cg.subgraph(nodes)\n",
    "print(\"Subgraph of nodes linked to\", node, \"contains\", subg.size(), \"nodes\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,20))\n",
    "pos = nx.kamada_kawai_layout(subg, weight=\"weight\")\n",
    "colours = [(n==node and'g') or 'y' for n in subg.nodes]\n",
    "ecol = [e[2] for e in subg.edges.data('weight')]\n",
    "ecol = 'grey'\n",
    "nx.draw_networkx(subg, pos, edge_color=ecol, font_weight=\"bold\", font_color='r', arrows=False, node_color=colours)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "df10[df10.loc1==node].sort_values('count', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df10.to_csv(\"advocate-locations-collocation.csv\")\n",
    "locations.to_csv(\"advocate-locations.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "locations[locations['entity'] == 'Kingswood'].head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}