reverse_engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping the MSA through Reverse Engineering\n",
    "\n",
    "To get the data used in this module, we used an online website which had many databases based on Slavery in Maryland. We  focused on Run-Away Advertisement to be the primary topic for this module. We used the data by extracting it from the database on the website, which is called \"Web Scraping\". Web Scraping a web page involves fetching it and extracting from it. Fetching is the downloading of a page (which a browser does when you view the page). Once fetched, then extraction can take place. This notebook shows how we got the data, which we used for the other notebooks in this module. In the notebook, we will explain the data transcription process, the Legacy of Slavery website (the website we got the data from), technical challenges which were involved with Web Scraping, and the code for our web scraper. \n",
    "\n",
    "# Data Transcription\n",
    "\n",
    "To get this data, we used the online database which listed all Run-Away Slave Advertisements in Maryland. \n",
    "\n",
    "The website we used to get this data is: http://slavery2.msa.maryland.gov/pages/Search.aspx\n",
    "\n",
    "Slavery in Maryland lasted around 200 years from its beginnings in 1642. Most of the advertisements were posted on local Newspapers as well as advertisements posted on boards around the cities. Most of the data was transcribed by scanning local newspapers as well as scanning various advertisements posted on boards about run-away slaves. After the scanning process was finished, data was then entered into the database on the website.\n",
    "\n",
    "Here is a screenshot of the website we are web scraping from:\n",
    "\n",
    "![screenshot of MSA website](./first_detail_page.png)\n",
    "\n",
    "\n",
    "# Legacy of Slavery Website  \n",
    "\n",
    "The Legacy of Slavery in Maryland website consists of databases, maps, case studies, records, and more about this topic. The database used for this module was published on this website. The website was designed for normal web browsing, meaning that any user could find and browse data on the website. For this module, we did not have access to any backend databases. Thus, we had to somehow extract the data from the website so that we could use it in this module. Since we wanted to work with only the Run-Away Slave Advertisement Data, we had to extract this data by Web Scraping. Web Scraping a web page involves fetching it and extracting from it. Fetching is the downloading of a page (which a browser does when you view the page). Once fetched, then extraction can take place. The content of a page may be parsed, searched, reformatted, its data copied into a spreadsheet, and so on. Web scrapers typically take something out of a page, to make use of it for another purpose somewhere else.\n",
    "\n",
    "# Technical Challenges with Web Scraping\n",
    "\n",
    "While we were extracting data from the database, we encountered a few technical challenges with Web Scraping from the website. There were challenges making programmatic requests on the website. Futhermore, pagination was also a particular issue. Since there were multiple pages in the database, it was challenging to put all the information in one, large dataset. \n",
    "\n",
    "To resolve this issue, we found an article which explained our problem. The article provided us a solution. The article which helped us resolve our issue is: http://toddhayton.com/2015/05/04/scraping-aspnet-pages-with-ajax-pagination/\n",
    "\n",
    "We read the article thoroughly and learned how to resolve this particular issue. We took the generalized code from the article and then adapted it to our module. This article helped us resolve our issue on pagination and let us Webscrape more thoroughly. As a result, we Webscraped from the multiple pages and created a single, large dataset of all Run-Away Slave Advertisements.\n",
    "\n",
    "# Our Web Scraper \n",
    "\n",
    "The code for the Web Scraper was written by Todd Hayton and Greg Jansen. We first imported Python libraries: \"re\", \"urlparse\", and \"mechanize\". We also imported a library called \"Beautiful Soup\" which helped us with the main extraction of data. This library pulls data out of HTML and XML files. We then define the scrape function which is the primary function in this program which scrapes the data from the website. We start with the data extraction from the first page. After the program is done extracting the data from the first page, it moves to the second page, and so on. The program looks for the \"Next Page\" button on the webpage when it is finished extracting data from a page. The program keeps running until the \"Next Page\" button is not available. Once the data extraction is finished, all of the data is saved in one, large dataset.\n",
    "\n",
    "The code below demonstrates how we Web Scraped from the Legacy of Slavery website. \n",
    "\n",
    "Here is the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!/usr/bin/env python\n",
    "\n",
    "\"\"\"\n",
    "Python script for scraping the results from http://slavery2.msa.maryland.gov/pages/Search.aspx?\n",
    "\n",
    "Based on an informative article by Tom Hayton:\n",
    "  http://toddhayton.com/2015/05/04/scraping-aspnet-pages-with-ajax-pagination/\n",
    "\"\"\"\n",
    "\n",
    "__author__ = 'Greg Jansen'\n",
    "__author__ = 'Todd Hayton'\n",
    "\n",
    "import re\n",
    "from urllib.parse import urlparse\n",
    "import mechanize\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "class LegacyScraper(object):\n",
    "    def __init__(self):\n",
    "        self.url = \"http://slavery2.msa.maryland.gov/pages/Search.aspx?browse=run&page=1&pgsize=50&id=ArchivalResearch_LoSC_RunawayAds\"\n",
    "        print(self.url)\n",
    "        self.br = mechanize.Browser()\n",
    "        self.br.addheaders = [('User-agent',\n",
    "                               'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7')]\n",
    "\n",
    "    def scrape(self):\n",
    "        '''\n",
    "        Scrape all of the runaway slave ads\n",
    "        '''\n",
    "        self.br.open(self.url)\n",
    "\n",
    "        # GET DATABASE BROWSE LIST\n",
    "        raw = self.br.response().read()\n",
    "        s = BeautifulSoup(raw, \"lxml\")\n",
    "        saved_form = s.find('form', { \"id\": 'ctl00'}).prettify()\n",
    "\n",
    "        # SUBMIT FORM: RADIOBUTTON Click \"Detailed\"\n",
    "        for f in self.br.forms():\n",
    "            self.br.form = f\n",
    "        self.br.form.new_control('hidden', '__ASYNCPOST', {'value': 'true'})\n",
    "        self.br.form.new_control('hidden', 'ctl00$CenterContentPlaceHolder$ScriptManager1', {'value': 'ctl00$CenterContentPlaceHolder$UpdatePanel1|ctl00$CenterContentPlaceHolder$rblDisplayMode$2'})\n",
    "        self.br.form.fixup()\n",
    "        self.br.form.set_all_readonly(False)\n",
    "        rm = [ 'ctl00$CenterContentPlaceHolder$btnTabBasic',\n",
    "          'ctl00$CenterContentPlaceHolder$btnTabAdvanced',\n",
    "          'ctl00$CenterContentPlaceHolder$btnTabBrowse',\n",
    "          'ctl00$CenterContentPlaceHolder$btnTabDynamic',\n",
    "          'ctl00$CenterContentPlaceHolder$btnTabIndex',\n",
    "          'ctl00$CenterContentPlaceHolder$btnSubBasicSearch',\n",
    "          'ctl00$CenterContentPlaceHolder$btnSubBasicClear']\n",
    "        for ctl in rm:\n",
    "            ctl = self.br.form.find_control(ctl)\n",
    "            self.br.form.controls.remove(ctl)\n",
    "        self.br.form['__EVENTTARGET'] = 'ctl00$CenterContentPlaceHolder$rblDisplayMode$2'\n",
    "        self.br.form['__EVENTARGUMENT'] = ''\n",
    "        self.br.form['ctl00$CenterContentPlaceHolder$rblDisplayMode'] = ['detail']\n",
    "        print('\\n'.join(['%s:%s (%s)' % (c.name,c.value,c.disabled) for c in self.br.form.controls if c.name != '__VIEWSTATE']))\n",
    "        self.br.submit()\n",
    "\n",
    "        pageno = 1\n",
    "        while True:\n",
    "            if pageno > 2:\n",
    "                break\n",
    "            resp = self.br.response().read()\n",
    "            f = lambda A, n=4: [A[i:i+n] for i in range(0, len(A), n)]  # decodes XHR response\n",
    "            a = f(resp.split('|'))\n",
    "            content = None\n",
    "            view_state = None\n",
    "            for b in a:\n",
    "                if len(b) == 4:\n",
    "                    if b[1] == 'updatePanel':\n",
    "                        content = b[3]\n",
    "                    if b[2] == '__VIEWSTATE':\n",
    "                        print('GOT VIEWSTATE')\n",
    "                        view_state = b[3]\n",
    "            self.save_page(content, pageno)\n",
    "\n",
    "            # Check if last page\n",
    "            s = BeautifulSoup(content, 'lxml')\n",
    "            next_button = s.find('input', { 'title': 'Next Grayed Out'})\n",
    "            if next_button is not None:\n",
    "                break\n",
    "            pageno += 1\n",
    "\n",
    "            # Regenerate form for next page\n",
    "            html = saved_form.encode('utf8')\n",
    "            resp = mechanize.make_response(html, [(\"Content-Type\", \"text/html\")],\n",
    "                                           self.br.geturl(), 200, \"OK\")\n",
    "            self.br.set_response(resp)\n",
    "            form = None\n",
    "            for f in self.br.forms():\n",
    "                self.br.form = f\n",
    "            self.br.form.set_all_readonly(False)\n",
    "\n",
    "            event_target = 'ctl00$CenterContentPlaceHolder$UpdatePanel1|ctl00$CenterContentPlaceHolder$imgButtonNext'\n",
    "            self.br.form['__EVENTTARGET'] = event_target\n",
    "            self.br.form['__EVENTARGUMENT'] = ''\n",
    "            self.br.form['__VIEWSTATE'] = view_state\n",
    "            self.br.form.new_control('hidden', '__ASYNCPOST', {'value': 'true'})\n",
    "            self.br.form.new_control('hidden', 'ctl00$CenterContentPlaceHolder$ScriptManager1', {'value': 'ctl00$CenterContentPlaceHolder$UpdatePanel1|ctl00$CenterContentPlaceHolder$rblDisplayMode$2'})\n",
    "            self.br.form['ctl00$CenterContentPlaceHolder$rblDisplayMode'] = ['detail']\n",
    "            self.br.form['ctl00$CenterContentPlaceHolder$ddlPage'] = [ str(pageno) ]\n",
    "            self.br.form.fixup()\n",
    "            for ctl in rm:\n",
    "                ctl = self.br.form.find_control(ctl)\n",
    "                self.br.form.controls.remove(ctl)\n",
    "            print('\\n'.join(['%s:%s (%s)' % (c.name,c.value,c.disabled) for c in self.br.form.controls if c.name != '__VIEWSTATE']))\n",
    "            self.br.submit()\n",
    "\n",
    "\n",
    "    def save_page(self, rawcontent, pageno):\n",
    "        ###################################\n",
    "        # DATA SCRAPE/SAVE LOGIC GOES HERE\n",
    "        ###################################\n",
    "        print('Page {0}'.format(pageno))\n",
    "        text_file = open(\"pages/page-{0}.html\".format(pageno), \"w\")\n",
    "        text_file.write(rawcontent)\n",
    "        text_file.close()\n",
    "\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    scraper = LegacyScraper()\n",
    "    scraper.scrape()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}