Day1_Introduction_to_NLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Day1-Introduction-to-NLP.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "gOVvHiWnZB7a",
        "NbCkV3WlDKuw"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# (Day 1) Introduction to Natural Language Processing\n",
        "\n",
        "Welcome to Day 1 of the Language Science Summer School 2022! \n"
      ],
      "metadata": {
        "id": "gOVvHiWnZB7a"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "In this short course, we will show you a glimpse of some of the cool stuff researchers have been doing with Natural Language Processing (NLP) systems. \n",
        "From story generation ([1](https://transformer.huggingface.co/)), to paraphrasing, or automatic reporting based on hundreds of documents ([2](https://www.askviable.com/)), or even helping customers filling their taxes ([3](https://www.keepertax.com/)), NLP research is assisting and automating decisions in various domains. \n"
      ],
      "metadata": {
        "id": "yr1VDMDLAQKm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "## Agenda for today\n",
        "\n",
        "We split this interactive notebook into 3 parts. In each part, we will ask you to interact with an NLP model and find interesting aspects.\n",
        "\n",
        "- [15 min] Demo 1. Natural Language Inference\n",
        "- [15 min] Demo 2. Question Generation\n",
        "- [15 min] Demo 3. Story generation"
      ],
      "metadata": {
        "id": "0qpyubSfAQPr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "## What to expect?\n",
        "\n",
        "Before jumping into all the fun, we would like to briefly explain the format of these notebooks. Everyday, we will have an interactive notebook concerning the topics you learned about in that same day. These notebooks are meant to be interactive and fun :) The idea is that you will interact and use the techniques, you've discussed in class and try to find some advantages or limitations of these NLP systems. It turns out that, even though we are able to do amazing things using NLP, some models are still pretty wrong in some circumstances!\n",
        "\n",
        "Don't worry! You won't do it alone! We're here to help and guide you!\n",
        "\n"
      ],
      "metadata": {
        "id": "DYbZSfNXAQN2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Part 1. Natural Language Inference\n",
        "\n",
        "In this section, we will use a large language model to automatically determine whether a given *hypothesis* follows logically from a *premise*. In other words, given that you only know the *premise*, can we infer the *hypothesis* from it? Is it true? \n"
      ],
      "metadata": {
        "id": "qxnVfpn3ceON"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Install and import libraries\n",
        "\n",
        "For this part of the notebook, we will use HuggingFace Transformers library -- a popular framework amongst NLP researchers that provides out-of-the-box implementations of current state-of-the-art models.\n",
        "\n",
        "Before using it, we need to install it.\n",
        "\n"
      ],
      "metadata": {
        "id": "K6BsKsmKfu0R"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "%%capture\n",
        "!pip install sentencepiece\n",
        "!pip install transformers"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4nK2kkHjfsLg",
        "outputId": "013b908f-1113-4956-f658-5e53561855d8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 102 ms, sys: 30.3 ms, total: 132 ms\n",
            "Wall time: 14.4 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.nn import functional as F\n",
        "from transformers import BertTokenizer, BertForSequenceClassification"
      ],
      "metadata": {
        "id": "EGfRofsVftZu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will load a variant of the BERT language model that was fine tuned on a dataset with thousands of NLI examples."
      ],
      "metadata": {
        "id": "3NENI31Ms9MZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "%%capture\n",
        "model_name = \"aloxatel/bert-base-mnli\" \n",
        "\n",
        "# Load the tokenizer\n",
        "tokenizer2 = BertTokenizer.from_pretrained(model_name)\n",
        "\n",
        "# Load the model\n",
        "model2 = BertForSequenceClassification.from_pretrained(model_name, num_labels=3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "NheJUmaLs7JG",
        "outputId": "2aca6f91-a4be-47d1-aa11-d6035d07cfb0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 9.33 s, sys: 1.55 s, total: 10.9 s\n",
            "Wall time: 15.2 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def get_nli_class(premise: str, hypothesis: str, with_score: bool=False):\n",
        "  \"\"\"\n",
        "  Given a model, determine whether the hypothesis is inferred from the premise.\n",
        "  \"\"\"\n",
        "  labels = [\"contradiction\", \"entailment\", \"neutral\"]\n",
        "\n",
        "  encoded = tokenizer2.encode_plus(premise + hypothesis, max_length=100, padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n",
        "  outputs = model2(**encoded)\n",
        "\n",
        "  outputs = F.softmax(outputs.logits, dim=-1)\n",
        "  outputs = outputs.detach().numpy()\n",
        "  \n",
        "  # Retrieve the argmax\n",
        "  label_ix = outputs.argmax()\n",
        "  print(\"Premise:\", premise, \"\\t | Hypothesis:\", hypothesis)\n",
        "  print(\"NLI label:\", labels[label_ix])\n",
        "  if with_score:\n",
        "    print(\"NLI score:\", round(outputs[0][label_ix], 3))"
      ],
      "metadata": {
        "id": "V7QukZC7DsSx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Natural Language Inference (NLI)\n",
        "\n",
        "\n",
        "In NLI, we are given two sentences: (1) a *premise* and (2) a *hypothesis*. The model must determine whether it is:\n",
        "- **entailment**: if the *hypothesis* follows logically from *premise*;\n",
        "- **contradiction**: if the *hypothesis* contradicts the *premise*;\n",
        "- **neutral**: if the *hypothesis* has no effect on the *premise*.\n",
        "\n",
        "A few examples are:\n",
        "\n",
        "| Premise | Hypothesis | Label |\n",
        "| ------- | ---------- | ----- |\n",
        "| This church choir sings to the masses as they sing joyous songs from the book at a church. | The church is filled with song.  | Entailment |\n",
        "| This church choir sings to the masses as they sing joyous songs from the book at a church. | A choir singing at a baseball game. | Contradiction |\n",
        "| This church choir sings to the masses as they sing joyous songs from the book at a church.     | The church has cracks in the ceiling.        | Neutral |\n",
        "\n",
        "\n",
        "\n",
        "In this part of the notebook, your task will be to use the method `get_nli_class` to find interesting model behaviors. You can do so by tweaking the `premise` or `hypothesis` parameters of the method. Here, is an example:\n"
      ],
      "metadata": {
        "id": "yKbv3HE6vQuK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "premise = \"The queen of England is bald.\"\n",
        "hypothesis = \"The queen is a man.\"\n",
        "\n",
        "get_nli_class(premise, hypothesis, True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bm4lAZK1r-jQ",
        "outputId": "006d9340-b1bc-4c24-d81e-22eb7578a217"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Premise: The queen of England is bald. \t | Hypothesis: The queen is a man.\n",
            "NLI label: entailment\n",
            "NLI score: 0.854\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercises\n",
        "\n",
        "### [3 min] Warmup\n",
        "\n",
        "In the next 2 to 3 minutes, play around with the to get familiar with the model. Try changing the inputs to the model, i.e., the *premise* and/or the *hypothesis*. \n",
        "\n"
      ],
      "metadata": {
        "id": "RQ2Fn3V3Wl75"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "premise = ... # TODO: Write a premise\n",
        "hypothesis = ... # TODO: Write an hypothesis\n",
        "\n",
        "get_nli_class(premise=premise, hypothesis=hypothesis)"
      ],
      "metadata": {
        "id": "g45NYm0eYDRZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "### [12 min] Exercise \n",
        "\n",
        "In groups, use the function `get_nli_class` and try to come up with examples where: \n",
        "\n",
        "- the model succeeds but humans would find hard.\n",
        "- the model fails and humans would find hard.\n",
        "- the model fails but humans would find easy.\n",
        "\n",
        "Find **at least one example** for each of the bullet points above."
      ],
      "metadata": {
        "id": "NbCkV3WlDKuw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Examples where **model succeeds and humans would find hard**"
      ],
      "metadata": {
        "id": "5YOHOKGfDVzX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "premise = ... # TODO: Write a premise\n",
        "hypothesis = ... # TODO: Write an hypothesis\n",
        "\n",
        "get_nli_class(premise=premise, hypothesis=hypothesis)"
      ],
      "metadata": {
        "id": "tz2IsSGRDa2T"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Examples where **model fails and humans would find hard**\n"
      ],
      "metadata": {
        "id": "28Kofwr6YDjO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "premise = ... # TODO: Write a premise\n",
        "hypothesis = ... # TODO: Write an hypothesis\n",
        "\n",
        "get_nli_class(premise=premise, hypothesis=hypothesis)"
      ],
      "metadata": {
        "id": "AX40n37tYDrP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "- Examples where **model fails but humans would find easy**\n"
      ],
      "metadata": {
        "id": "5RoLvq8bYDy0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "premise = ... # TODO: Write a premise\n",
        "hypothesis = ... # TODO: Write an hypothesis\n",
        "\n",
        "get_nli_class(premise=premise, hypothesis=hypothesis)"
      ],
      "metadata": {
        "id": "d0WUgNAeYD8J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Part 2. Generating questions\n",
        "\n",
        "In this section, we will resort to a different NLP model to generate questions automatically for us. Yup, that's right! This model is able to pose questions for our answers.\n"
      ],
      "metadata": {
        "id": "i85KbEmncehu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Install and import libraries\n",
        "\n",
        "In this section, we install and import the necessary frameworks. We also preload the models that will be used for the demo."
      ],
      "metadata": {
        "id": "d9qvi9xAcw82"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "%%capture\n",
        "from transformers import AutoModelWithLMHead, AutoTokenizer\n",
        "\n",
        "model_name = \"mrm8488/t5-base-finetuned-question-generation-ap\"\n",
        "\n",
        "# Download tokenizer \n",
        "tokenizer3 = AutoTokenizer.from_pretrained(model_name)\n",
        "\n",
        "# Download language model\n",
        "model3 = AutoModelWithLMHead.from_pretrained(model_name)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 141
        },
        "id": "u5_Jc0CacwHm",
        "outputId": "47f8ce68-a46f-4b9f-f032-832d9ce24a08"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n",
            "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n",
            "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n",
            "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 20.6 s, sys: 3.68 s, total: 24.3 s\n",
            "Wall time: 36.6 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def get_question(context, answer, max_length=64):\n",
        "  input_text = f\"answer: {answer}  context: {context} </s>\"\n",
        "  features = tokenizer3([input_text], return_tensors='pt')\n",
        "\n",
        "  output = model3.generate(input_ids=features['input_ids'], \n",
        "               attention_mask=features['attention_mask'],\n",
        "               max_length=max_length)\n",
        "\n",
        "  question = tokenizer3.decode(output[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)\n",
        "  print(question)\n",
        "  print()\n",
        "  return question\n"
      ],
      "metadata": {
        "id": "IK2MzG36cNY5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Question Generation\n",
        "\n",
        "Let us now discuss how we can interact with our model. The model expects two different inputs: \n",
        "- **context**: two or three sentences that you will base the question and answers on. \n",
        "- **answer**: what you want the question to be about. \n",
        "\n",
        "\n",
        "Note that, for this particular model, it is better to specify *answers* that are contained in the provided *context*. \n",
        "\n",
        "In this part of the notebook, your task will be to use the method `generate_question` to find interesting model behaviors. You can do so by tweaking the `context` or `answer` parameters of the method. Here, is an example:"
      ],
      "metadata": {
        "id": "udK0wnv2dhuG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time \n",
        "context = \"Welcome to the Cognitive Science Summer School 2022.\"\\\n",
        "          \"Catarina and Sameer are excited to tell you all about the\"\\\n",
        "          \"current research in NLP. Team up in groups of 4 and get ready\"\\\n",
        "          \"for five days of fun.\"\n",
        "answer = \"to tell you all about the NLP research\"\n",
        "\n",
        "get_question(answer, context)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rp_iycLvg1J7",
        "outputId": "1ed5ee28-3c48-4fc4-dbc4-b57a35c16332"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "question: What is the purpose of the Cognitive Science Summer School 2022?\n",
            "\n",
            "CPU times: user 3.26 s, sys: 363 ms, total: 3.62 s\n",
            "Wall time: 3.62 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercises\n",
        "\n",
        "### [3 min] Warmup\n",
        "\n",
        "In the next 2 to 3 minutes, play around with the to get familiar with the model. Try changing the inputs to the model, i.e., the *premise* and/or the *hypothesis*. \n"
      ],
      "metadata": {
        "id": "enlA8ACWdeGd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time \n",
        "context = ... # TODO: change the context\n",
        "answer = ... # TODO: change the answer\n",
        "\n",
        "get_question(answer, context)"
      ],
      "metadata": {
        "id": "O0lPyGcXSu9x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "### [12 min] Exercise \n",
        "\n",
        "In groups, use the function `get_question` and try to come up with examples where: \n",
        "\n",
        "- the model fails and humans would find hard.\n",
        "- the model fails but humans would find easy.\n",
        "- the model succeeds but humans could struggle.\n",
        "\n",
        "Find **at least one example** for each of the bullet points above."
      ],
      "metadata": {
        "id": "AToCpCq-SuWK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- the model fails and humans would also find hard.\n"
      ],
      "metadata": {
        "id": "FyIzNbDNjJw8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "context = ... # TODO: Write context\n",
        "answer = ... # TODO: Write an answer\n",
        "\n",
        "get_question(answer, context)"
      ],
      "metadata": {
        "id": "TrHSObJqjJ4d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "- the model fails but humans would find easy.\n"
      ],
      "metadata": {
        "id": "OWnvROnKjKEl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "context = ... # TODO: Write context\n",
        "answer = ... # TODO: Write an answer\n",
        "\n",
        "get_question(answer, context)"
      ],
      "metadata": {
        "id": "0kdPOHglidCJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "- the model succeeds but humans could find hard."
      ],
      "metadata": {
        "id": "LYW2kCI3jJeq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "context = ... # TODO: Write context\n",
        "answer = ... # TODO: Write an answer\n",
        "\n",
        "get_question(answer, context)"
      ],
      "metadata": {
        "id": "sJw23vxajJpa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Part 3. Story generation \n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "6Kz8KI_ivbhG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this part, we will ask you to interact with a large language model and use it to **generate different stories**. \n",
        "We will interact with a demo made available at [HuggingFace Spaces](https://huggingface.co/spaces/merve/GPT-2-story-gen).\n"
      ],
      "metadata": {
        "id": "cIOE2oJc_xU1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Story Generation \n",
        "\n",
        "In this demo, the model has a single input -- that we will tweak with! \n",
        "-  **prompt** is the text that you provide to the model and that will be used to generate a short story. Note, that this prompt will be the prefix of the generated story and the model will only append content to the story.\n",
        "\n",
        "After defining your prompt and clicking *\"Submit\"*, the model will generate a story.\n"
      ],
      "metadata": {
        "id": "RXI8B07ijCUx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## [15 min] Exercise\n",
        "\n",
        "In groups, use the story generation [demo](https://huggingface.co/spaces/merve/GPT-2-story-gen) described before to find examples where the output is interesting or wrong. In particular, try to come up with prompts that you wouldn't expect the model to perform well at. "
      ],
      "metadata": {
        "id": "0U1nUrOO7B_v"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Copy this cell and fill it in with your own examples! \n",
        "\n",
        "```\n",
        "# Example 1.\n",
        "prompt:\n",
        "category:\n",
        "generated text: \n",
        "```\n",
        "\n",
        "\n",
        "```\n",
        "# Example 2.\n",
        "prompt:\n",
        "category:\n",
        "generated text: \n",
        "```"
      ],
      "metadata": {
        "id": "SX2hdZ6n8woh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Concluding remarks\n",
        "\n",
        "In today's notebook, we explored a few applications of different language models (including [BERT](https://arxiv.org/abs/1810.04805), [GPT2](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf), and [T5](https://github.com/google-research/text-to-text-transfer-transformer)).\n",
        "Just as these demos, many other applications have been proposed and we challenge you to head over to [HuggingFace Spaces](https://huggingface.co/spaces?sort=likes) and have fun interacting with other applications, such as  [news-article summarization](https://huggingface.co/spaces/nickmuchi/article-text-summarizer), [rap lyrics generation](https://huggingface.co/spaces/Cropinky/gpt2-rap-songs), among others.\n",
        "Optionally, if you're more into gaming 🎮, then [these AI-generated adventures](https://play.aidungeon.io/main/home) may be more exciting to explore!\n",
        "\n",
        "\n",
        "As a last note, we would like to point you to an article that was fully generated by a GPT-base model. If you thought the generations in part 1 of this notebook were dull, then you must read this article titled [_A robot wrote this entire article. Are you scared yet, human?_](https://www.theguardian.com/commentisfree/2020/sep/08/robot-wrote-this-article-gpt-3)."
      ],
      "metadata": {
        "id": "QtNy5qNJt_qQ"
      }
    }
  ]
}