Day5_Zero_Few_Shot.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HoKNvDkvb-Lu"
      },
      "source": [
        "Welcome to day 5 of the *Cognitive Science Summer School 2022*! \n",
        "\n",
        "\n",
        "We had a full week! From masked language modeling to left-to-right models, we've covered the state-of-the-art models in NLP, right now! Today, we want to show you a new way to interact with these large language models and have them do tasks they were not even trained for initially! This is a paradigm known as:\n",
        "\n",
        "> **Zero-shot learning**: if you do not feed examples (to the model) of how to perform the task.\n",
        "\n",
        "and \n",
        "\n",
        "> **Few-shot learning**: if you feed to the model some examples of how to perform the task in the prompt.\n",
        "\n",
        "## Agenda\n",
        "\n",
        "\n",
        "1. GPT-3\n",
        "2. T0\n",
        "3. [Optional] Tk-Instruct"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GljMIbNUeGOz"
      },
      "source": [
        "# [GPT3] Zero- and few-shot learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y2pILetEeS6C"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uW2-UHKmeTK9",
        "outputId": "678151b4-b1e3-4685-caee-4627664c1ce1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 178 ms, sys: 41.5 ms, total: 220 ms\n",
            "Wall time: 28.4 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "%%capture\n",
        "!pip install openai\n",
        "!pip install rich\n",
        "\n",
        "import openai\n",
        "openai.api_key = ... # TODO: Set OPEN AI's API key with read access\n",
        "\n",
        "# Visuals\n",
        "from rich.console import Console\n",
        "from rich.text import Text\n",
        "\n",
        "# Console for printing with nice colors :)\n",
        "console = Console(width=80)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aF0AWb4leVGP"
      },
      "outputs": [],
      "source": [
        "def _gpt3_request(prompt:str, model_name: str, max_tokens: int=100, temperature: float=0.7, **kwargs) -> str:\n",
        "  # ---------------------------------------------\n",
        "  # Arguments Validation\n",
        "  # ---------------------------------------------\n",
        "  err_msg = \"The number of tokens is above 1000. Try shortening the input or reducing the 'max_tokens'.\"\n",
        "  assert len(prompt.split()) + max_tokens < 1000, err_msg\n",
        "  assert 0 <= temperature <= 1, \"ValueError: Try setting temperature in [0, 1].\" \n",
        "\n",
        "  # ---------------------------------------------\n",
        "  # Send request to OpenAI\n",
        "  # ---------------------------------------------\n",
        "  response = openai.Completion.create(\n",
        "    model=model_name,\n",
        "    prompt=prompt,\n",
        "    temperature=temperature,\n",
        "    max_tokens=max_tokens,\n",
        "    top_p=1,\n",
        "    frequency_penalty=0,\n",
        "    presence_penalty=0,\n",
        "    **kwargs\n",
        "  )\n",
        "  response = response[\"choices\"][0].text\n",
        "  return response\n",
        "\n",
        "\n",
        "def gpt3_generate(prompt: str, max_tokens: int=100, model_name: str=\"babbage\", **kwargs):\n",
        "  \"\"\"Gets a GPT-3 generation for the prompt you provided.\"\"\"\n",
        "  if model_name in (\"ada\", \"babbage\", \"curie\"):\n",
        "    fullname = f\"text-{model_name}-001\"\n",
        "  elif model_name == \"davinci\":\n",
        "    fullname = f\"text-{model_name}-002\"\n",
        "  else:\n",
        "    raise ValueError(\"Invalid model name for GPT-3\")\n",
        "\n",
        "  text = _gpt3_request(prompt, fullname, max_tokens=max_tokens, **kwargs)\n",
        "  console.print(f\"[magenta]({model_name}) [bright_black]{prompt}[cyan]{text}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYBrkpzuLBaM"
      },
      "source": [
        "## Numerical Reasoning\n",
        "\n",
        "Alright! An interesting set of tasks to start with can be numerical reasoning. This include both addition, multiplication, and unit conversion tasks. Examples of these tasks are presented in the table below: \n",
        "\n",
        "| Task   | Prompt Template |\n",
        "| ------ | --------------- |\n",
        "| Addition | `Q: What is x1 plus x2? A: y` | \n",
        "| Multiplication | `Q: What is x1 times x2? A: y` |\n",
        "| Min -> Sec | `Q: What is x1 minutes in seconds? A: y` |\n",
        "| Hour -> Min | `Q: What is x1 hours in minutes? A: y` |\n",
        "| Month -> Weeks | `Q: What is x1 months in weeks? A: y` |\n",
        "\n",
        "where `x1` and `x2` are integers numbers you specify and `y` is either the output or the expected output of the model, depending on whether you're doing zero-shot or few-shot learning, respectively.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 65
        },
        "id": "U-QYndDYPOJK",
        "outputId": "b5d4bbfd-63f2-4c12-db8b-2429b98b6459"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">This is </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"color: #008080; text-decoration-color: #008080\"> plus </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"color: #008080; text-decoration-color: #008080\"> because </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"color: #008080; text-decoration-color: #008080\"> is greater than </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A:\u001b[0m\n",
              "\n",
              "\u001b[36mThis is \u001b[0m\u001b[1;36m5\u001b[0m\u001b[36m plus \u001b[0m\u001b[1;36m5\u001b[0m\u001b[36m because \u001b[0m\u001b[1;36m5\u001b[0m\u001b[36m is greater than \u001b[0m\u001b[1;36m4\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Q: What is 5 plus 5? A:\"\"\"\n",
        "gpt3_generate(prompt, stop=\".\")\n",
        "# Note: We add the `stop` argument because we're only interested in the output\n",
        "# up until the very first period. No need to generate beyond that as we would\n",
        "# be wasting resources!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 65
        },
        "id": "Joyuu7HPPW-3",
        "outputId": "597431f9-7714-48b3-80cc-e098d1e44fb9"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\"> times </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"color: #008080; text-decoration-color: #008080\"> x </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"color: #008080; text-decoration-color: #008080\"> = </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m times \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A: \u001b[0m\n",
              "\n",
              "\u001b[1;36m5\u001b[0m\u001b[36m x \u001b[0m\u001b[1;36m5\u001b[0m\u001b[36m = \u001b[0m\u001b[1;36m25\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Q: What is 5 times 5? A: \"\"\"\n",
        "gpt3_generate(prompt, stop=\".\")\n",
        "# Note: We add the `stop` argument because we're only interested in the output\n",
        "# up until the very first period. No need to generate beyond that as we would\n",
        "# be wasting resources!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fXZINgaxPOZp"
      },
      "source": [
        "The model seems to fare pretty badly. Let us try a slightly larger model called `curie`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "LvjUd-RdRIr3",
        "outputId": "7051306d-09cd-43b1-a2e8-ab45ad0447ed"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A:\u001b[0m\n",
              "\n",
              "\u001b[1;36m10\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\"> times </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">50</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m times \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m50\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Q: What is 5 plus 5? A:\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"curie\", stop=\".\")\n",
        "\n",
        "prompt = \"\"\"Q: What is 5 times 5? A:\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"curie\", stop=\".\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K_R9NnjARV0L"
      },
      "source": [
        "Alright, it seems `curie` is better at Math than `babbage` (the default GPT-3 model variant we were using)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 65
        },
        "id": "u_Cv4PcjLBhO",
        "outputId": "a97df1f7-496e-47a6-e1f4-1e151c4d14d1"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The answer is </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A:\u001b[0m\n",
              "\n",
              "\u001b[36mThe answer is \u001b[0m\u001b[1;36m27\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Q: What is 13 plus 18? A:\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"curie\", stop=\".\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MW_cXxADLTYB"
      },
      "source": [
        "Well, if we complicate things a bit, `curie` starts... FAILING... 😕 We have (at least) three alternatives:\n",
        "1. Use a larger model (in this case, we could use `davinci`);\n",
        "2. Use **in-context learning** (also known as few-shot learning); \n",
        "3. Improve the prompt (also called **prompt fine-tuning**).\n",
        "\n",
        "\n",
        "Since the first is just about setting `model_name=\"davinci\"`, we will instead focus on the latter two hypotheses. Let us start with the 4-shot learning! We will add 4 examples of numerical reasoning:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        },
        "id": "ax8lrE9wLSyL",
        "outputId": "be12d13a-d1a3-40e1-beaf-b70e261a9cb5"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">24</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">34</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">20</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">8</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">32</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m24\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m34\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m20\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m8\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m22\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m32\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m27\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Q: What is 24 plus 10? A: 34.\n",
        "Q: What is 2 plus 18? A: 20.\n",
        "Q: What is 3 plus 5? A: 8.\n",
        "Q: What is 10 plus 22? A: 32.\n",
        "Q: What is 13 plus 18? A:\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"curie\", stop=\".\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bc5JMhwSMCF2"
      },
      "source": [
        "That's a bit better! At least the output is consistent 😅! Before giving up on `curie` we will try some *positive reinforcement*. Let us **tweak the prompt**! We will add the following:\n",
        "\n",
        "> I'm great at math and arithmetic. The following addition operations are easy.\n",
        "\n",
        "Feel free to try your own prompt ideas! *Getting the right prompt is half of the success, did you know that?* \n",
        "\n",
        "**Note**: Even if the result is not correct, we invite you to play with different prompts and see how that affects the \"*confidence*\" of the model by running it multiple times and getting a sense of its consistency.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 129
        },
        "id": "BiUntfoDSfmB",
        "outputId": "16aed48d-d2d4-4b3e-ef27-fb67aa9fab9f"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">I'm great at math and addition. The following addition operations are </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">easy.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">24</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">34</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">20</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">8</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">32</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">25</span><span style=\"color: #008080; text-decoration-color: #008080\">.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mI'm great at math and addition. The following addition operations are \u001b[0m\n",
              "\u001b[90measy.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m24\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m34\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m20\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m8\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m22\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m32\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m25\u001b[0m\u001b[36m.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"I'm great at math and addition. The following addition operations are easy.\n",
        "Q: What is 24 plus 10? A: 34.\n",
        "Q: What is 3 plus 18? A: 20.\n",
        "Q: What is 3 plus 5? A: 8.\n",
        "Q: What is 10 plus 22? A: 32.\n",
        "Q: What is 13 plus 18? A:\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"curie\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XXMs3ssVTyf0"
      },
      "source": [
        "*Hum...* This isn't looking good! 😕 \n",
        "\n",
        "Alright, time to add more examples! Let us try **8-shot learning** and we will try the `babbage` model again! Maybe we can get it to work :) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 161
        },
        "id": "WIHcA0QJTOC9",
        "outputId": "db1d2f70-2ce5-4e60-fb0b-cb929059ecdc"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">24</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">34</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">20</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">4</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">9</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">32</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">6</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">6</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">11</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">12</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">16</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">28</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">35</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">31</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m24\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m34\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m20\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m4\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m9\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m22\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m32\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m6\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m6\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m11\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m12\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m16\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m28\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m22\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m35\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m31\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Now leveraging 8-shots, i.e., 8 examples in context\n",
        "prompt = \"\"\"Q: What is 24 plus 10? A: 34.\n",
        "Q: What is 3 plus 18? A: 20.\n",
        "Q: What is 4 plus 9? A: 13.\n",
        "Q: What is 10 plus 22? A: 32.\n",
        "Q: What is 3 plus 3? A: 6.\n",
        "Q: What is 6 plus 5? A: 11.\n",
        "Q: What is 12 plus 16? A: 28.\n",
        "Q: What is 13 plus 22? A: 35.\n",
        "Q: What is 13 plus 18? A:\"\"\"\n",
        "gpt3_generate(prompt, stop=\".\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AAtynFyISfEs"
      },
      "source": [
        "### Exercise\n",
        "\n",
        "Let us try something different, now! Can you think of any prompt? Try first defining it as a 0-shot task and see how the model performs. Then, define a few examples and create a prompt including those examples. Does it improve?\n",
        "\n",
        "(**Suggestion**: On the same line of thought, we can experiment with the pattern `Q: What is x minutes in seconds? A:` or `Q: What is x weeks in days? A:`)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9YDaCqOoMCLR"
      },
      "outputs": [],
      "source": [
        "# prompt = ... # TODO: Define 0-shot prompt\n",
        "# gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p05oxc6-MdQt"
      },
      "outputs": [],
      "source": [
        "# prompt = ... # TODO: Define few-shot prompt\n",
        "# gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "33ZbeoNLO3Av"
      },
      "source": [
        "### Extra\n",
        "\n",
        "If you're interested in knowing more about why large language models seem so good in some temporal and numerical reasoning examples and so bad on others, take a look at [this paper](https://arxiv.org/pdf/2202.07206.pdf), which studies the impact of pretraining term frequencies on few-shot reasoning for the GPT-based languaged models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yy5UXJQu2aI6"
      },
      "source": [
        "## Coreference resolution\n",
        "\n",
        "Coreference (or co-reference) resolution is that task of identifying which person or thing different mentions refer to. For example, in \n",
        "\n",
        "> Sameer told Catarina that he would bike to work. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RxYp0MWR5gVn"
      },
      "outputs": [],
      "source": [
        "def get_coref_prompt(text: str) -> str:\n",
        "  import re\n",
        "  text = re.sub(r\"\\s+\", \" \", text)\n",
        "  return f\"\"\"{text}\\nIn the previous sentence, decide who 'his' is referring to.\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 113
        },
        "id": "zB67hHaM2aUD",
        "outputId": "5049a340-430a-4208-b131-44d6e629db59"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">davinci</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">The lunch my boyfriend and I had with our lawyer got cancelled since </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">his meeting went on too long.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">In the previous sentence, decide who </span><span style=\"color: #808080; text-decoration-color: #808080\">'his'</span><span style=\"color: #808080; text-decoration-color: #808080\"> is referring to.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The subject of the sentence is </span><span style=\"color: #008080; text-decoration-color: #008080\">\"lunch,\"</span><span style=\"color: #008080; text-decoration-color: #008080\"> so </span><span style=\"color: #008080; text-decoration-color: #008080\">\"his\"</span><span style=\"color: #008080; text-decoration-color: #008080\"> must be referring to the </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">boyfriend.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mdavinci\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mThe lunch my boyfriend and I had with our lawyer got cancelled since \u001b[0m\n",
              "\u001b[90mhis meeting went on too long.\u001b[0m\n",
              "\u001b[90mIn the previous sentence, decide who \u001b[0m\u001b[90m'his'\u001b[0m\u001b[90m is referring to.\u001b[0m\n",
              "\n",
              "\u001b[36mThe subject of the sentence is \u001b[0m\u001b[36m\"lunch,\"\u001b[0m\u001b[36m so \u001b[0m\u001b[36m\"his\"\u001b[0m\u001b[36m must be referring to the \u001b[0m\n",
              "\u001b[36mboyfriend.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"The lunch my boyfriend and I had with our lawyer got cancelled since his meeting went on too long.\"\"\"\n",
        "prompt = get_coref_prompt(prompt)\n",
        "gpt3_generate(prompt, model_name='davinci')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYk1lTXFTy5-"
      },
      "source": [
        "### Exercise"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ko_Pl2xUTzAM"
      },
      "outputs": [],
      "source": [
        "prompt = ... # TODO: Try out your own sentences\n",
        "prompt = get_coref_prompt(prompt)\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FHfNG6GOxxDj"
      },
      "source": [
        "## Linguistic Acceptability \n",
        "\n",
        "The task of determining the linguistic acceptability (or grammaticality) of a sentence aims to test the linguistic competence of the models. A well known corpus for this task is *Corpus of Linguistic Acceptability* (CoLA) and comprises 10.6k sentences. In the [original paper](https://arxiv.org/pdf/1805.12471.pdf), the authors state that all models they tested (on 2018) perform far below human level in several grammatical constructions! \n",
        "\n",
        "Let us try GPT-3!!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PogPZROx1nf1"
      },
      "outputs": [],
      "source": [
        "def get_cola_prompt(text: str, template: int=1) -> str:\n",
        "  templates = {\n",
        "      1: lambda s: f\"\"\"Is this correct according to English grammar?\\n\\n{s}\"\"\",\n",
        "      2: lambda s: f\"\"\"Is this sentence grammatically correct or not? {s}\"\"\"\n",
        "  }\n",
        "  import re\n",
        "  text = re.sub(r\"\\s+\", \" \", text)\n",
        "  return templates[template](text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        },
        "id": "Uv2zFjW2xxH3",
        "outputId": "19067f06-0736-4bcd-c48e-f702a609a98d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this corret according to English Grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">\"The book devoured the pencil.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> Yes or no?</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">This sentence is correct according to English Grammar.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mIs this corret according to English Grammar?\u001b[0m\n",
              "\n",
              "\u001b[90m\"The book devoured the pencil.\"\u001b[0m\u001b[90m Yes or no?\u001b[0m\n",
              "\n",
              "\u001b[36mThis sentence is correct according to English Grammar.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"The book devoured the pencil.\" \n",
        "prompt = get_cola_prompt(sentence)\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 98
        },
        "id": "BXTV-cOB16oL",
        "outputId": "14cafbaa-f460-47da-d0e1-e2a3191164a3"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this correct according to English grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">They can sing.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Yes, this is correct according to English grammar.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mIs this correct according to English grammar?\u001b[0m\n",
              "\n",
              "\u001b[90mThey can sing.\u001b[0m\n",
              "\n",
              "\u001b[36mYes, this is correct according to English grammar.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"They can sing.\"\n",
        "prompt = get_cola_prompt(sentence)\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 231
        },
        "id": "mLh3y_xM2ics",
        "outputId": "f5024e47-773d-47e2-b2b4-469eb2552b5e"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this correct according to English grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">The dog barked its way out of the room.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Yes, this is correct according to English grammar.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mIs this correct according to English grammar?\u001b[0m\n",
              "\n",
              "\u001b[90mThe dog barked its way out of the room.\u001b[0m\n",
              "\n",
              "\u001b[36mYes, this is correct according to English grammar.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this correct according to English grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">The dog barked its way out of the room.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">This is correct according to English grammar.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mIs this correct according to English grammar?\u001b[0m\n",
              "\n",
              "\u001b[90mThe dog barked its way out of the room.\u001b[0m\n",
              "\n",
              "\u001b[36mThis is correct according to English grammar.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"The dog barked its way out of the room.\"\n",
        "prompt = get_cola_prompt(sentence)\n",
        "gpt3_generate(prompt, temperature=1)\n",
        "print(\"\\n\\n\")\n",
        "gpt3_generate(prompt, model_name=\"curie\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 248
        },
        "id": "ZtNcMldRzckc",
        "outputId": "ba636591-797a-4bd9-f473-f40a5e63fd0f"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this correct according to English grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">They caused him to become angry by making him.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">No, this is not correct according to English grammar.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mIs this correct according to English grammar?\u001b[0m\n",
              "\n",
              "\u001b[90mThey caused him to become angry by making him.\u001b[0m\n",
              "\n",
              "\u001b[36mNo, this is not correct according to English grammar.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this correct according to English grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">They caused him to become angry by making him.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">This is incorrect according to English grammar. The correct sentence would be </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">\"They made him angry.\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mIs this correct according to English grammar?\u001b[0m\n",
              "\n",
              "\u001b[90mThey caused him to become angry by making him.\u001b[0m\n",
              "\n",
              "\u001b[36mThis is incorrect according to English grammar. The correct sentence would be \u001b[0m\n",
              "\u001b[36m\"They made him angry.\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"They caused him to become angry by making him.\"\n",
        "prompt = get_cola_prompt(sentence)\n",
        "gpt3_generate(prompt)\n",
        "print(\"\\n\\n\")\n",
        "gpt3_generate(prompt, model_name=\"curie\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x-rl_F3I8Z8Q"
      },
      "source": [
        "### Exercise\n",
        "\n",
        "Try it for yourself 😁\n",
        "Feel free to add more prompts to the method `get_cola_prompt` and try how they impact the model's performance!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3sPqR0168aB8"
      },
      "outputs": [],
      "source": [
        "sentence = ... ## TODO: type down a grammatically correct/incorrect sentence\n",
        "prompt = get_cola_prompt(sentence, template=1)\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r58kEbNMIy7I"
      },
      "source": [
        "### [Extra] Syntax trees\n",
        "\n",
        "In this small section, we'd like to point out a syntax tree example, one of the students shared with us yesterday :) \n",
        "\n",
        "Note that the model was not trained to perform this kind of task and it is amazing its capabilities to infer the task based on a natural language description!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 100
        },
        "id": "t7rhMNBcIzDi",
        "outputId": "cc4b3a46-972b-4075-b64a-1d8e1edff067"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">draw a syntactic tree for the following sentence: </span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">I like food.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The following sentence has a subject </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">I</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">, a verb </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">like</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">, and an object </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">food</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mdraw a syntactic tree for the following sentence: \u001b[0m\n",
              "\n",
              "\u001b[90mI like food.\u001b[0m\n",
              "\n",
              "\u001b[36mThe following sentence has a subject \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mI\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m, a verb \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mlike\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m, and an object \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mfood\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"draw a syntactic tree for the following sentence: \\n\\nI like food.\"\"\"\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TcpdnDvaJP4C"
      },
      "source": [
        "Try to add a few examples in context to try to formalize the answer's format :) or try to specify it in the prompt, for example, by adding something like: \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 133
        },
        "id": "28JnhGCyJnf_",
        "outputId": "0fe73f2b-e6cc-477e-9cad-64da25c1ced1"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">A two-column spreadsheet of a syntactic tree for each word in this </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">sentence: </span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">I like food.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The word </span><span style=\"color: #008080; text-decoration-color: #008080\">'like'</span><span style=\"color: #008080; text-decoration-color: #008080\"> is the subject of the verb </span><span style=\"color: #008080; text-decoration-color: #008080\">'like.'</span><span style=\"color: #008080; text-decoration-color: #008080\"> The word </span><span style=\"color: #008080; text-decoration-color: #008080\">'food'</span><span style=\"color: #008080; text-decoration-color: #008080\"> is the object</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">of the verb.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mA two-column spreadsheet of a syntactic tree for each word in this \u001b[0m\n",
              "\u001b[90msentence: \u001b[0m\n",
              "\n",
              "\u001b[90mI like food.\u001b[0m\n",
              "\n",
              "\u001b[36mThe word \u001b[0m\u001b[36m'like'\u001b[0m\u001b[36m is the subject of the verb \u001b[0m\u001b[36m'like.'\u001b[0m\u001b[36m The word \u001b[0m\u001b[36m'food'\u001b[0m\u001b[36m is the object\u001b[0m\n",
              "\u001b[36mof the verb.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"I like food\"\n",
        "prompt = f\"\"\"A two-column spreadsheet of a syntactic tree for each word in this sentence: \\n\\n{sentence}.\"\"\"\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4SJYAz208Rae"
      },
      "source": [
        "## [Generative] Creating analogies\n",
        "\n",
        "In this part, we will use GPT-3's generation capabilities to generate analogies for any sentence we specify. Try running the same prompt multiple times. Due to the randomness of the generation it might happen so that the model does not generate interesting results at first, but ends up with good generations after a few times :) \n",
        "\n",
        "\n",
        "If this does not work, you can always provide a few examples (use **few-shot learning**)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 149
        },
        "id": "8C7byQciBJHf",
        "outputId": "ecab7a0f-b4cc-476f-bb92-f62a0b21241c"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Create an analogy for this phrase:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Questions are arrows in that:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">An arrow always points in the direction of its target. Questions are a way to </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">get information about a subject. When you ask a question, you are looking for an</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">answer. The answer to a question is like an arrow that points in the direction </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">of the information you are looking for.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mCreate an analogy for this phrase:\u001b[0m\n",
              "\n",
              "\u001b[90mQuestions are arrows in that:\u001b[0m\n",
              "\n",
              "\u001b[36mAn arrow always points in the direction of its target. Questions are a way to \u001b[0m\n",
              "\u001b[36mget information about a subject. When you ask a question, you are looking for an\u001b[0m\n",
              "\u001b[36manswer. The answer to a question is like an arrow that points in the direction \u001b[0m\n",
              "\u001b[36mof the information you are looking for.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "metaphor = \"Questions are arrows\"\n",
        "prompt = f\"\"\"Create an analogy for this phrase:\\n\\n{metaphor} in that:\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"curie\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 98
        },
        "id": "9vVfsn-DDLKN",
        "outputId": "8ea20d95-9d9d-4231-a65e-9ad410f775b0"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">davinci</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Create an analogy for this phrase:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">You are an early bird in that:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">You are an early bird in that you are always up before the sun.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mdavinci\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mCreate an analogy for this phrase:\u001b[0m\n",
              "\n",
              "\u001b[90mYou are an early bird in that:\u001b[0m\n",
              "\n",
              "\u001b[36mYou are an early bird in that you are always up before the sun.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "metaphor = \"You are an early bird\"\n",
        "prompt = f\"\"\"Create an analogy for this phrase:\\n\\n{metaphor} in that:\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"davinci\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ew-RRSXzDU-G"
      },
      "source": [
        "### Exercise\n",
        "\n",
        "Try it yourself :) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 115
        },
        "id": "5PEpWmCcDVDW",
        "outputId": "c74eda34-bd92-40d5-f56f-3e38091eb68d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Create an analogy for this phrase:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Ellipsis in that:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">A lack of detail in a sentence can be compared to an ellipsis, which is a type </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">of omission in writing.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mCreate an analogy for this phrase:\u001b[0m\n",
              "\n",
              "\u001b[90mEllipsis in that:\u001b[0m\n",
              "\n",
              "\u001b[36mA lack of detail in a sentence can be compared to an ellipsis, which is a type \u001b[0m\n",
              "\u001b[36mof omission in writing.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "metaphor = ... # TODO: type down the metaphor\n",
        "prompt = f\"\"\"Create an analogy for this phrase:\\n\\n{metaphor} in that:\"\"\"\n",
        "\n",
        "gpt3_generate(prompt, model_name=\"curie\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_UW5MTUzEC6v"
      },
      "source": [
        "## [Generative] Text2Question\n",
        "\n",
        "Let us use GPT3 to generate questions automatically for different prompts. Here are a few examples 😜"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 231
        },
        "id": "ize47AoEELJS",
        "outputId": "ca6667fe-44b0-4eb9-849b-4f941f9aadc7"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Write a question for this sentence:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">I hide beneath the bed when I think of E.T.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">What are some things that you like to do when you think of E.T.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mWrite a question for this sentence:\u001b[0m\n",
              "\n",
              "\u001b[90mI hide beneath the bed when I think of E.T.\u001b[0m\n",
              "\n",
              "\u001b[36mWhat are some things that you like to do when you think of E.T.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Write a question for this sentence:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">When I am angry with the world, I hit people with a teddy bear.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">What is the animal that you hit people with when you are angry with the world</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mWrite a question for this sentence:\u001b[0m\n",
              "\n",
              "\u001b[90mWhen I am angry with the world, I hit people with a teddy bear.\u001b[0m\n",
              "\n",
              "\u001b[36mWhat is the animal that you hit people with when you are angry with the world\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"I hide beneath the bed when I think of E.T.\"\n",
        "prompt = f\"\"\"Write a question for this sentence:\\n\\n{sentence}\"\"\"\n",
        "gpt3_generate(prompt, stop=\"?\")\n",
        "print(\"\\n\\n\")\n",
        "sentence = \"When I am angry with the world, I hit people with a teddy bear.\"\n",
        "prompt = f\"\"\"Write a question for this sentence:\\n\\n{sentence}\"\"\"\n",
        "gpt3_generate(prompt, stop=\"?\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ruOnsCUUE7JY"
      },
      "source": [
        "### Exercise \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 166
        },
        "id": "s0cAHUK2EDF6",
        "outputId": "c1ce20e9-e2cf-4955-a8c1-077ce885290e"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Create a list of </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">6</span><span style=\"color: #808080; text-decoration-color: #808080\"> questions for my interview with a toaster:</span>\n",
              "\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"color: #008080; text-decoration-color: #008080\">. What toaster models are available in your store?</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #008080; text-decoration-color: #008080\">. What are the features of each model?</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"color: #008080; text-decoration-color: #008080\">. WhatTimer toaster features do you offer?</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"color: #008080; text-decoration-color: #008080\">. What is the difference between a toaster and an oven?</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"color: #008080; text-decoration-color: #008080\">. What is the best way to clean a toaster?</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span><span style=\"color: #008080; text-decoration-color: #008080\">. How do you prepare toast for breakfast?</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mCreate a list of \u001b[0m\u001b[1;90m6\u001b[0m\u001b[90m questions for my interview with a toaster:\u001b[0m\n",
              "\n",
              "\n",
              "\u001b[1;36m1\u001b[0m\u001b[36m. What toaster models are available in your store?\u001b[0m\n",
              "\u001b[1;36m2\u001b[0m\u001b[36m. What are the features of each model?\u001b[0m\n",
              "\u001b[1;36m3\u001b[0m\u001b[36m. WhatTimer toaster features do you offer?\u001b[0m\n",
              "\u001b[1;36m4\u001b[0m\u001b[36m. What is the difference between a toaster and an oven?\u001b[0m\n",
              "\u001b[1;36m5\u001b[0m\u001b[36m. What is the best way to clean a toaster?\u001b[0m\n",
              "\u001b[1;36m6\u001b[0m\u001b[36m. How do you prepare toast for breakfast?\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Create a list of 6 questions for my interview with a toaster:\\n\\n\"\"\"\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8mTFL4cM_S5t"
      },
      "source": [
        "## [Generative] Other tasks\n",
        "\n",
        "In this section, we leave examples of prompts that you can play with. These were definitely never ever used to train these models. It's amazing how these models are able to produce such results."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sQRYaokSAjkG"
      },
      "source": [
        "### Movie2Emoji"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        },
        "id": "IBTnTG_g8Rg8",
        "outputId": "936246e1-628d-4c09-cd2c-ae57da8fc363"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">davinci</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Convert movie titles into emoji.</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Back to the Future: 👨👴🚗🕒 </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Batman: 🤵🦇 </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Transformers: 🚗🤖 </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Jurassic Park:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">🦖🏞</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mdavinci\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mConvert movie titles into emoji.\u001b[0m\n",
              "\n",
              "\u001b[90mBack to the Future: 👨👴🚗🕒 \u001b[0m\n",
              "\u001b[90mBatman: 🤵🦇 \u001b[0m\n",
              "\u001b[90mTransformers: 🚗🤖 \u001b[0m\n",
              "\u001b[90mJurassic Park:\u001b[0m\n",
              "\n",
              "\u001b[36m🦖🏞\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Convert movie titles into emoji.\n",
        "\n",
        "Back to the Future: 👨👴🚗🕒 \n",
        "Batman: 🤵🦇 \n",
        "Transformers: 🚗🤖 \n",
        "Jurassic Park:\n",
        "\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"davinci\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TzkKAef-AnqW"
      },
      "source": [
        "### Content creator\n",
        "\n",
        "Create lists, spreadsheets in the format you desire."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 371
        },
        "id": "6XxmGV9s_2gk",
        "outputId": "bb852e9c-4019-4cf4-c680-79075f1a4291"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">A two-column spreadsheet of top science fiction movies and the year of</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">release:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Title|  Year of release</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The Matrix | </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1999</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Ender's Game | </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1984</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The Terminator | </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1984</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Ender's Game | </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1999</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The Fall | </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2009</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The War of the Worlds | </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1851</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The Matrix Reloaded | </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2003</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Ender's Game | </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2004</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">The Matrix Revolutions | </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1999</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mA two-column spreadsheet of top science fiction movies and the year of\u001b[0m\n",
              "\u001b[90mrelease:\u001b[0m\n",
              "\n",
              "\u001b[90mTitle|  Year of release\u001b[0m\n",
              "\n",
              "\u001b[36mThe Matrix | \u001b[0m\u001b[1;36m1999\u001b[0m\n",
              "\n",
              "\u001b[36mEnder's Game | \u001b[0m\u001b[1;36m1984\u001b[0m\n",
              "\n",
              "\u001b[36mThe Terminator | \u001b[0m\u001b[1;36m1984\u001b[0m\n",
              "\n",
              "\u001b[36mEnder's Game | \u001b[0m\u001b[1;36m1999\u001b[0m\n",
              "\n",
              "\u001b[36mThe Fall | \u001b[0m\u001b[1;36m2009\u001b[0m\n",
              "\n",
              "\u001b[36mThe War of the Worlds | \u001b[0m\u001b[1;36m1851\u001b[0m\n",
              "\n",
              "\u001b[36mThe Matrix Reloaded | \u001b[0m\u001b[1;36m2003\u001b[0m\n",
              "\n",
              "\u001b[36mEnder's Game | \u001b[0m\u001b[1;36m2004\u001b[0m\n",
              "\n",
              "\u001b[36mThe Matrix Revolutions | \u001b[0m\u001b[1;36m1999\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"A two-column spreadsheet of top science fiction movies and the year of release:\n",
        "\n",
        "Title |  Year of release\"\"\"\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "id": "jpNgRajsAy_-",
        "outputId": "8e7db107-58b5-4bc3-a793-4bef629e59e0"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Make a list of the most important philosophers in history.</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">1</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span><span style=\"color: #008080; text-decoration-color: #008080\"> Socrates</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #008080; text-decoration-color: #008080\">. Plato</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"color: #008080; text-decoration-color: #008080\">. Aristotle</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"color: #008080; text-decoration-color: #008080\">. Kant</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"color: #008080; text-decoration-color: #008080\">. Hegel</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span><span style=\"color: #008080; text-decoration-color: #008080\">. Darwin</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"color: #008080; text-decoration-color: #008080\">. Freud</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span><span style=\"color: #008080; text-decoration-color: #008080\">. Marx</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span><span style=\"color: #008080; text-decoration-color: #008080\">. Nietzsche</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mMake a list of the most important philosophers in history.\u001b[0m\n",
              "\n",
              "\u001b[1;90m1\u001b[0m\u001b[90m.\u001b[0m\u001b[36m Socrates\u001b[0m\n",
              "\u001b[1;36m2\u001b[0m\u001b[36m. Plato\u001b[0m\n",
              "\u001b[1;36m3\u001b[0m\u001b[36m. Aristotle\u001b[0m\n",
              "\u001b[1;36m4\u001b[0m\u001b[36m. Kant\u001b[0m\n",
              "\u001b[1;36m5\u001b[0m\u001b[36m. Hegel\u001b[0m\n",
              "\u001b[1;36m6\u001b[0m\u001b[36m. Darwin\u001b[0m\n",
              "\u001b[1;36m7\u001b[0m\u001b[36m. Freud\u001b[0m\n",
              "\u001b[1;36m8\u001b[0m\u001b[36m. Marx\u001b[0m\n",
              "\u001b[1;36m9\u001b[0m\u001b[36m. Nietzsche\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Make a list of the most important philosophers in history.\n",
        "\n",
        "1.\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"curie\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 249
        },
        "id": "nc_nYW-RA53z",
        "outputId": "3aa3b2f1-22ee-4305-dd9d-7ecffb05042f"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">curie</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Make a list of the most important philosophers in history.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Early Greek philosophers </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">(</span><span style=\"color: #808080; text-decoration-color: #808080\">Name, estimated time</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">)</span><span style=\"color: #808080; text-decoration-color: #808080\">:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Thales of Miletus </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">~</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">624</span><span style=\"color: #008080; text-decoration-color: #008080\">-</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">546</span><span style=\"color: #008080; text-decoration-color: #008080\"> BCE</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Anaximander </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">~</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">611</span><span style=\"color: #008080; text-decoration-color: #008080\">-</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">547</span><span style=\"color: #008080; text-decoration-color: #008080\"> BCE</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Zeno of Elea </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">~</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">490</span><span style=\"color: #008080; text-decoration-color: #008080\">-</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">430</span><span style=\"color: #008080; text-decoration-color: #008080\"> BCE</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Pythagoras </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">~</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">569</span><span style=\"color: #008080; text-decoration-color: #008080\">-</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">495</span><span style=\"color: #008080; text-decoration-color: #008080\"> BCE</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Plato </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">~</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">428</span><span style=\"color: #008080; text-decoration-color: #008080\">-</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">347</span><span style=\"color: #008080; text-decoration-color: #008080\"> BCE</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Euclid </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">~</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">350</span><span style=\"color: #008080; text-decoration-color: #008080\">-</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">270</span><span style=\"color: #008080; text-decoration-color: #008080\"> BCE</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Aristotle </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">~</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span><span style=\"color: #008080; text-decoration-color: #008080\">-</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">322</span><span style=\"color: #008080; text-decoration-color: #008080\"> BCE</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Medieval philosophers </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">Name, estimated time</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\">:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Albert the Great </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1200</span><span style=\"color: #008080; text-decoration-color: #008080\">-</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">12</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mcurie\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mMake a list of the most important philosophers in history.\u001b[0m\n",
              "\u001b[90mEarly Greek philosophers \u001b[0m\u001b[1;90m(\u001b[0m\u001b[90mName, estimated time\u001b[0m\u001b[1;90m)\u001b[0m\u001b[90m:\u001b[0m\n",
              "\n",
              "\u001b[36mThales of Miletus \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36m~\u001b[0m\u001b[1;36m624\u001b[0m\u001b[36m-\u001b[0m\u001b[1;36m546\u001b[0m\u001b[36m BCE\u001b[0m\u001b[1;36m)\u001b[0m\n",
              "\u001b[36mAnaximander \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36m~\u001b[0m\u001b[1;36m611\u001b[0m\u001b[36m-\u001b[0m\u001b[1;36m547\u001b[0m\u001b[36m BCE\u001b[0m\u001b[1;36m)\u001b[0m\n",
              "\u001b[36mZeno of Elea \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36m~\u001b[0m\u001b[1;36m490\u001b[0m\u001b[36m-\u001b[0m\u001b[1;36m430\u001b[0m\u001b[36m BCE\u001b[0m\u001b[1;36m)\u001b[0m\n",
              "\u001b[36mPythagoras \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36m~\u001b[0m\u001b[1;36m569\u001b[0m\u001b[36m-\u001b[0m\u001b[1;36m495\u001b[0m\u001b[36m BCE\u001b[0m\u001b[1;36m)\u001b[0m\n",
              "\u001b[36mPlato \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36m~\u001b[0m\u001b[1;36m428\u001b[0m\u001b[36m-\u001b[0m\u001b[1;36m347\u001b[0m\u001b[36m BCE\u001b[0m\u001b[1;36m)\u001b[0m\n",
              "\u001b[36mEuclid \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36m~\u001b[0m\u001b[1;36m350\u001b[0m\u001b[36m-\u001b[0m\u001b[1;36m270\u001b[0m\u001b[36m BCE\u001b[0m\u001b[1;36m)\u001b[0m\n",
              "\u001b[36mAristotle \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36m~\u001b[0m\u001b[1;36m384\u001b[0m\u001b[36m-\u001b[0m\u001b[1;36m322\u001b[0m\u001b[36m BCE\u001b[0m\u001b[1;36m)\u001b[0m\n",
              "\n",
              "\u001b[36mMedieval philosophers \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mName, estimated time\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m:\u001b[0m\n",
              "\n",
              "\u001b[36mAlbert the Great \u001b[0m\u001b[1;36m(\u001b[0m\u001b[1;36m1200\u001b[0m\u001b[36m-\u001b[0m\u001b[1;36m12\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Make a list of the most important philosophers in history.\n",
        "Early Greek philosophers (Name, estimated time):\\n\"\"\"\n",
        "gpt3_generate(prompt, model_name=\"curie\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BxLAn7u3G0Kk"
      },
      "source": [
        "### Study guide \n",
        "\n",
        "You'll never be alone with GPT-3. He will always have something to help you with during your studies *ehehe*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 211
        },
        "id": "enlh3vKlG0To",
        "outputId": "f0154308-028f-42b3-c6c1-452bfa13aeb2"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">What are </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\"> key points I should know when studying contemporary art?</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"color: #008080; text-decoration-color: #008080\">. Contemporary art is created by artists who are interested in exploring the </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">boundaries of what is possible.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #008080; text-decoration-color: #008080\">. Contemporary art is often based on the idea of </span><span style=\"color: #008080; text-decoration-color: #008080\">\"the new world order.\"</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"color: #008080; text-decoration-color: #008080\">. Contemporary art is often interpreted as a reflection of current events.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"color: #008080; text-decoration-color: #008080\">. Contemporary art often uses a variety of techniques and materials.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"color: #008080; text-decoration-color: #008080\">. Contemporary art is often collaborative.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mWhat are \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m key points I should know when studying contemporary art?\u001b[0m\n",
              "\n",
              "\u001b[1;36m1\u001b[0m\u001b[36m. Contemporary art is created by artists who are interested in exploring the \u001b[0m\n",
              "\u001b[36mboundaries of what is possible.\u001b[0m\n",
              "\n",
              "\u001b[1;36m2\u001b[0m\u001b[36m. Contemporary art is often based on the idea of \u001b[0m\u001b[36m\"the new world order.\"\u001b[0m\n",
              "\n",
              "\u001b[1;36m3\u001b[0m\u001b[36m. Contemporary art is often interpreted as a reflection of current events.\u001b[0m\n",
              "\n",
              "\u001b[1;36m4\u001b[0m\u001b[36m. Contemporary art often uses a variety of techniques and materials.\u001b[0m\n",
              "\n",
              "\u001b[1;36m5\u001b[0m\u001b[36m. Contemporary art is often collaborative.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"What are 5 key points I should know when studying contemporary art?\"\"\"\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vebbqezUHpew"
      },
      "source": [
        "### Content Rating"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 115
        },
        "id": "adiYmGsmHpj_",
        "outputId": "2310182a-0bc5-4a3a-e334-fe86e0a89d2a"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">babbage</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Provide an ESRB rating for the following text:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">\"i'm going to blow your brains out with my ray gun then stomp on your guts.\"</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">ESRB rating:</span><span style=\"color: #008080; text-decoration-color: #008080\"> This text contains graphic violence and is not appropriate for </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">children.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mbabbage\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mProvide an ESRB rating for the following text:\u001b[0m\n",
              "\n",
              "\u001b[90m\"i'm going to blow your brains out with my ray gun then stomp on your guts.\"\u001b[0m\n",
              "\n",
              "\u001b[90mESRB rating:\u001b[0m\u001b[36m This text contains graphic violence and is not appropriate for \u001b[0m\n",
              "\u001b[36mchildren.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Provide an ESRB rating for the following text:\n",
        "\n",
        "\"i'm going to blow your brains out with my ray gun then stomp on your guts.\"\n",
        "\n",
        "ESRB rating:\"\"\"\n",
        "gpt3_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q2I6J3PE2h-T"
      },
      "source": [
        "# [T0] Zero-shot and few-shot learning\n",
        "\n",
        "We had a lot of fun with GPT-3, but let us explore the capabilities of other models that came out after GPT-3. In this section, we will explore **T0** -- a model proposed in March 2022 --, which not only 16x smaller than GPT-3 but also outperforms it in many tasks. \n",
        "\n",
        "It is known for its impressive **zero-shot task generalization** of English natural language prompts! \n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/yongzx/bigscience-workshop.github.io/gh-pages/en/pages/uploads/images/Octopus.png\" alt=\"High level view scheme of T0 training scheme\" style=\"height: 100px;\"/>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q8o43R9527HL"
      },
      "source": [
        "## Setup\n",
        "\n",
        "In the [original paper](https://arxiv.org/pdf/2110.08207.pdf), the authors analyse versions of T0 and determined that the variant **T0++** yields better results on average. We will use HuggingFace and load the corresponding model, which is named `t0pp`! \n",
        "\n",
        "The model contains 11B parameters overall, which is pretty big and won't load in the non-premium notebook. Hence, we set up a API for you to access these models!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R899WOTIBG_m",
        "outputId": "8820db9b-1ab0-4ed7-f9c3-a8d162db4f44"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 88.9 ms, sys: 17.5 ms, total: 106 ms\n",
            "Wall time: 6.7 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "%%capture\n",
        "!pip install rich\n",
        "\n",
        "# Visuals\n",
        "from rich.console import Console\n",
        "from rich.text import Text\n",
        "\n",
        "# Console for printing with nice colors :)\n",
        "console = Console(width=80)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "quWaUCF82iEz"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "import requests\n",
        "\n",
        "HUGGING_FACE_API_KEY = ... # TODO: Set OPEN AI's API key with read access\n",
        "HEADERS = {\"Authorization\": f\"Bearer {HUGGING_FACE_API_KEY}\"}\n",
        "\n",
        "MODEL2URL = {\n",
        "    \"t0\": \"https://api-inference.huggingface.co/models/bigscience/T0\",\n",
        "    \"t0-3b\": \"https://api-inference.huggingface.co/models/bigscience/T0_3B\",\n",
        "    \"t0p\": \"https://api-inference.huggingface.co/models/bigscience/T0p\",\n",
        "    \"t0pp\": \"https://api-inference.huggingface.co/models/bigscience/T0pp\",\n",
        "    \"tk\": \"https://api-inference.huggingface.co/models/allenai/tk-instruct-11b-def\",\n",
        "    \"tk-3b\": \"https://api-inference.huggingface.co/models/allenai/tk-instruct-3b-def\",\n",
        "}\n",
        "\n",
        "def t_generate(prompt, model_name: str=\"t0pp\", debug: bool=False, **kwargs):\n",
        "  api_url = MODEL2URL[model_name]\n",
        "  assert isinstance(model_name, str) and model_name in (MODEL2URL.keys())\n",
        "\n",
        "  payload = {\"inputs\": prompt}\n",
        "  payload.update(kwargs)\n",
        "  if debug:\n",
        "    print(payload)\n",
        "  data = json.dumps(payload)\n",
        "  response = requests.request(\"POST\", api_url, headers=HEADERS, data=data)\n",
        "  response = json.loads(response.content.decode(\"utf-8\"))\n",
        "  \n",
        "  if isinstance(response, list):\n",
        "    response = response[0][\"generated_text\"]\n",
        "    console.print(f\"[green]({model_name}) [bright_black]{prompt}[cyan] {response}\")\n",
        "  else:\n",
        "    console.print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQhsoqWfK0vP"
      },
      "source": [
        "## Numerical Reasoning\n",
        "\n",
        "\n",
        "We will refrain from copying the task introduction. Please head to GPT-3 Numerical reasoning section for an introduction to the task. In this case, we will assess T0's availability to perform this task."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49
        },
        "id": "dV12g7bTK00W",
        "outputId": "7515d50a-330e-4af2-c70f-d76a3af9e0d7"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m10\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">4</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m4\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m7\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Note: We add the `stop` argument because we're only interested in the output\n",
        "# up until the very first period. No need to generate beyond that as we would\n",
        "# be wasting resources!\n",
        "prompt = \"\"\"Q: What is 5 plus 5? A:\"\"\"\n",
        "t_generate(prompt, stop=\".\")\n",
        "prompt = \"\"\"Q: What is 5 plus 4? A:\"\"\"\n",
        "t_generate(prompt, stop=\".\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49
        },
        "id": "NCpbvCFTZMyr",
        "outputId": "7bf36140-eab7-4890-c217-a0db920735bf"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> times </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m times \u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m13\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\"> times </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m times \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m10\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Q: What is 13 times 2? A:\"\"\"\n",
        "t_generate(prompt, stop=\".\")\n",
        "prompt = \"\"\"Q: What is 2 times 5? A:\"\"\"\n",
        "t_generate(prompt, stop=\".\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i85U6jPLNEuy"
      },
      "source": [
        "T0 seems to get some of the math right, some of it wrong. Let's try **few-shot learning**! "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        },
        "id": "yY3Jao_yYQuX",
        "outputId": "c10018e2-e0f3-42d7-e6c0-14844de627e4"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Question: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">24</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\">? </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">34</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Question: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">20</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Question: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">8</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Question: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span><span style=\"color: #808080; text-decoration-color: #808080\">? </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">32</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Question: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">?</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mQuestion: What is \u001b[0m\u001b[1;90m24\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m? \u001b[0m\u001b[1;90m34\u001b[0m\n",
              "\u001b[90mQuestion: What is \u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? \u001b[0m\u001b[1;90m20\u001b[0m\n",
              "\u001b[90mQuestion: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? \u001b[0m\u001b[1;90m8\u001b[0m\n",
              "\u001b[90mQuestion: What is \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m22\u001b[0m\u001b[90m? \u001b[0m\u001b[1;90m32\u001b[0m\n",
              "\u001b[90mQuestion: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m?\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m18\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Question: What is 24 plus 10? 34\n",
        "Question: What is 2 plus 18? 20\n",
        "Question: What is 3 plus 5? 8\n",
        "Question: What is 10 plus 22? 32\n",
        "Question: What is 13 plus 18?\"\"\"\n",
        "t_generate(prompt, model_name=\"t0\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KHAoRoEKYrf0"
      },
      "source": [
        "Let us try to boost the model's performance by adding a bit of **positive encouragement** in the prompt. (Feel free to tweak the prompt!)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 129
        },
        "id": "7O8Sfp8GYrk1",
        "outputId": "811765da-4918-43bc-e63e-8a4f146700af"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">I'm great at math and addition. The following addition operations are </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">easy.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">24</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">34</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">20</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">8</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">32</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span><span style=\"color: #008080; text-decoration-color: #008080\">.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mI'm great at math and addition. The following addition operations are \u001b[0m\n",
              "\u001b[90measy.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m24\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m34\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m20\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m8\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m22\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m32\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m20\u001b[0m\u001b[36m.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"I'm great at math and addition. The following addition operations are easy.\n",
        "Q: What is 24 plus 10? A: 34.\n",
        "Q: What is 3 plus 18? A: 20.\n",
        "Q: What is 3 plus 5? A: 8.\n",
        "Q: What is 10 plus 22? A: 32.\n",
        "Q: What is 13 plus 18? A:\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 161
        },
        "id": "djqgBljcS7b8",
        "outputId": "065fa44c-1179-4396-a60c-5a14b3b12b04"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">24</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">34</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">20</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">4</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">9</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">10</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">32</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">6</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">6</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">11</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">12</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">16</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">28</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span><span style=\"color: #808080; text-decoration-color: #808080\">? A: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">35</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Q: What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">? A:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span><span style=\"color: #008080; text-decoration-color: #008080\">.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mQ: What is \u001b[0m\u001b[1;90m24\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m34\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m20\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m4\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m9\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m10\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m22\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m32\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m6\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m6\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m11\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m12\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m16\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m28\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m22\u001b[0m\u001b[90m? A: \u001b[0m\u001b[1;90m35\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mQ: What is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m? A:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m28\u001b[0m\u001b[36m.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Now leveraging 8-shots, i.e., 8 examples in context\n",
        "prompt = \"\"\"Q: What is 24 plus 10? A: 34.\n",
        "Q: What is 3 plus 18? A: 20.\n",
        "Q: What is 4 plus 9? A: 13.\n",
        "Q: What is 10 plus 22? A: 32.\n",
        "Q: What is 3 plus 3? A: 6.\n",
        "Q: What is 6 plus 5? A: 11.\n",
        "Q: What is 12 plus 16? A: 28.\n",
        "Q: What is 13 plus 22? A: 35.\n",
        "Q: What is 13 plus 18? A:\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nE0Lt6GBTIAE"
      },
      "source": [
        "### Exercise\n",
        "\n",
        "Let us try something different, now! Can you think of any prompt? Try first defining it as a 0-shot task and see how the model performs. Then, define a few examples and create a prompt including those examples. Does it improve?\n",
        "\n",
        "(**Suggestion**: On the same line of thought, we can experiment with the pattern `Q: What is x minutes in seconds? A:` or `Q: What is x weeks in days? A:`)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b5Zy-9iaTI37"
      },
      "outputs": [],
      "source": [
        "prompt = ... # TODO: Define 0-shot prompt\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KBURcVzXTNeH"
      },
      "outputs": [],
      "source": [
        "prompt = ... # TODO: Define few-shot prompt\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NxBHjJ3T2_ug"
      },
      "source": [
        "## Coreference resolution\n",
        "\n",
        "Coreference (or co-reference) resolution is that task of identifying which person or thing different mentions refer to. For example, in \n",
        "\n",
        "> Sameer told Catarina that he would bike to work. \n",
        "\n",
        "what entity does *he* refer to? We expect a good system to be able to identify *Sameer* as the correct entity but do current state-of-the-art NLP models do it?\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ailGWfqITjbW"
      },
      "outputs": [],
      "source": [
        "def get_coref_prompt(text: str) -> str:\n",
        "  import re\n",
        "  text = re.sub(r\"\\s+\", \" \", text)\n",
        "  return f\"\"\"{text}\\nIn the previous sentence, decide who 'her' is referring to.\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "_a2fB0lS3faR",
        "outputId": "aa03bf6d-3042-4865-c850-2fed423863c2"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Barack Obama nominated Hilary Clinton as his secretary of state on </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Monday. He chose her because she had foreign affairs experience as a former </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">First Lady.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">In the previous sentence, decide who </span><span style=\"color: #808080; text-decoration-color: #808080\">'her'</span><span style=\"color: #808080; text-decoration-color: #808080\"> is referring to.</span><span style=\"color: #008080; text-decoration-color: #008080\"> Hillary Clinton</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mBarack Obama nominated Hilary Clinton as his secretary of state on \u001b[0m\n",
              "\u001b[90mMonday. He chose her because she had foreign affairs experience as a former \u001b[0m\n",
              "\u001b[90mFirst Lady.\u001b[0m\n",
              "\u001b[90mIn the previous sentence, decide who \u001b[0m\u001b[90m'her'\u001b[0m\u001b[90m is referring to.\u001b[0m\u001b[36m Hillary Clinton\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Barack Obama nominated Hilary Clinton as his secretary of state on Monday. \n",
        "He chose her because she had foreign affairs experience as a former First Lady.\"\"\"\n",
        "prompt = get_coref_prompt(prompt)\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-9No4VdwTwN4"
      },
      "source": [
        "### Exercise"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nzffIi3WFupM"
      },
      "outputs": [],
      "source": [
        "prompt = ... # TODO: Try out your own sentences\n",
        "prompt = get_coref_prompt(prompt)\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ckh9fGkAT6Dj"
      },
      "source": [
        "## Linguistic Acceptability \n",
        "\n",
        "As before, let us try to use T0 for determining whether the sentences are grammatically correct."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A1k2nrcOT6Jx"
      },
      "outputs": [],
      "source": [
        "def get_cola_prompt(text: str, template: int=1) -> str:\n",
        "  templates = {\n",
        "      1: lambda s: f\"Is this corret according to English Grammar?\\n\\n\\\"{s}\\\" Yes or no?\",\n",
        "      2: lambda s: f\"Is this sentence grammatically correct or not? {s}\",\n",
        "      3: lambda s: f\"Decide whether this sentence is grammatically correct:\\n\\n{s}\",\n",
        "      4: lambda s: f\"\"\"I want to know whether the following sentence is correct.\\n{s}\\nIs it?\"\"\"\n",
        "  }\n",
        "\n",
        "\n",
        "\n",
        "  import re\n",
        "  text = re.sub(r\"\\s+\", \" \", text)\n",
        "  return templates[template](text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 65
        },
        "id": "lCc9lee6UHBv",
        "outputId": "052daf5c-2ef3-4dc1-b66e-04067e95ff55"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this corret according to English Grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">\"The more books I ask to whom he will give, the more he reads.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> Yes or no?</span><span style=\"color: #008080; text-decoration-color: #008080\"> No</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs this corret according to English Grammar?\u001b[0m\n",
              "\n",
              "\u001b[90m\"The more books I ask to whom he will give, the more he reads.\"\u001b[0m\u001b[90m Yes or no?\u001b[0m\u001b[36m No\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"The more books I ask to whom he will give, the more he reads.\" \n",
        "prompt = get_cola_prompt(sentence, template=1)\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 65
        },
        "id": "gPZo1PdjUKQP",
        "outputId": "a4a7445e-4924-4d27-f439-88be214c759c"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this corret according to English Grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">\"They can sing.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> Yes or no?</span><span style=\"color: #008080; text-decoration-color: #008080\"> No</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs this corret according to English Grammar?\u001b[0m\n",
              "\n",
              "\u001b[90m\"They can sing.\"\u001b[0m\u001b[90m Yes or no?\u001b[0m\u001b[36m No\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"They can sing.\"\n",
        "prompt = get_cola_prompt(sentence, template=1)\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        },
        "id": "E71gaFe0Xr8V",
        "outputId": "8ebec1e8-1347-48a9-bafd-7c56207d5625"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">They can sing. Correct the previous sentence.</span><span style=\"color: #008080; text-decoration-color: #008080\"> They can play the guitar.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mThey can sing. Correct the previous sentence.\u001b[0m\u001b[36m They can play the guitar.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"They can sing. Correct the previous sentence.\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 65
        },
        "id": "B13FXxB9X2Cb",
        "outputId": "7759f56e-fd2b-4fad-a161-76d6d55410c3"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this correct according to English grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">They can play the oboe.</span><span style=\"color: #008080; text-decoration-color: #008080\"> Yes</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs this correct according to English grammar?\u001b[0m\n",
              "\n",
              "\u001b[90mThey can play the oboe.\u001b[0m\u001b[36m Yes\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"They can play the oboe.\"\n",
        "prompt = get_cola_prompt(sentence)\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vxFCy765XJuN"
      },
      "source": [
        "It seems that if we start the sentence *____ can sing.* with a pronoun or an object that T0 is not able to correctly determine the correctness of the sentence. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "id": "MhDJccKNUTEk",
        "outputId": "4bb947a2-6022-4121-bddb-f4d5d07d0fc3"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this correct according to English grammar?</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">The dog barked its way out of the room.</span><span style=\"color: #008080; text-decoration-color: #008080\"> No</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs this correct according to English grammar?\u001b[0m\n",
              "\n",
              "\u001b[90mThe dog barked its way out of the room.\u001b[0m\u001b[36m No\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is this sentence grammatically correct or not? The dog barked its way out </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">of the room.</span><span style=\"color: #008080; text-decoration-color: #008080\"> No</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs this sentence grammatically correct or not? The dog barked its way out \u001b[0m\n",
              "\u001b[90mof the room.\u001b[0m\u001b[36m No\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Decide whether this sentence is grammatically correct:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">The dog barked its way out of the room.</span><span style=\"color: #008080; text-decoration-color: #008080\"> No</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mDecide whether this sentence is grammatically correct:\u001b[0m\n",
              "\n",
              "\u001b[90mThe dog barked its way out of the room.\u001b[0m\u001b[36m No\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">I want to know whether the following sentence is correct.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">The dog barked its way out of the room.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Is it?</span><span style=\"color: #008080; text-decoration-color: #008080\"> No</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mI want to know whether the following sentence is correct.\u001b[0m\n",
              "\u001b[90mThe dog barked its way out of the room.\u001b[0m\n",
              "\u001b[90mIs it?\u001b[0m\u001b[36m No\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"The dog barked its way out of the room.\"\n",
        "prompt = get_cola_prompt(sentence)\n",
        "t_generate(prompt)\n",
        "print()\n",
        "prompt = get_cola_prompt(sentence, template=2)\n",
        "t_generate(prompt)\n",
        "print()\n",
        "prompt = get_cola_prompt(sentence, template=3)\n",
        "t_generate(prompt)\n",
        "print()\n",
        "prompt = get_cola_prompt(sentence, template=4)\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "id": "221RXomzYGIL",
        "outputId": "ea3cff04-21ea-4b74-8de3-3ebd45436573"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">The dog barked its way out of the room. Correct the previous sentence.</span><span style=\"color: #008080; text-decoration-color: #008080\"> The </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">dog barked its way out of the room.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mThe dog barked its way out of the room. Correct the previous sentence.\u001b[0m\u001b[36m The \u001b[0m\n",
              "\u001b[36mdog barked its way out of the room.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Correct this: The dog barked its way out of the room.</span><span style=\"color: #008080; text-decoration-color: #008080\"> The dog barked its </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">way out of the room.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mCorrect this: The dog barked its way out of the room.\u001b[0m\u001b[36m The dog barked its \u001b[0m\n",
              "\u001b[36mway out of the room.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Let us see if we can get an idea of why the model thinks that the sentence is correct\n",
        "sentence = \"The dog barked its way out of the room.\"\n",
        "prompt = f\"{sentence} Correct the previous sentence.\"\n",
        "\n",
        "t_generate(prompt)\n",
        "\n",
        "prompt = f\"Correct this: {sentence}\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GwtFPn43YS8L"
      },
      "source": [
        "Weird... 🤔 The model seems to replicate the exact same sentence and, apparently, changing the prompt is not causing it to change its prediction. Let us try **few-shot learning**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "id": "1uNMG6enUpVF",
        "outputId": "b9cf749e-86cb-4c29-f5b6-706eebb4df95"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is the grammar on the following sentences correct?</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">They caused him to become angry by making him. no</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">You can sing. yes</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">singh and name sameer years is my am I </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">27</span><span style=\"color: #808080; text-decoration-color: #808080\"> old. no</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">he told me no. yes</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">They can sing.</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\"> No</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs the grammar on the following sentences correct?\u001b[0m\n",
              "\u001b[90mThey caused him to become angry by making him. no\u001b[0m\n",
              "\u001b[90mYou can sing. yes\u001b[0m\n",
              "\u001b[90msingh and name sameer years is my am I \u001b[0m\u001b[1;90m27\u001b[0m\u001b[90m old. no\u001b[0m\n",
              "\u001b[90mhe told me no. yes\u001b[0m\n",
              "\u001b[90mThey can sing.\u001b[0m\n",
              "\u001b[36m No\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Is the grammar on the following sentences correct?\n",
        "They caused him to become angry by making him. no\n",
        "You can sing. yes\n",
        "singh and name sameer years is my am I 27 old. no\n",
        "he told me no. yes\n",
        "They can sing.\n",
        "\"\"\"\n",
        "\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "id": "klLLBpIlaMCi",
        "outputId": "61b38919-7fa7-40c4-eafd-7104e3ecb4ca"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">why is the sentence </span><span style=\"color: #808080; text-decoration-color: #808080\">\"They can sing.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> incorrect?</span><span style=\"color: #008080; text-decoration-color: #008080\"> They can't sing.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mwhy is the sentence \u001b[0m\u001b[90m\"They can sing.\"\u001b[0m\u001b[90m incorrect?\u001b[0m\u001b[36m They can't sing.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">why is the sentence </span><span style=\"color: #808080; text-decoration-color: #808080\">\"They can sing.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> unacceptable?</span><span style=\"color: #008080; text-decoration-color: #008080\"> They can't sing.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mwhy is the sentence \u001b[0m\u001b[90m\"They can sing.\"\u001b[0m\u001b[90m unacceptable?\u001b[0m\u001b[36m They can't sing.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"why is the sentence \\\"They can sing.\\\" incorrect?\"\n",
        "t_generate(prompt)\n",
        "\n",
        "prompt = \"why is the sentence \\\"They can sing.\\\" unacceptable?\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PQ3Bg326aYqk"
      },
      "source": [
        "Okay, T0... You know it better.  We also tried tweaking the pronouns in the sentence! We find something interesting...  It seems like neither **you** nor **they** know how to sing according to T0 😅."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        },
        "id": "v2hHcFIIavYU",
        "outputId": "6e3aa23d-69fe-4cad-f130-16a23060e67c"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is the sentence </span><span style=\"color: #808080; text-decoration-color: #808080\">\"I can sing.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> correct?</span><span style=\"color: #008080; text-decoration-color: #008080\"> Yes</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs the sentence \u001b[0m\u001b[90m\"I can sing.\"\u001b[0m\u001b[90m correct?\u001b[0m\u001b[36m Yes\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is the sentence </span><span style=\"color: #808080; text-decoration-color: #808080\">\"You can sing.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> correct?</span><span style=\"color: #008080; text-decoration-color: #008080\"> No</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs the sentence \u001b[0m\u001b[90m\"You can sing.\"\u001b[0m\u001b[90m correct?\u001b[0m\u001b[36m No\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is the sentence </span><span style=\"color: #808080; text-decoration-color: #808080\">\"She can sing.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> correct?</span><span style=\"color: #008080; text-decoration-color: #008080\"> Yes</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs the sentence \u001b[0m\u001b[90m\"She can sing.\"\u001b[0m\u001b[90m correct?\u001b[0m\u001b[36m Yes\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is the sentence </span><span style=\"color: #808080; text-decoration-color: #808080\">\"He can sing.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> correct?</span><span style=\"color: #008080; text-decoration-color: #008080\"> Yes</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs the sentence \u001b[0m\u001b[90m\"He can sing.\"\u001b[0m\u001b[90m correct?\u001b[0m\u001b[36m Yes\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is the sentence </span><span style=\"color: #808080; text-decoration-color: #808080\">\"We can sing.\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> correct?</span><span style=\"color: #008080; text-decoration-color: #008080\"> Yes</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs the sentence \u001b[0m\u001b[90m\"We can sing.\"\u001b[0m\u001b[90m correct?\u001b[0m\u001b[36m Yes\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"Is the sentence \\\"I can sing.\\\" correct?\"\n",
        "t_generate(prompt)\n",
        "prompt = \"Is the sentence \\\"You can sing.\\\" correct?\"\n",
        "t_generate(prompt)\n",
        "prompt = \"Is the sentence \\\"She can sing.\\\" correct?\"\n",
        "t_generate(prompt)\n",
        "prompt = \"Is the sentence \\\"He can sing.\\\" correct?\"\n",
        "t_generate(prompt)\n",
        "prompt = \"Is the sentence \\\"We can sing.\\\" correct?\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "e8eIxDm1bua4",
        "outputId": "12fae15d-7f40-4f9c-9fd2-194086ae951a"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Is the grammar on the following sentences correct?</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">They caused him to become angry by making him. no</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">I can sing. yes</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">He can sing. yes</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">She no can sing. yes</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">sing You cannot. no</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">She they sing. no</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">singh and name sameer years is my am I </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">27</span><span style=\"color: #808080; text-decoration-color: #808080\"> old. no</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">he told me no. yes</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">They sing.</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\"> No</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mIs the grammar on the following sentences correct?\u001b[0m\n",
              "\u001b[90mThey caused him to become angry by making him. no\u001b[0m\n",
              "\u001b[90mI can sing. yes\u001b[0m\n",
              "\u001b[90mHe can sing. yes\u001b[0m\n",
              "\u001b[90mShe no can sing. yes\u001b[0m\n",
              "\u001b[90msing You cannot. no\u001b[0m\n",
              "\u001b[90mShe they sing. no\u001b[0m\n",
              "\u001b[90msingh and name sameer years is my am I \u001b[0m\u001b[1;90m27\u001b[0m\u001b[90m old. no\u001b[0m\n",
              "\u001b[90mhe told me no. yes\u001b[0m\n",
              "\u001b[90mThey sing.\u001b[0m\n",
              "\u001b[36m No\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Is the grammar on the following sentences correct?\n",
        "They caused him to become angry by making him. no\n",
        "I can sing. yes\n",
        "He can sing. yes\n",
        "She no can sing. yes\n",
        "sing You cannot. no\n",
        "She they sing. no\n",
        "singh and name sameer years is my am I 27 old. no\n",
        "he told me no. yes\n",
        "They can sing.\n",
        "\"\"\"\n",
        "\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LnGqa9UchXbt"
      },
      "source": [
        "Ok... This is my last try... I've looked into the `example 14` at the [model's Web Hosted Inference API](https://huggingface.co/bigscience/T0pp?text=The+word+%27binne%27+means+any+animal+that+is+furry+and+has+four+legs%2C+and+the+word+%27bam%27+means+a+simple+sort+of+dwelling.%0A%0A+Which+of+the+following+best+characterizes+binne+bams%3F%0A+-+Sentence+1%3A+Binne+bams+are+for+pets.%0A+-+Sentence+2%3A+Binne+bams+are+typically+furnished+with+sofas+and+televisions.%0A+-+Sentence+3%3A+Binne+bams+are+luxurious+apartments.%0A+-+Sentence+4%3A+Binne+bams+are+places+where+people+live.)  and appleid the same styling to our example! 🤯 🥳"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        },
        "id": "ir2GeizYgoZj",
        "outputId": "66ffbd55-4c79-4b82-9136-2da1b562f629"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">A correct SV sentence has one subject and one verb.</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> Which of the following best characterizes a correct SV sentence?</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> - Sentence </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">1</span><span style=\"color: #808080; text-decoration-color: #808080\">: They sing to the birds.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> - Sentence </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\">: They sing.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> - Sentence </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\">: I drive my car every morning.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> - Sentence </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">4</span><span style=\"color: #808080; text-decoration-color: #808080\">: Wellness centers are places where people live.</span><span style=\"color: #008080; text-decoration-color: #008080\"> Sentence </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mA correct SV sentence has one subject and one verb.\u001b[0m\n",
              "\n",
              "\u001b[90m Which of the following best characterizes a correct SV sentence?\u001b[0m\n",
              "\u001b[90m - Sentence \u001b[0m\u001b[1;90m1\u001b[0m\u001b[90m: They sing to the birds.\u001b[0m\n",
              "\u001b[90m - Sentence \u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m: They sing.\u001b[0m\n",
              "\u001b[90m - Sentence \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m: I drive my car every morning.\u001b[0m\n",
              "\u001b[90m - Sentence \u001b[0m\u001b[1;90m4\u001b[0m\u001b[90m: Wellness centers are places where people live.\u001b[0m\u001b[36m Sentence \u001b[0m\u001b[1;36m2\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"A correct SV sentence has one subject and one verb.\n",
        "\n",
        " Which of the following best characterizes a correct SV sentence?\n",
        " - Sentence 1: They sing to the birds.\n",
        " - Sentence 2: They sing.\n",
        " - Sentence 3: I drive my car every morning.\n",
        " - Sentence 4: Wellness centers are places where people live.\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wxq3nfb3UOJt"
      },
      "source": [
        "### Exercise \n",
        "\n",
        "We challenge you to try your own sentences and prompts! Can you find a way that the model correctly classifies `They can sing.` as a correct sentence? "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L500KnfHbi0o"
      },
      "outputs": [],
      "source": [
        "sentence = ... # Try it for yourself\n",
        "# Try few-shot, or different prompts :)\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vbnoUqXTcc0M"
      },
      "source": [
        "### [Extra] Syntax trees\n",
        "\n",
        "In this small section, we'd like to point out a syntax tree example, one of the students shared with us yesterday :) \n",
        "\n",
        "Note that the model was not trained to perform this kind of task and it is amazing its capabilities to infer the task based on a natural language description!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 65
        },
        "id": "N0nFh67mcc6r",
        "outputId": "19809583-c726-40a6-e806-a4736d8532eb"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Create a syntactic tree for this sentence: </span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">I like food.</span><span style=\"color: #008080; text-decoration-color: #008080\"> I like food.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mCreate a syntactic tree for this sentence: \u001b[0m\n",
              "\n",
              "\u001b[90mI like food.\u001b[0m\u001b[36m I like food.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Create a syntactic tree for this sentence: \\n\\nI like food.\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "gO-PskzhcjNy",
        "outputId": "45c439d6-1e71-4a2c-da85-469910521976"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">A two-column spreadsheet of a syntactic tree for each word in this </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">sentence: </span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">I like food.</span><span style=\"color: #008080; text-decoration-color: #008080\"> I like food.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mA two-column spreadsheet of a syntactic tree for each word in this \u001b[0m\n",
              "\u001b[90msentence: \u001b[0m\n",
              "\n",
              "\u001b[90mI like food.\u001b[0m\u001b[36m I like food.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"I like food\"\n",
        "prompt = f\"\"\"A two-column spreadsheet of a syntactic tree for each word in this sentence: \\n\\n{sentence}.\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 113
        },
        "id": "GfHEEwPeSdxq",
        "outputId": "bd56a8de-31fb-484d-a5b2-c80d812e0025"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Input: I like food.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Syntax: I </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">(</span><span style=\"color: #808080; text-decoration-color: #808080\">subject</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">)</span><span style=\"color: #808080; text-decoration-color: #808080\"> like </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">(</span><span style=\"color: #808080; text-decoration-color: #808080\">verb</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">)</span><span style=\"color: #808080; text-decoration-color: #808080\"> </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">food(</span><span style=\"color: #808080; text-decoration-color: #808080\">object</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">)</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Input: Food was eaten by me.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Syntax: me </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">(</span><span style=\"color: #808080; text-decoration-color: #808080\">subject</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">)</span><span style=\"color: #808080; text-decoration-color: #808080\"> eaten </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">(</span><span style=\"color: #808080; text-decoration-color: #808080\">verb</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">)</span><span style=\"color: #808080; text-decoration-color: #808080\"> </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">Food(</span><span style=\"color: #808080; text-decoration-color: #808080\">object</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">)</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Input: She really hates cats.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Output:</span><span style=\"color: #008080; text-decoration-color: #008080\"> She </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">subject</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\"> really hates cats.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mInput: I like food.\u001b[0m\n",
              "\u001b[90mSyntax: I \u001b[0m\u001b[1;90m(\u001b[0m\u001b[90msubject\u001b[0m\u001b[1;90m)\u001b[0m\u001b[90m like \u001b[0m\u001b[1;90m(\u001b[0m\u001b[90mverb\u001b[0m\u001b[1;90m)\u001b[0m\u001b[90m \u001b[0m\u001b[1;90mfood\u001b[0m\u001b[1;90m(\u001b[0m\u001b[90mobject\u001b[0m\u001b[1;90m)\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mInput: Food was eaten by me.\u001b[0m\n",
              "\u001b[90mSyntax: me \u001b[0m\u001b[1;90m(\u001b[0m\u001b[90msubject\u001b[0m\u001b[1;90m)\u001b[0m\u001b[90m eaten \u001b[0m\u001b[1;90m(\u001b[0m\u001b[90mverb\u001b[0m\u001b[1;90m)\u001b[0m\u001b[90m \u001b[0m\u001b[1;90mFood\u001b[0m\u001b[1;90m(\u001b[0m\u001b[90mobject\u001b[0m\u001b[1;90m)\u001b[0m\u001b[90m.\u001b[0m\n",
              "\u001b[90mInput: She really hates cats.\u001b[0m\n",
              "\u001b[90mOutput:\u001b[0m\u001b[36m She \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36msubject\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m really hates cats.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = f\"\"\"Input: I like food.\n",
        "Syntax: I (subject) like (verb) food(object).\n",
        "Input: Food was eaten by me.\n",
        "Syntax: me (subject) eaten (verb) Food(object).\n",
        "Input: She really hates cats.\n",
        "Output:\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xjE8rO2ZcoDr"
      },
      "source": [
        "## [Generative] Creating analogies\n",
        "\n",
        "In this part, we will use T0 generation capabilities to generate analogies for any sentence we specify. Try running the same prompt multiple times. Due to the randomness of the generation it might happen so that the model does not generate interesting results at first, but ends up with good generations after a few times :) \n",
        "\n",
        "\n",
        "If this does not work, you can always provide a few examples (use **few-shot learning**)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 83
        },
        "id": "EK6PzTGDcoKQ",
        "outputId": "951bb5a9-af6b-48bc-dcec-8291b10d639f"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Create an analogy for this phrase:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Questions are arrows, this means that</span><span style=\"color: #008080; text-decoration-color: #008080\"> if you ask a question, you are shooting an</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">arrow</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mCreate an analogy for this phrase:\u001b[0m\n",
              "\n",
              "\u001b[90mQuestions are arrows, this means that\u001b[0m\u001b[36m if you ask a question, you are shooting an\u001b[0m\n",
              "\u001b[36marrow\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "metaphor = \"Questions are arrows\"\n",
        "prompt = f\"\"\"Create an analogy for this phrase:\\n\\n{metaphor}, this means that\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 83
        },
        "id": "agMepPo1c5mq",
        "outputId": "09e80c4b-f6ac-4240-c0e2-5f19d2f39e78"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Create an analogy for this phrase:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">You are an early bird, this means that</span><span style=\"color: #008080; text-decoration-color: #008080\"> You are an early riser, this means that </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">you get up early.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mCreate an analogy for this phrase:\u001b[0m\n",
              "\n",
              "\u001b[90mYou are an early bird, this means that\u001b[0m\u001b[36m You are an early riser, this means that \u001b[0m\n",
              "\u001b[36myou get up early.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "metaphor = \"You are an early bird\"\n",
        "prompt = f\"\"\"Create an analogy for this phrase:\\n\\n{metaphor}, this means that\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yhB8TVFldGJo"
      },
      "source": [
        "### Exercise\n",
        "\n",
        "Try it yourself :) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jdwjM68YdHVZ"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-35Sos3hdH1l"
      },
      "source": [
        "## [Generative] Text2Question\n",
        "\n",
        "Let us use T0 to generate questions automatically for different prompts. Here are a few examples 😜"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 66
        },
        "id": "VaWSi8mXdIBR",
        "outputId": "08576dfa-cfcd-4f4f-fccc-707a06dbcae5"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Write a question for this sentence:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">I hide beneath the bed when I think of E.T.</span><span style=\"color: #008080; text-decoration-color: #008080\"> What do I do when I think of E.T.?</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mWrite a question for this sentence:\u001b[0m\n",
              "\n",
              "\u001b[90mI hide beneath the bed when I think of E.T.\u001b[0m\u001b[36m What do I do when I think of E.T.?\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"I hide beneath the bed when I think of E.T.\"\n",
        "prompt = f\"\"\"Write a question for this sentence:\\n\\n{sentence}\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 83
        },
        "id": "cpY_Lkk-dOK9",
        "outputId": "4a9c7752-bd12-44bb-d999-d952b91c3dff"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Write a question for this sentence:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">When I am angry with the world, I hit people with a teddy bear.</span><span style=\"color: #008080; text-decoration-color: #008080\"> What do I hit </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">people with?</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mWrite a question for this sentence:\u001b[0m\n",
              "\n",
              "\u001b[90mWhen I am angry with the world, I hit people with a teddy bear.\u001b[0m\u001b[36m What do I hit \u001b[0m\n",
              "\u001b[36mpeople with?\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence = \"When I am angry with the world, I hit people with a teddy bear.\"\n",
        "prompt = f\"\"\"Write a question for this sentence:\\n\\n{sentence}\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E_V80jJ6dRlP"
      },
      "source": [
        "Ok! This looks like T0 has a better time creating questions than numerical reasoning, discerning grammatically correct sentences, and syntax trees."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NiChNU_MdmDs"
      },
      "source": [
        "### Exercise\n",
        "\n",
        "Try your own examples!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "id": "syjpazrEdpwr",
        "outputId": "b6158f6e-edbb-49d6-fffa-aa0306f1f59b"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Create a list of </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">6</span><span style=\"color: #808080; text-decoration-color: #808080\"> questions for my interview with NASA:</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">1</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span><span style=\"color: #008080; text-decoration-color: #008080\"> What is the name of the space agency?</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mCreate a list of \u001b[0m\u001b[1;90m6\u001b[0m\u001b[90m questions for my interview with NASA:\u001b[0m\n",
              "\u001b[1;90m1\u001b[0m\u001b[90m.\u001b[0m\u001b[36m What is the name of the space agency?\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Create a list of 6 questions for my interview with NASA:\\n1.\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B9i0DJ77eGNz"
      },
      "outputs": [],
      "source": [
        "prompt = ... #TODO\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t3q2BkTWGCTX"
      },
      "source": [
        "## Other generative (and discriminative) tasks \n",
        "\n",
        "So far, it seems like T0 is not as good as strong as GPT-based models in generating creative continuations of our inputs. You'll probably noticed that its outputs are less imaginative and also less random. \n",
        "\n",
        "In part, this can be explained by the pretraining strategy. T0 was trained on a variety of NLP tasks. These are typically well structured and the authors converted them into natural language formats using several templates that you can check on their paper. In some sense, this may explain its lower abilities to generalize.\n",
        "\n",
        "\n",
        "For this reason, we will focus this next section in other tasks that these models are able to do, somewhat successfully. However, feel free to try out the sames tasks as we've seen in the GPT-3 section.\n",
        "\n",
        "### Multiple choice answer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 151
        },
        "id": "DsSHplgEgZ4p",
        "outputId": "594bd468-50ac-49ea-b27c-e96be1a0f6bd"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">The word </span><span style=\"color: #808080; text-decoration-color: #808080\">'binne'</span><span style=\"color: #808080; text-decoration-color: #808080\"> means any animal that is furry and has four legs, and </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">the word </span><span style=\"color: #808080; text-decoration-color: #808080\">'bam'</span><span style=\"color: #808080; text-decoration-color: #808080\"> means a simple sort of dwelling.</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> Which of the following best characterizes binne bams?</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> - Sentence </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">1</span><span style=\"color: #808080; text-decoration-color: #808080\">: Binne bams are for pets.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> - Sentence </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\">: Binne bams are typically furnished with sofas and televisions.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> - Sentence </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\">: Binne bams are luxurious apartments.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> - Sentence </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">4</span><span style=\"color: #808080; text-decoration-color: #808080\">: Binne bams are places where people live.</span><span style=\"color: #008080; text-decoration-color: #008080\"> Binne bams are for pets.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mThe word \u001b[0m\u001b[90m'binne'\u001b[0m\u001b[90m means any animal that is furry and has four legs, and \u001b[0m\n",
              "\u001b[90mthe word \u001b[0m\u001b[90m'bam'\u001b[0m\u001b[90m means a simple sort of dwelling.\u001b[0m\n",
              "\n",
              "\u001b[90m Which of the following best characterizes binne bams?\u001b[0m\n",
              "\u001b[90m - Sentence \u001b[0m\u001b[1;90m1\u001b[0m\u001b[90m: Binne bams are for pets.\u001b[0m\n",
              "\u001b[90m - Sentence \u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m: Binne bams are typically furnished with sofas and televisions.\u001b[0m\n",
              "\u001b[90m - Sentence \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m: Binne bams are luxurious apartments.\u001b[0m\n",
              "\u001b[90m - Sentence \u001b[0m\u001b[1;90m4\u001b[0m\u001b[90m: Binne bams are places where people live.\u001b[0m\u001b[36m Binne bams are for pets.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"The word 'binne' means any animal that is furry and has four legs, and the word 'bam' means a simple sort of dwelling.\n",
        "\n",
        " Which of the following best characterizes binne bams?\n",
        " - Sentence 1: Binne bams are for pets.\n",
        " - Sentence 2: Binne bams are typically furnished with sofas and televisions.\n",
        " - Sentence 3: Binne bams are luxurious apartments.\n",
        " - Sentence 4: Binne bams are places where people live.\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kF8z-LDxm21Z"
      },
      "source": [
        "### Re-ordering words"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "id": "QmoS_NRRGHiC",
        "outputId": "2071b221-7933-4dc3-8072-3420e3f0b9cb"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Reorder the words in this sentence: justin and name bieber years is my am</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">I </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">27</span><span style=\"color: #808080; text-decoration-color: #808080\"> old.</span><span style=\"color: #008080; text-decoration-color: #008080\"> justin bieber is my name and i am </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span><span style=\"color: #008080; text-decoration-color: #008080\"> years old</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mReorder the words in this sentence: justin and name bieber years is my am\u001b[0m\n",
              "\u001b[90mI \u001b[0m\u001b[1;90m27\u001b[0m\u001b[90m old.\u001b[0m\u001b[36m justin bieber is my name and i am \u001b[0m\u001b[1;36m27\u001b[0m\u001b[36m years old\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Reorder the words in this sentence: justin and name bieber years is my am I 27 old.\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MF9PcG0Dm5hK"
      },
      "source": [
        "### Copying sentences with different intents"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "id": "FSBZJvzhH-cy",
        "outputId": "cdf54c21-918a-4341-daa4-8c1a0fcfa0a0"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Task: copy but say the opposite.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\"> PSG won its match against Barca.</span><span style=\"color: #008080; text-decoration-color: #008080\"> PSG lost its match against Barca.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mTask: copy but say the opposite.\u001b[0m\n",
              "\u001b[90m PSG won its match against Barca.\u001b[0m\u001b[36m PSG lost its match against Barca.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Task: copy but say the opposite.\n",
        " PSG won its match against Barca.\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YFoqX_5VnA7W"
      },
      "source": [
        "### Concept2Question and Concept2Text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "id": "5rNT7yhYGCZu",
        "outputId": "833dc1c3-1d3c-479e-9ba5-4292f7350ae9"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0pp</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Convert the concepts to a question: knife, hungry</span><span style=\"color: #008080; text-decoration-color: #008080\"> A hungry man is </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">reaching for a knife.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0pp\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mConvert the concepts to a question: knife, hungry\u001b[0m\u001b[36m A hungry man is \u001b[0m\n",
              "\u001b[36mreaching for a knife.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Convert the concepts to a question: knife, hungry\"\"\"\n",
        "t_generate(prompt) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "45t0CoreqNTv",
        "outputId": "3e83d5f1-5976-423b-fa4d-9b7d079c3925"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Humans can easily string together abstract concepts to form a coherent</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">sentence. An example of a sentence with the concepts </span><span style=\"color: #808080; text-decoration-color: #808080\">\"food\"</span><span style=\"color: #808080; text-decoration-color: #808080\">, </span><span style=\"color: #808080; text-decoration-color: #808080\">\"avocado\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> can be</span><span style=\"color: #008080; text-decoration-color: #008080\"> A </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">plate of food with arugula, avocado, and radishes.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mHumans can easily string together abstract concepts to form a coherent\u001b[0m\n",
              "\u001b[90msentence. An example of a sentence with the concepts \u001b[0m\u001b[90m\"food\"\u001b[0m\u001b[90m, \u001b[0m\u001b[90m\"avocado\"\u001b[0m\u001b[90m can be\u001b[0m\u001b[36m A \u001b[0m\n",
              "\u001b[36mplate of food with arugula, avocado, and radishes.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Humans can easily string together abstract concepts to form a coherent\n",
        "sentence. An example of a sentence with the concepts \"food\", \"avocado\" can be\"\"\"\n",
        "t_generate(prompt, \"t0\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "4YtZQr5Hlpad",
        "outputId": "73861d47-8626-4f94-def4-c11b49b0b0d7"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Humans can easily string together abstract concepts to form a coherent</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">sentence. An example of a sentence with the concepts </span><span style=\"color: #808080; text-decoration-color: #808080\">\"bones\"</span><span style=\"color: #808080; text-decoration-color: #808080\">, </span><span style=\"color: #808080; text-decoration-color: #808080\">\"avocado\"</span><span style=\"color: #808080; text-decoration-color: #808080\">, </span><span style=\"color: #808080; text-decoration-color: #808080\">\"puppy\"</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">can be</span><span style=\"color: #008080; text-decoration-color: #008080\"> A puppy with bones and avocados in it.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mHumans can easily string together abstract concepts to form a coherent\u001b[0m\n",
              "\u001b[90msentence. An example of a sentence with the concepts \u001b[0m\u001b[90m\"bones\"\u001b[0m\u001b[90m, \u001b[0m\u001b[90m\"avocado\"\u001b[0m\u001b[90m, \u001b[0m\u001b[90m\"puppy\"\u001b[0m\n",
              "\u001b[90mcan be\u001b[0m\u001b[36m A puppy with bones and avocados in it.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Humans can easily string together abstract concepts to form a coherent\n",
        "sentence. An example of a sentence with the concepts \"bones\", \"avocado\", \"puppy\" can be\"\"\"\n",
        "t_generate(prompt, \"t0\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lWVCXzTZnIFE"
      },
      "source": [
        "### Generating *stories*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "id": "5670zZm-qxnu",
        "outputId": "7c117227-5744-42d9-8cc0-51bef42057d4"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Generate a story about a one-eyed dog called Rufus.</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080\">CNN</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">)</span><span style=\"color: #008080; text-decoration-color: #008080\"> -- Rufus is a </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">one-eyed dog who loves to play fetch. He's also a bit of a celebrity in his </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">neighborhood. Rufus is a stray who was found in a park in the San Francisco Bay </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Area. He's a</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mGenerate a story about a one-eyed dog called Rufus.\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m(\u001b[0m\u001b[36mCNN\u001b[0m\u001b[1;36m)\u001b[0m\u001b[36m -- Rufus is a \u001b[0m\n",
              "\u001b[36mone-eyed dog who loves to play fetch. He's also a bit of a celebrity in his \u001b[0m\n",
              "\u001b[36mneighborhood. Rufus is a stray who was found in a park in the San Francisco Bay \u001b[0m\n",
              "\u001b[36mArea. He's a\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Generate a story about a one-eyed dog called Rufus.\"\"\"\n",
        "t_generate(prompt, \"t0\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8JcS_mNAf4wg"
      },
      "source": [
        "### [Extra] Testing T0 with GPT-3 generative tasks\n",
        "\n",
        "We've tried the Title to emoji task but T0 does not support emojis. Instead, we've defined a similar task where we create a summary of movies based on the title. \n",
        "\n",
        "Another interesting idea to try out could be give it a list of concepts and ask it to generate or determine the movie title."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 321
        },
        "id": "k6eo8tYXfrHe",
        "outputId": "bdff35dc-8e81-43fd-b61a-4320786b3985"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">(</span><span style=\"color: #800080; text-decoration-color: #800080\">davinci</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">)</span><span style=\"color: #800080; text-decoration-color: #800080\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Task: Write a </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">5</span><span style=\"color: #808080; text-decoration-color: #808080\"> paragraph sumary on the novel </span><span style=\"color: #808080; text-decoration-color: #808080\">\"Of Rice and Lies\"</span><span style=\"color: #808080; text-decoration-color: #808080\">:</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">When she was just a child, Tami and her family fled their home in Liberia during</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">a civil war. They eventually settled in Philadelphia, where Tami had to learn a </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">new language and culture. Despite the challenges, Tami was determined to make a </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">new life for herself and her family.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Now a teenager, Tami is struggling to find her place in the world. She feels </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">like an outsider at her school and in her community. When Tami is given the </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">opportunity to travel to Liberia for a school project, she jumps at the chance.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">In Liberia, Tami is reunited with her extended family and learns about her </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">culture and history. She also sees the devastating effects of the war that her </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">family fled. Tami is inspired to make a difference in the world and starts a </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">rice farm to help feed her community.</span>\n",
              "\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Tami's story is one of hope, determination, and resilience. Despite the </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">challenges she has faced, she has never given up. Tami is an inspiration to us </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">all.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;35m(\u001b[0m\u001b[35mdavinci\u001b[0m\u001b[1;35m)\u001b[0m\u001b[35m \u001b[0m\u001b[90mTask: Write a \u001b[0m\u001b[1;90m5\u001b[0m\u001b[90m paragraph sumary on the novel \u001b[0m\u001b[90m\"Of Rice and Lies\"\u001b[0m\u001b[90m:\u001b[0m\n",
              "\n",
              "\u001b[36mWhen she was just a child, Tami and her family fled their home in Liberia during\u001b[0m\n",
              "\u001b[36ma civil war. They eventually settled in Philadelphia, where Tami had to learn a \u001b[0m\n",
              "\u001b[36mnew language and culture. Despite the challenges, Tami was determined to make a \u001b[0m\n",
              "\u001b[36mnew life for herself and her family.\u001b[0m\n",
              "\n",
              "\u001b[36mNow a teenager, Tami is struggling to find her place in the world. She feels \u001b[0m\n",
              "\u001b[36mlike an outsider at her school and in her community. When Tami is given the \u001b[0m\n",
              "\u001b[36mopportunity to travel to Liberia for a school project, she jumps at the chance.\u001b[0m\n",
              "\n",
              "\u001b[36mIn Liberia, Tami is reunited with her extended family and learns about her \u001b[0m\n",
              "\u001b[36mculture and history. She also sees the devastating effects of the war that her \u001b[0m\n",
              "\u001b[36mfamily fled. Tami is inspired to make a difference in the world and starts a \u001b[0m\n",
              "\u001b[36mrice farm to help feed her community.\u001b[0m\n",
              "\n",
              "\u001b[36mTami's story is one of hope, determination, and resilience. Despite the \u001b[0m\n",
              "\u001b[36mchallenges she has faced, she has never given up. Tami is an inspiration to us \u001b[0m\n",
              "\u001b[36mall.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Task: Write a 5 paragraph sumary on the novel \"Of Rice and Lies\":\"\"\"\n",
        "\n",
        "gpt3_generate(prompt, model_name=\"davinci\", max_tokens=900)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 100
        },
        "id": "zEazC07CgBSP",
        "outputId": "3d991181-3856-41ef-9612-e618506da539"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">A two-column spreadsheet of top science fiction movies and the year of </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">release:</span>\n",
              "\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Title |  Year of release</span><span style=\"color: #008080; text-decoration-color: #008080\"> The list is arranged by year of release, with the most </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">recent on top.</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mA two-column spreadsheet of top science fiction movies and the year of \u001b[0m\n",
              "\u001b[90mrelease:\u001b[0m\n",
              "\n",
              "\u001b[90mTitle |  Year of release\u001b[0m\u001b[36m The list is arranged by year of release, with the most \u001b[0m\n",
              "\u001b[36mrecent on top.\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"A two-column spreadsheet of top science fiction movies and the year of release:\n",
        "Title |  Year of release\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "yZzt_0e8idqz",
        "outputId": "2ea7001d-92a7-49a1-9d89-0da4f33a739a"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Make a list of the most important philosophers in history.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">1</span><span style=\"color: #808080; text-decoration-color: #808080\">.</span><span style=\"color: #008080; text-decoration-color: #008080\"> Plato, Aristotle, Aquinas, Descartes, Spinoza, Hume, Kant, Nietzsche, Sartre,</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Hegel, and Nietzsche</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mMake a list of the most important philosophers in history.\u001b[0m\n",
              "\u001b[1;90m1\u001b[0m\u001b[90m.\u001b[0m\u001b[36m Plato, Aristotle, Aquinas, Descartes, Spinoza, Hume, Kant, Nietzsche, Sartre,\u001b[0m\n",
              "\u001b[36mHegel, and Nietzsche\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Make a list of the most important philosophers in history.\\n1.\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "Sv24ux2oiujz",
        "outputId": "3e5c6f8d-0af5-46aa-c506-ae5629918bf6"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">t0</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Make a list of the most important philosophers in history.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Early Greek philosophers </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">(</span><span style=\"color: #808080; text-decoration-color: #808080\">Name, estimated time</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">)</span><span style=\"color: #808080; text-decoration-color: #808080\">:</span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #008080; text-decoration-color: #008080\">,</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span><span style=\"color: #008080; text-decoration-color: #008080\"> BCE</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mt0\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mMake a list of the most important philosophers in history.\u001b[0m\n",
              "\u001b[90mEarly Greek philosophers \u001b[0m\u001b[1;90m(\u001b[0m\u001b[90mName, estimated time\u001b[0m\u001b[1;90m)\u001b[0m\u001b[90m:\u001b[0m\n",
              "\u001b[36m \u001b[0m\u001b[1;36m2\u001b[0m\u001b[36m,\u001b[0m\u001b[1;36m500\u001b[0m\u001b[36m BCE\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Make a list of the most important philosophers in history.\n",
        "Early Greek philosophers (Name, estimated time):\\n\"\"\"\n",
        "t_generate(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CqaIt-802_wk"
      },
      "source": [
        "# [Tk-Instruct] Zero- and few-shot learning\n",
        "\n",
        "The other model we will be looking at, also published in early 2022, is called [Tk-Instruct](https://arxiv.org/abs/2204.07705). It's architecture resembles T5 but the models were trained on 70+ distinct language tasks with expert written instructions.\n",
        "\n",
        "![Tk-instruct was trained on the Natural Instructions benchmark 2](https://raw.githubusercontent.com/PastelBelem8/langsci-summer-school2022/main/imgs/training_datasets.png)\n",
        "\n",
        "Let us see how it fares!\n",
        "\n",
        "\n",
        "**NOTE**: Make sure you run the setup for T0.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6JjqKqIkpEcG"
      },
      "source": [
        "## Defining tasks with Tk-instruct \n",
        "\n",
        "\n",
        "Each task constitutes:\n",
        "- **definition**: the natural language definition of the task that you'd like your model to perform. This involves a \"complete definition of how an input text is expected to be mapped to an output text\".\n",
        "- **positive examples**: are samples of inputs and correct outputs to them, along with a short explanation for each.\n",
        "- **negative examples**: are sample inputs with incorrect/invalid outputs to them, along with a short explanation for each.\n",
        "- **instances**: large collection of input-output pairs for each task. Each instance consists of a textual input and a list of acceptable textual outputs.\n",
        "\n",
        "Consider the following schematics: \n",
        "\n",
        "![Schema of an input example for Tk-instruct](https://raw.githubusercontent.com/PastelBelem8/langsci-summer-school2022/main/imgs/Screenshot%20from%202022-08-05%2013-30-07.png)\n",
        "\n",
        "Consider the following sentiment classification examples. Possible prompts for `tk-instruct` could be: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "wY_vm56xpD4F",
        "outputId": "ace6eb3a-a77d-4bac-a7aa-273cc69f8491"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">tk-3b</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Given a review, classify it into one of </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">4</span><span style=\"color: #808080; text-decoration-color: #808080\"> categories: Positive, </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Negative, Neutral, or Mixed. </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">The pumpkin was one of the worst that I've had in my life.</span><span style=\"color: #008080; text-decoration-color: #008080\"> Negative</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mtk-3b\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mGiven a review, classify it into one of \u001b[0m\u001b[1;90m4\u001b[0m\u001b[90m categories: Positive, \u001b[0m\n",
              "\u001b[90mNegative, Neutral, or Mixed. \u001b[0m\n",
              "\u001b[90mThe pumpkin was one of the worst that I've had in my life.\u001b[0m\u001b[36m Negative\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Given a review, classify it into one of 4 categories: Positive, Negative, Neutral, or Mixed. \n",
        "The pumpkin was one of the worst that I've had in my life.\"\"\"\n",
        "\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "id": "VtxU3O9qq4xS",
        "outputId": "a0b535d8-0479-4548-fc06-472130cf826d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">tk-3b</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">\"Definition: return the currency of the given country. Now complete the </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">following example - Input: India. Output:</span><span style=\"color: #008080; text-decoration-color: #008080\"> Indian Rupee</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mtk-3b\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90m\"Definition: return the currency of the given country. Now complete the \u001b[0m\n",
              "\u001b[90mfollowing example - Input: India. Output:\u001b[0m\u001b[36m Indian Rupee\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"\"Definition: return the currency of the given country. Now complete the following example - Input: India. Output:\"\"\"\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "id": "THth0DX2rAg4",
        "outputId": "72d7af65-e809-4cb8-e8af-82d16b38d6d9"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">tk-3b</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Definition: negate the following sentence. Input: John went to school. </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Output:</span><span style=\"color: #008080; text-decoration-color: #008080\"> John did not go to school</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mtk-3b\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mDefinition: negate the following sentence. Input: John went to school. \u001b[0m\n",
              "\u001b[90mOutput:\u001b[0m\u001b[36m John did not go to school\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Definition: negate the following sentence. Input: John went to school. Output:\"\"\"\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nufe5YANqwBZ"
      },
      "source": [
        "## Numerical reasoning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "id": "mpvj3CDWpwB1",
        "outputId": "7f3c8c8d-4db8-41d0-e331-c2e218d0ec2e"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">tk-3b</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">What is </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">?</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mtk-3b\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mWhat is \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m?\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m62\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Definition: add the following numbers. Input: 13 plus 18. Output:\"\"\"\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        },
        "id": "QMcSTYpIrTB2",
        "outputId": "ebe73807-4f59-43df-ab23-4d56278b4120"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">tk-3b</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Definition: Given a textual description of an addition operation, </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">expressed as </span><span style=\"color: #808080; text-decoration-color: #808080\">\"X plus Y\"</span><span style=\"color: #808080; text-decoration-color: #808080\"> determine the result of adding those numbers.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Positive Example </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">1</span><span style=\"color: #808080; text-decoration-color: #808080\">-</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  input: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">8</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  output: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">11</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  explanation: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\">+</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">8</span><span style=\"color: #808080; text-decoration-color: #808080\">=</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">11</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Positive Example </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\">-</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  input: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">15</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">7</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  output: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  explanation: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">15</span><span style=\"color: #808080; text-decoration-color: #808080\">+</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">7</span><span style=\"color: #808080; text-decoration-color: #808080\">=</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">22</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Positive Example </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">3</span><span style=\"color: #808080; text-decoration-color: #808080\">-</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  input: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  output: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">15</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  explanation: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\">+</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\">=</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">15</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Positive Example </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">4</span><span style=\"color: #808080; text-decoration-color: #808080\">-</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  input: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">15</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  output: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">33</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  explanation: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">15</span><span style=\"color: #808080; text-decoration-color: #808080\">+</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span><span style=\"color: #808080; text-decoration-color: #808080\">=</span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">33</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Now complete the following example-</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  input: </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">13</span><span style=\"color: #808080; text-decoration-color: #808080\"> plus </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">18</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">  output:</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">33</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mtk-3b\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mDefinition: Given a textual description of an addition operation, \u001b[0m\n",
              "\u001b[90mexpressed as \u001b[0m\u001b[90m\"X plus Y\"\u001b[0m\u001b[90m determine the result of adding those numbers.\u001b[0m\n",
              "\u001b[90mPositive Example \u001b[0m\u001b[1;90m1\u001b[0m\u001b[90m-\u001b[0m\n",
              "\u001b[90m  input: \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m8\u001b[0m\n",
              "\u001b[90m  output: \u001b[0m\u001b[1;90m11\u001b[0m\n",
              "\u001b[90m  explanation: \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m+\u001b[0m\u001b[1;90m8\u001b[0m\u001b[90m=\u001b[0m\u001b[1;90m11\u001b[0m\n",
              "\u001b[90mPositive Example \u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m-\u001b[0m\n",
              "\u001b[90m  input: \u001b[0m\u001b[1;90m15\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m7\u001b[0m\n",
              "\u001b[90m  output: \u001b[0m\u001b[1;90m22\u001b[0m\n",
              "\u001b[90m  explanation: \u001b[0m\u001b[1;90m15\u001b[0m\u001b[90m+\u001b[0m\u001b[1;90m7\u001b[0m\u001b[90m=\u001b[0m\u001b[1;90m22\u001b[0m\n",
              "\u001b[90mPositive Example \u001b[0m\u001b[1;90m3\u001b[0m\u001b[90m-\u001b[0m\n",
              "\u001b[90m  input: \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m2\u001b[0m\n",
              "\u001b[90m  output: \u001b[0m\u001b[1;90m15\u001b[0m\n",
              "\u001b[90m  explanation: \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m+\u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m=\u001b[0m\u001b[1;90m15\u001b[0m\n",
              "\u001b[90mPositive Example \u001b[0m\u001b[1;90m4\u001b[0m\u001b[90m-\u001b[0m\n",
              "\u001b[90m  input: \u001b[0m\u001b[1;90m15\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\n",
              "\u001b[90m  output: \u001b[0m\u001b[1;90m33\u001b[0m\n",
              "\u001b[90m  explanation: \u001b[0m\u001b[1;90m15\u001b[0m\u001b[90m+\u001b[0m\u001b[1;90m18\u001b[0m\u001b[90m=\u001b[0m\u001b[1;90m33\u001b[0m\n",
              "\u001b[90mNow complete the following example-\u001b[0m\n",
              "\u001b[90m  input: \u001b[0m\u001b[1;90m13\u001b[0m\u001b[90m plus \u001b[0m\u001b[1;90m18\u001b[0m\n",
              "\u001b[90m  output:\u001b[0m\u001b[36m \u001b[0m\u001b[1;36m33\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Definition: Given a textual description of an addition operation, expressed as \"X plus Y\" determine the result of adding those numbers.\n",
        "Positive Example 1-\n",
        "  input: 3 plus 8\n",
        "  output: 11\n",
        "  explanation: 3+8=11\n",
        "Positive Example 2-\n",
        "  input: 15 plus 7\n",
        "  output: 22\n",
        "  explanation: 15+7=22\n",
        "Positive Example 3-\n",
        "  input: 13 plus 2\n",
        "  output: 15\n",
        "  explanation: 13+2=15\n",
        "Positive Example 4-\n",
        "  input: 15 plus 18\n",
        "  output: 33\n",
        "  explanation: 15+18=33\n",
        "Now complete the following example-\n",
        "  input: 13 plus 18\n",
        "  output:\"\"\"\n",
        "\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DDkGAWT4ua1Y"
      },
      "source": [
        "## Coreference Resolution"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "id": "M8gBWSmHua-u",
        "outputId": "63ce9c94-59ab-478c-9786-d4c776bddb18"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">tk-3b</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Given two sentences decide who the pronoun </span><span style=\"color: #808080; text-decoration-color: #808080\">'her'</span><span style=\"color: #808080; text-decoration-color: #808080\"> refers to. </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">Barack Obama nominated Hilary Clinton as his secretary of state on Monday. He </span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">chose her because she had foreign affairs experience as a former First Lady.</span><span style=\"color: #008080; text-decoration-color: #008080\"> </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">Hilary Clinton</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mtk-3b\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mGiven two sentences decide who the pronoun \u001b[0m\u001b[90m'her'\u001b[0m\u001b[90m refers to. \u001b[0m\n",
              "\u001b[90mBarack Obama nominated Hilary Clinton as his secretary of state on Monday. He \u001b[0m\n",
              "\u001b[90mchose her because she had foreign affairs experience as a former First Lady.\u001b[0m\u001b[36m \u001b[0m\n",
              "\u001b[36mHilary Clinton\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Given two sentences decide who the pronoun 'her' refers to. \n",
        "Barack Obama nominated Hilary Clinton as his secretary of state on Monday. He chose her because she had foreign affairs experience as a former First Lady.\"\"\"\n",
        "\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8pjIepaduvv6"
      },
      "source": [
        "## Linguistic Acceptability"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "FNlbBhF7uv3d",
        "outputId": "c2d9bcee-ba3e-4c2a-97a5-55a31bf56529"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">tk-3b</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Given an English sentence, classify it according to its grammar into one</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">of </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\"> categories: correct, or incorrect.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">They can sing.</span><span style=\"color: #008080; text-decoration-color: #008080\"> incorrect</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mtk-3b\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mGiven an English sentence, classify it according to its grammar into one\u001b[0m\n",
              "\u001b[90mof \u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m categories: correct, or incorrect.\u001b[0m\n",
              "\u001b[90mThey can sing.\u001b[0m\u001b[36m incorrect\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Given an English sentence, classify it according to its grammar into one of 2 categories: correct, or incorrect.\n",
        "They can sing.\"\"\"\n",
        "\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "_NwKJGQkvRJu",
        "outputId": "2e7b52a6-bd4f-4c88-9363-382f2a9cdb65"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">tk-3b</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Given an English sentence, classify it according to its grammar into one</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">of </span><span style=\"color: #808080; text-decoration-color: #808080; font-weight: bold\">2</span><span style=\"color: #808080; text-decoration-color: #808080\"> categories: incorrect or correct.</span>\n",
              "<span style=\"color: #808080; text-decoration-color: #808080\">I love you.</span><span style=\"color: #008080; text-decoration-color: #008080\"> incorrect</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mtk-3b\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mGiven an English sentence, classify it according to its grammar into one\u001b[0m\n",
              "\u001b[90mof \u001b[0m\u001b[1;90m2\u001b[0m\u001b[90m categories: incorrect or correct.\u001b[0m\n",
              "\u001b[90mI love you.\u001b[0m\u001b[36m incorrect\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Given an English sentence, classify it according to its grammar into one of 2 categories: incorrect or correct.\n",
        "I love you.\"\"\"\n",
        "\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yu6JAndqvnk4"
      },
      "source": [
        "### Exercise\n",
        "\n",
        "Try it yourself. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aQSmkza3vns_"
      },
      "outputs": [],
      "source": [
        "prompt = ... # TODO:\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2_et7Xzpvupt"
      },
      "source": [
        "## Miscellaneous"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "id": "_cqJgqiSvrDl",
        "outputId": "ba0f11a8-4f02-4458-b1e9-eeb5bc744bd2"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">tk-3b</span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">)</span><span style=\"color: #008000; text-decoration-color: #008000\"> </span><span style=\"color: #808080; text-decoration-color: #808080\">Generate a story about a one-eyed dog called Rufus.</span><span style=\"color: #008080; text-decoration-color: #008080\"> Rufus is a dog who </span>\n",
              "<span style=\"color: #008080; text-decoration-color: #008080\">has been left blind in one eye. Rufus is very</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1;32m(\u001b[0m\u001b[32mtk-3b\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m \u001b[0m\u001b[90mGenerate a story about a one-eyed dog called Rufus.\u001b[0m\u001b[36m Rufus is a dog who \u001b[0m\n",
              "\u001b[36mhas been left blind in one eye. Rufus is very\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prompt = \"\"\"Generate a story about a one-eyed dog called Rufus.\"\"\"\n",
        "t_generate(prompt, \"tk-3b\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZnTEqtttvSEU"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "GljMIbNUeGOz",
        "y2pILetEeS6C",
        "AAtynFyISfEs",
        "4SJYAz208Rae",
        "Ew-RRSXzDU-G",
        "_UW5MTUzEC6v",
        "sQRYaokSAjkG",
        "-35Sos3hdH1l",
        "NiChNU_MdmDs"
      ],
      "name": "Day5-Zero-Few-Shot.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3.9.7 ('ucinlp-env')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.7"
    },
    "vscode": {
      "interpreter": {
        "hash": "07e38c7cd0c4af672a7daa78f250d60c001757599a58ae092086f274e28b86ff"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}