KTA3 - Multi-Task and Transfer Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "15883cac",
   "metadata": {},
   "source": [
    "# Multi-Task and Transfer Learning\n",
    "\n",
    "you can take the code of the model's architecture and training loop from the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce0a8101",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install avalanche-lib==0.5.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6e8387b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import avalanche\n",
    "import torchvision\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07d59804",
   "metadata": {},
   "source": [
    "# Datasets\n",
    "\n",
    "image classification datasets with 32x32 images\n",
    "\n",
    "- CIFAR10: 10 classes\n",
    "- CIFAR100: 100 classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7a5b50d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\w-32\\mambaforge\\envs\\791aa\\lib\\site-packages\\avalanche\\benchmarks\\scenarios\\supervised.py:389: UserWarning: stream generator will be converted to a list.\n",
      "  warnings.warn(\"stream generator will be converted to a list.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "from avalanche.benchmarks import SplitCIFAR10, SplitCIFAR100\n",
    "from avalanche.benchmarks import benchmark_with_validation_stream\n",
    "\n",
    "task0 = benchmark_with_validation_stream(SplitCIFAR10(n_experiences=1), validation_size=0.3, shuffle=True)\n",
    "task1 = benchmark_with_validation_stream(SplitCIFAR100(n_experiences=1), validation_size=0.3, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae14b46",
   "metadata": {},
   "source": [
    "Task 0 (CIFAR10) returns triplets `<x, y, 0>`, while Task 1 (CIFAR100) returns `<x, y, 1>`. You can use task labels in your code to select task-aware modules (e.g. a different classification head for each task)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a0ffe3b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 1)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# from avalanche.benchmarks.utils import make_classification_dataset\n",
    "\n",
    "# task 0 with task label 0\n",
    "t0_train_data = task0.train_stream[0].dataset\n",
    "t0_valid_data = task0.valid_stream[0].dataset\n",
    "t0_test_data = task0.test_stream[0].dataset\n",
    "\n",
    "# task 1 with task label 1\n",
    "t1_train_data = task1.train_stream[0].dataset\n",
    "t1_train_data = t1_train_data.update_data_attribute(\"targets_task_labels\", [1 for el in range(len(t1_train_data))])\n",
    "t1_valid_data = task1.valid_stream[0].dataset\n",
    "t1_valid_data = t1_valid_data.update_data_attribute(\"targets_task_labels\", [1 for el in range(len(t1_valid_data))])\n",
    "t1_test_data = task1.test_stream[0].dataset\n",
    "t1_test_data = t1_test_data.update_data_attribute(\"targets_task_labels\", [1 for el in range(len(t1_test_data))])\n",
    "\n",
    "# check task labels\n",
    "t0_train_data[0][2], t1_train_data[0][2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e23b0544",
   "metadata": {},
   "source": [
    "# Transfer Learning\n",
    "Implement the \"Simple Transfer Learning\" recipe that we have seen in the previous lecture.\n",
    "- pretrain on CIFAR10\n",
    "- finetune on CIFAR100\n",
    "\n",
    "Is it better than training from scratch?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09adfb6d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "89cab2d1",
   "metadata": {},
   "source": [
    "# Multi-Task Learning\n",
    "\n",
    "Train the two tasks together with a multi-head architecture\n",
    "- modify the ResNet18 with a multi-head classifier\n",
    "- modify the training loop to jointly sample from both tasks\n",
    "\n",
    "is it better than learning each task independently?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "784b7e68",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}