05_train_llm_starter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1dfae479-9399-492d-acaa-d9751615ee86",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Finetuning a language model\n",
    "\n",
    "<!--- @wandbcode{dlai_05} -->\n",
    "\n",
    "Let's see how to finetune a language model to generate character backstories using HuggingFace Trainer with wandb integration. We'll use a tiny language model (`TinyStories-33M`) due to resource constraints, but the lessons you learn here should be applicable to large models too!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a1f0e67f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "from datasets import load_dataset\n",
    "from transformers import AutoModelForCausalLM\n",
    "from transformers import Trainer, TrainingArguments\n",
    "import transformers\n",
    "transformers.set_seed(42)\n",
    "\n",
    "import wandb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f79c25e3-5f18-4457-84e1-ed2c0d262222",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mkawamura-ryota-94\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wandb.login(anonymous=\"allow\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2286ae41-213d-480d-a4ba-8c4e2e1c4771",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_checkpoint = \"roneneldan/TinyStories-33M\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fd80268-c4a1-4e1a-aed3-cd5c3ab4d48f",
   "metadata": {},
   "source": [
    "### Preparing data\n",
    "\n",
    "We'll start by loading a dataset containing Dungeons and Dragons character biographies from Huggingface. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9288a8e-b19b-4bd2-a72c-7dda03632282",
   "metadata": {},
   "source": [
    "> You can expect to get some warning here, this is ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a7535b8b-d220-44e8-a56c-97e250c36596",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0db40193e33141f39cf961753d35940e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/262 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset parquet/MohamedRashad--characters_backstories to C:/Users/kawam/.cache/huggingface/datasets/MohamedRashad___parquet/MohamedRashad--characters_backstories-6398ba4bb1a6e421/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8bbbdcfe85c042069ab68c046c3cd951",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a01797123e384480a2acadc86ce32b43",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/2.07M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17e2475a0a094bf2a22613f154831046",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset parquet downloaded and prepared to C:/Users/kawam/.cache/huggingface/datasets/MohamedRashad___parquet/MohamedRashad--characters_backstories-6398ba4bb1a6e421/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec. Subsequent calls will reuse this data.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b882d1eecce43fd8248c441bf37a8ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds = load_dataset('MohamedRashad/characters_backstories')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "13caeb7f-8a07-4ca2-a770-5b627238c2ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': 'Generate Backstory based on following information\\nCharacter Name: Dewin \\nCharacter Race: Halfling\\nCharacter Class: Sorcerer bard\\n\\nOutput:\\n',\n",
       " 'target': 'Dewin thought he was a wizard, but it turned out it was the draconic blood in his veins that brought him eldritch power.  Music classes in wizarding college taught him yet another use for his power, and when he was expelled he took up adventuring'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at one example\n",
    "ds[\"train\"][400]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7dae9106-8015-43da-a6d9-1124dee4bdde",
   "metadata": {},
   "outputs": [],
   "source": [
    "# As this dataset has no validation split, we will create one\n",
    "ds = ds[\"train\"].train_test_split(test_size=0.2, seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7ea1602d-504b-43de-87ad-fcb35b9e61f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20c4433505604e5d82b45c43bdc59530",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)okenizer_config.json:   0%|          | 0.00/722 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\kawam\\.pyenv\\pyenv-win\\versions\\3.11.3\\Lib\\site-packages\\huggingface_hub\\file_download.py:133: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\\Users\\kawam\\.cache\\huggingface\\hub. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.\n",
      "To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development\n",
      "  warnings.warn(message)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "98ccc9a07d69412e910cc287ea03a14a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)olve/main/vocab.json:   0%|          | 0.00/798k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20811600d2de476f85e7c4c4dce3c52f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)olve/main/merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "217600eba5c94f1e8770bebc34e9d2ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)cial_tokens_map.json:   0%|          | 0.00/438 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1857 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/465 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# We'll create a tokenizer from model checkpoint\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=False)\n",
    "\n",
    "# We'll need padding to have same length sequences in a batch\n",
    "tokenizer.pad_token = tokenizer.eos_token\n",
    "\n",
    "# Define a tokenization function that first concatenates text and target\n",
    "def tokenize_function(example):\n",
    "    merged = example[\"text\"] + \" \" + example[\"target\"]\n",
    "    batch = tokenizer(merged, padding='max_length', truncation=True, max_length=128)\n",
    "    batch[\"labels\"] = batch[\"input_ids\"].copy()\n",
    "    return batch\n",
    "\n",
    "# Apply it on our dataset, and remove the text columns\n",
    "tokenized_datasets = ds.map(tokenize_function, remove_columns=[\"text\", \"target\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a42417b8-ffa8-4d96-92ea-d8d949d87d5e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generate Backstory based on following information\n",
      "Character Name: Mr. Gale\n",
      "Character Race: Half-orc\n",
      "Character Class: Cleric\n",
      "\n",
      "Output:\n",
      " Growing up the only half-orc in a small rural town was rough. His mother didn't survive childbirth and so was raised in a church in a high mountain pass, his attention was always drawn by airships passing through, and dreams of an escape. Leaving to strike out on his own as early as he could he made a living for most of his life as an airship sailor, and occasionally a pirate. A single storm visits him throughout his life, marking every major\n"
     ]
    }
   ],
   "source": [
    "# Let's check out one prepared example\n",
    "print(tokenizer.decode(tokenized_datasets[\"train\"][900]['input_ids']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e8d6b17-a63d-41f1-92cf-416064b52156",
   "metadata": {},
   "source": [
    "### Training\n",
    "Let's finetune a pretrained language model on our dataset using HF Transformers and their wandb integration. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b4f131eb-979e-40f6-9e28-19756beaa8e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7578a31a9e2045a4bfa669e57cd795c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)lve/main/config.json:   0%|          | 0.00/968 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4a560a364cf2438abd3f6fbf8389c8d3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading pytorch_model.bin:   0%|          | 0.00/291M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# We will train a causal (autoregressive) language model from a pretrained checkpoint\n",
    "model = AutoModelForCausalLM.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7345ab23-8d12-4d4c-a39d-bb2202bff218",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.15.8"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>C:\\Users\\kawam\\DeepLearning.AI\\Evaluating-and-Debugging-Generative-AI\\wandb\\run-20230803_213551-mhvho6p8</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning/runs/mhvho6p8' target=\"_blank\">zany-butterfly-1</a></strong> to <a href='https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning' target=\"_blank\">https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning/runs/mhvho6p8' target=\"_blank\">https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning/runs/mhvho6p8</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Start a new wandb run\n",
    "run = wandb.init(project='dlai_lm_tuning', job_type=\"training\", anonymous=\"allow\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d74ee155-3c30-4ef2-9c4d-fd8ee222c50c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define training arguments\n",
    "model_name = model_checkpoint.split(\"/\")[-1]\n",
    "training_args = TrainingArguments(\n",
    "    f\"{model_name}-finetuned-characters-backstories\",\n",
    "    report_to=\"wandb\", # we need one line to track experiments in wandb\n",
    "    num_train_epochs=1,\n",
    "    logging_steps=1,\n",
    "    evaluation_strategy = \"epoch\",\n",
    "    learning_rate=1e-4,\n",
    "    weight_decay=0.01,\n",
    "    no_cuda=True, # force cpu use, will be renamed `use_cpu`\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "af62105f-a478-436f-88a2-5c1d78b9d20a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll use HF Trainer\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_datasets[\"train\"],\n",
    "    eval_dataset=tokenized_datasets[\"test\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "01958a56-c22a-4a27-bc71-41c59fc97f05",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\kawam\\.pyenv\\pyenv-win\\versions\\3.11.3\\Lib\\site-packages\\transformers\\optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='233' max='233' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [233/233 08:02, Epoch 1/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.942200</td>\n",
       "      <td>3.345127</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=233, training_loss=3.75183794119839, metrics={'train_runtime': 484.7648, 'train_samples_per_second': 3.831, 'train_steps_per_second': 0.481, 'total_flos': 40423258718208.0, 'train_loss': 3.75183794119839, 'epoch': 1.0})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's train!\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7911e43f-f4ce-4855-9f68-662438af8d24",
   "metadata": {},
   "outputs": [],
   "source": [
    "transformers.logging.set_verbosity_error() # suppress tokenizer warnings\n",
    "\n",
    "prefix = \"Generate Backstory based on following information Character Name: \"\n",
    "\n",
    "prompts = [\n",
    "    \"Frogger Character Race: Aarakocra Character Class: Ranger Output: \",\n",
    "    \"Smarty Character Race: Aasimar Character Class: Cleric Output: \",\n",
    "    \"Volcano Character Race: Android Character Class: Paladin Output: \",\n",
    "]\n",
    "\n",
    "table = wandb.Table(columns=[\"prompt\", \"generation\"])\n",
    "\n",
    "for prompt in prompts:\n",
    "    input_ids = tokenizer.encode(prefix + prompt, return_tensors=\"pt\")\n",
    "    output = model.generate(input_ids, do_sample=True, max_new_tokens=50, top_p=0.3)\n",
    "    output_text = tokenizer.decode(output[0], skip_special_tokens=True)\n",
    "    table.add_data(prefix + prompt, output_text)\n",
    "    \n",
    "wandb.log({'tiny_generations': table})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7caafaf-dfb7-413d-a329-8520aea5f73a",
   "metadata": {},
   "source": [
    "**Note**: LLM's don't always generate the same results. Your generated characters and backstories may differ from the video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3083c6a3-fdb8-44ab-a028-c0a222a2fdef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: left ; width: auto;} td:nth-child(2) {text-align: left ; width: 100%}\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; justify-content: flex-start; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\"><h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>eval/loss</td><td>▁</td></tr><tr><td>eval/runtime</td><td>▁</td></tr><tr><td>eval/samples_per_second</td><td>▁</td></tr><tr><td>eval/steps_per_second</td><td>▁</td></tr><tr><td>train/epoch</td><td>▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇████</td></tr><tr><td>train/global_step</td><td>▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇████</td></tr><tr><td>train/learning_rate</td><td>████▇▇▇▇▇▆▆▆▆▆▆▅▅▅▅▅▄▄▄▄▄▄▃▃▃▃▃▂▂▂▂▂▂▁▁▁</td></tr><tr><td>train/loss</td><td>█▅▃▂▂▃▄▁▂▁▂▂▂▂▃▃▂▂▂▁▂▂▂▂▃▂▂▂▁▂▂▃▁▁▁▁▂▂▁▁</td></tr><tr><td>train/total_flos</td><td>▁</td></tr><tr><td>train/train_loss</td><td>▁</td></tr><tr><td>train/train_runtime</td><td>▁</td></tr><tr><td>train/train_samples_per_second</td><td>▁</td></tr><tr><td>train/train_steps_per_second</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\"><h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>eval/loss</td><td>3.34513</td></tr><tr><td>eval/runtime</td><td>37.7081</td></tr><tr><td>eval/samples_per_second</td><td>12.332</td></tr><tr><td>eval/steps_per_second</td><td>1.565</td></tr><tr><td>train/epoch</td><td>1.0</td></tr><tr><td>train/global_step</td><td>233</td></tr><tr><td>train/learning_rate</td><td>0.0</td></tr><tr><td>train/loss</td><td>3.9422</td></tr><tr><td>train/total_flos</td><td>40423258718208.0</td></tr><tr><td>train/train_loss</td><td>3.75184</td></tr><tr><td>train/train_runtime</td><td>484.7648</td></tr><tr><td>train/train_samples_per_second</td><td>3.831</td></tr><tr><td>train/train_steps_per_second</td><td>0.481</td></tr></table><br/></div></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run <strong style=\"color:#cdcd00\">zany-butterfly-1</strong> at: <a href='https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning/runs/mhvho6p8' target=\"_blank\">https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning/runs/mhvho6p8</a><br/> View job at <a href='https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning/jobs/QXJ0aWZhY3RDb2xsZWN0aW9uOjg3NjMyNDQ2/version_details/v0' target=\"_blank\">https://wandb.ai/kawamura-ryota-94/dlai_lm_tuning/jobs/QXJ0aWZhY3RDb2xsZWN0aW9uOjg3NjMyNDQ2/version_details/v0</a><br/>Synced 5 W&B file(s), 1 media file(s), 3 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>.\\wandb\\run-20230803_213551-mhvho6p8\\logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfa3bfcc-2885-4eb8-8a18-a236c69e1a98",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}