Code

LUMI scripts - Mosaic/llm-foundry#190

Code

Merged

rlrs merged 21 commits intomaincentre-for-humanities-computing/danish-foundation-models:mainfrom lumicentre-for-humanities-computing/danish-foundation-models:lumiCopy head branch name to clipboardJan 15, 2024

+524

Commits

Commits on Oct 20, 2023

• initial scripts for lumi training

  rlrs

  committed
  e73f3e8

  Copy full SHA for e73f3e8

• add requirements for HF 5.7 env

  rlrs

  committed
  166da6a

  Copy full SHA for 166da6a

• add LUMI readme

  rlrs

  committed
  7d239be

  Copy full SHA for 7d239be

Commits on Nov 16, 2023

• Merge remote-tracking branch 'origin/main' into lumi

  rlrs

  committed
  75526b9

  Copy full SHA for 75526b9

• working LUMI scripts

  rlrs

  committed
  124e67a

  Copy full SHA for 124e67a

Commits on Dec 9, 2023

• working scripts for multinode training

  rlrs

  committed
  02c886c

  Copy full SHA for 02c886c

Commits on Dec 20, 2023

• update submodules and venv

  rlrs

  committed
  844e2ae

  Copy full SHA for 844e2ae

• Update README.md

  rlrs

  authored
  fe0bb46

  Copy full SHA for fe0bb46

Commits on Jan 2, 2024

• add modified speedy data tokenization script

  rlrs

  committed
  c14a9e5

  Copy full SHA for c14a9e5

• refactor, generate splits

  rlrs

  committed
  62d9c79

  Copy full SHA for 62d9c79

• update readme

  rlrs

  committed
  609824a

  Copy full SHA for 609824a

Commits on Jan 3, 2024

• fix a dataset conversion error

  rlrs

  committed
  fac0119

  Copy full SHA for fac0119

Commits on Jan 4, 2024

• remove unused scripts

  rlrs

  committed
  59ff133

  Copy full SHA for 59ff133

• move scripts

  rlrs

  committed
  a026d6e

  Copy full SHA for a026d6e

• update scripts

  rlrs

  committed
  68022d2

  Copy full SHA for 68022d2

• update lumi readme

  rlrs

  committed
  17f2ac2

  Copy full SHA for 17f2ac2

• update continue script

  rlrs

  committed
  53236d3

  Copy full SHA for 53236d3

• composer/

  rlrs

  committed
  030cfc5

  Copy full SHA for 030cfc5

• Merge branch 'main' into lumi

  rlrs

  authored
  d60d128

  Copy full SHA for d60d128

Commits on Jan 15, 2024

• Merge branch 'main' into lumi

  rlrs

  authored
  1d32dbd

  Copy full SHA for 1d32dbd

• Update and rename convert_dataset_json.py to jsonl_to_mds.py

  rlrs

  authored
  1f25b55

  Copy full SHA for 1f25b55