LUMI scripts - Mosaic/llm-foundry

[rlrs]

(Continued) Pretraining setup for LUMI. These scripts are now for a mosaicai/llm-foundry stack. Everything should work.

[rlrs]

And yes I know that there's a Huggingface token in there. It's invalid, need to find a better way to manage that.

[github-actions]

This PR is stale because it has been open 1+ days with no activity. Feel free to either 1) remove the stale label or 2) comment. If nothing happens, this will be closed in 7 days.

[rlrs]

Please don't close my draft PR because it's not active over xmas 😅

[rlrs]

Okay, so this whole thing works now. There are a few unused scripts containing some of my other attempts at setting things up on LUMI - I should perhaps move these somewhere else to keep things clean.

The important files are:

• make_venv.sh - creates the Python venv that all the nodes use to run the pretraining code.
• continue_mistral_mosaic.sh - this is the SLURM sbatch script that describes how many nodes to run on etc., and launches the following script in the correct Singularity container on each node.
• mosaic_in_container.sh - this script is run on in the container on each node and it simply sets up a few things before running the given training command.
• continue-mistral-7b.yaml - configuration file that describes which model to train, which hyperparams, which data, which evals etc.

Additionally, I've added two submodules (so you have to clone with --recurse-submodules) since these are core dependencies that we need to keep track of, and perhaps we should pin them to a certain commit instead of the head of a branch.
Everything else is unused in the current setup.

[KennethEnevoldsen]

Looks good but have a few questions. Will try to make it run on LUMI as well (after Christmas) and that might lead to more questions, but that does not need to hold the PR back.

[KennethEnevoldsen]

Okay, so this whole thing works now. There are a few unused scripts containing some of my other attempts at setting things up on LUMI - I should perhaps move these somewhere else to keep things clean.

Stuff that is still being worked on feel free to keep that, but stuff that could be deleted and recovered from the history (if needed) might as well be deleted

perhaps we should pin them to a certain commit instead of the head of a branch.

def. pin them

[github-actions]

This PR is stale because it has been open 1+ days with no activity. Feel free to either 1) remove the stale label or 2) comment. If nothing happens, this will be closed in 7 days.

[KennethEnevoldsen]

@rlrs will remove the stale label (this will give it another 7 days) as I assume you might be on vacation

[github-actions]

This PR is stale because it has been open 1+ days with no activity. Feel free to either 1) remove the stale label or 2) comment. If nothing happens, this will be closed in 7 days.

[KennethEnevoldsen]

Looks good!

[github-actions]

This PR is stale because it has been open 1+ days with no activity. Feel free to either 1) remove the stale label or 2) comment. If nothing happens, this will be closed in 7 days.