Skip to content

Commit

Permalink
Update README.md to be more detailed
Browse files Browse the repository at this point in the history
  • Loading branch information
jshuadvd committed Jul 20, 2024
1 parent 61ff458 commit 54d8b56
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,10 @@ To train on a custom dataset:
Hyperparameter Tuning
LongRoPE's performance can be sensitive to hyperparameters. Key parameters to tune include:

`population_size`, `num_mutations`, and `num_crossovers` in the lambda factor search
Learning rate and scheduler parameters for fine-tuning
gradient_accumulation_steps for training stability

- **Base Length**: The initial context length used for fine-tuning.
- **Target Length**: The final context length to extend to.
- **Population Size**: The number of candidate solutions to search for during the search algorithm.
Expand Down

0 comments on commit 54d8b56

Please sign in to comment.