autotuning throws [Errno 36] File name too long

[eavidan]

I am trying to run an autotunning example, but keep getting the following error
would appreciate some help :)

my command is
deepspeed --autotuning run --num_gpus 2 --num_nodes 1 run_clm.py --model_name_or_path gpt2-xl --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --do_train --do_eval --output_dir /tmp/test-clm --deepspeed ds_config-auto.json

and the ds_config-auto.json

{
    "train_micro_batch_size_per_gpu": "auto",
    "fp16": {
      "enabled": true
    },
    "autotuning": {
      "enabled": true,
      "overwrite": false,
      "fast": true,
      "arg_mappings": {
        "train_micro_batch_size_per_gpu": "--per_device_train_batch_size",
        "gradient_accumulation_steps ": "--gradient_accumulation_steps"
      }
    }
  }

And the error

[2023-02-05 06:15:40,705] [WARNING] [runner.py:179:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-02-05 06:15:40,814] [INFO] [autotuner.py:59:__init__] Created autotuning experiments directory: autotuning_exps
[2023-02-05 06:15:40,815] [INFO] [autotuner.py:72:__init__] Created autotuning resutls directory: autotuning_exps
[2023-02-05 06:15:40,815] [INFO] [autotuner.py:200:_get_resource_manager] active_resources = OrderedDict([('localhost', [0, 1, 2, 3, 4, 5, 6, 7])])
[2023-02-05 06:15:40,815] [INFO] [runner.py:326:run_autotuning] [Start] Running autotuning
[2023-02-05 06:15:40,815] [INFO] [autotuner.py:418:tune] Fast mode is enabled. Tuning micro batch size only.
[2023-02-05 06:15:40,815] [INFO] [autotuner.py:689:model_info_profile_run] Starting model info profile run.
0%|          | 0/1 [00:00<?, ?it/s][2023-02-05 06:15:40,830] [INFO] [scheduler.py:352:run_experiment] Scheduler wrote ds_config to autotuning_results/profile_model_info/ds_config.json, /tmp/global/files/4b10_CliExecutor_path/autotuning_results/profile_model_info/ds_config.json
functionexecutor-run-bash-command-51ccb83-3173517951: [2023-02-05 06:15:40,831] [INFO] [scheduler.py:359:run_experiment] Scheduler wrote exp to autotuning_results/profile_model_info/exp.json, /tmp/global/files/4b10_CliExecutor_path/autotuning_results/profile_model_info/exp.json
[2023-02-05 06:15:40,832] [INFO] [scheduler.py:387:run_experiment] Launching exp_id = 0, exp_name = profile_model_info, with resource = localhost:0,1, and ds_config = /tmp/global/files/4b10_CliExecutor_path/autotuning_results/profile_model_info/ds_config.json
localhost: ssh: connect to host localhost port 22: Cannot assign requested address
pdsh@myhost: localhost: ssh exited with exit code 255
[2023-02-05 06:15:56,440] [INFO] [scheduler.py:443:clean_up] Done cleaning up exp_id = 0 on the following workers: localhost
[2023-02-05 06:15:56,441] [INFO] [scheduler.py:403:run_experiment] Done running exp_id = 0, exp_name = profile_model_info, with resource = localhost:0,1
100%|██████████| 1/1 [00:25<00:00, 25.02s/it][2023-02-05 06:16:05,847] [ERROR] [autotuner.py:728:model_info_profile_run] The model is not runnable with DeepSpeed with error = [Errno 36] File name too long: 'eyJ0cmFpbl9taWNyb19iYXRjaF9zaXplX3Blcl9ncHUiOiAxLCAiZnAxNiI6IHsiZW5hYmxlZCI6IHRydWV9LCAiYXV0b3R1bmluZyI6IHsiZW5hYmxlZCI6IHRydWUsICJtb2RlbF9pbmZvX3BhdGgiOiAiYXV0b3R1bmluZ19yZXN1bHRzL3Byb2ZpbGVfbW9kZWxfaW5mby9tb2RlbF9pbmZvLmpzb24iLCAibW9kZWxfaW5mbyI6IHsicHJvZmlsZSI6IHRydWV9LCAibWV0cmljX3BhdGgiOiAiYXV0b3R1bmluZ19yZXN1bHRzL3Byb2ZpbGVfbW9kZWxfaW5mby9tZXRyaWNzLmpzb24ifSwgInplcm9fb3B0aW1pemF0aW9uIjogeyJzdGFnZSI6IDN9LCAibWVtb3J5X2JyZWFrX2Rvd24iOiBmYWxzZX0='

[2023-02-05 06:16:05,847] [INFO] [runner.py:331:run_autotuning] [End] Running autotuning
[2023-02-05 06:16:05,847] [INFO] [autotuner.py:1153:run_after_tuning] No optimal DeepSpeed configuration found by autotuning.

[coolws]

same as you, has anyone solved this issue?

[SulRash]

Same here too, anybody end up fixing this? I'm having it happen on Azure to be particular.

[Zijie-Tian]

same problem.