[Q&A] High CPU usage due to too many "Worker Processes" in NVFlare

[dima1997]

Python version (python3 -V)

3.8

NVFlare version (python3 -m pip list | grep "nvflare")

2.3.8

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 18.04.3 LTS

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

Hello,

I'm using NVIDIA FLARE to run federated learning trainings across two clients and a server that were set up on the same machine. Each time a job is summited, many identical "nvflare.private.fed.app.client.worker_process" are spawned increasing the CPU and memory usage.

Could you help me reduce these "worker_process" in order to reduce CPU usage too, please ?

Thanks in advance,

DI MARIA, Franco Martin

Environment
Nvflare Version: 2.3.8
GPU Type: Quadro RTX 6000
Nvidia Driver Version: 460.80
CUDA Version: 11.2
CUDNN Version: 7.6.5
Operating System + Version: Ubuntu 18.04.3 LTS
Python Version (if applicable): 3.8.0

[YuanTingHsieh]

Hi @dima1997

Since you are running 2 clients and 1 server on the same machine.
One way to reduce the CPU usage is that you just run your job using "nvflare simulator".

And note that some of these processes/threads are spawned by the application.
So it varies job by job.
You might want to check how to reduce usage there (like reducing workers of torch.dataloader)

[dima1997]

Hi @YuanTingHsieh,

Thanks for the advice.

I find the same problem of many processes/threads being spawned when executing clients on different machines.

Is there a manner to reduce it in reduce CPU usage in this case too ?

Thanks in advance,

[YuanTingHsieh]

@dima1997 can you share what job you are running? Or what job from our examples is closer to your job