how to improve the efficiency of HPC process?

[Sun-Liu-Yang]

I am trying to use MPI functionality for large-scale parallel computing on a supercomputer cluster. The number of solving units is in the tens of millions, and the number of degrees of freedom of nodes is in the billions.
When I was solving a problem with millions of degrees of freedom, I found that the solving efficiency was already very slow. The official suggested a single core degree of freedom of 20000, but currently my single core solving degree of freedom is only about 10000.
What aspects should I improve the calculation settings to significantly increase computational efficiency and meet the demand for solving billions of degrees of freedom?

[GiudGiud]

Hello

This is a complex problem and the simplest step to take is simply to profile a shorter run to see which step in your simulation is not scaling to tens of millions of cores. The limiting step is likely specific to your problem and to your solver setup.
https://mooseframework.inl.gov/application_development/profiling.html

[GiudGiud]

but when the model mesh from the abaqus(the .inp file ), only in one node can dealing fast.

do you mean with the split mesh? Reading an inp file directly would be slow since it's replicated.

There is an other way to run split meshes that might work better.
Remove --use-split and instead use a FileMeshGenerator with the file set to 2024-11-14-55.cpa.gz.
And add --distributed-mesh to the command line

[Sun-Liu-Yang]

I have try this method to solve the problem ,it seems not work inter-node.

[Sun-Liu-Yang]

this figure is the same model solved on 1nodes with 60cpus.

[GiudGiud]

I wonder if the partitioner is the problem here.

If it returned two different partitions on both runs, it would fail.

Can you try commenting it out? Parmetis is the default anyway, you are simply changing how we access parmetis by specifying this PetscPartitioner

[Sun-Liu-Yang]

you means when splitting the mesh and submit the job，commenting the [Partitioner] block out?
or
only commenting the [Partitioner] block out when submitting the job?

[Sun-Liu-Yang]

Just now, I comment the [Partitioner] block out both the two step, but it dont woke.

[GiudGiud]

I read the thread again. I think your executable may be hanging as soon as there is internode communication. The mesh preparation is the first step to have such communication.

You can try the same 2 node run without a split mesh to double check. You should see the same issue.
Do runs with just a Cartesian mesh generator solve properly? For them the first communication step will be during the solve

[Sun-Liu-Yang]

yes, the problem will be solve properly when the model mesh is constructed by the Moose's mesh geneartor (such as GeneratedMesh object )on more than one nodes.

you means that when dealing with the problem using the external mesh (such as .e, .inp file ), I should take more time for node communicatin until the preparing mesh state finshed ?

[GiudGiud]

yes, the problem will be solve properly when the model mesh is constructed by the Moose's mesh geneartor (such as GeneratedMesh object )on more than one nodes.

Ok that's good to know. So it is the FileMeshGenerator that is slow.
At this point it would be good to find out why.
You can either use a profiler (make sure to pick a mesh that is not too big so the mesh loading finishes):
https://mooseframework.inl.gov/application_development/profiling.html
or simply use a debugger and interrupt the code while it is "preparing" to see what it is actually is doing
https://mooseframework.inl.gov/application_development/debugging.html

[Sun-Liu-Yang]

Thank you very much , I will try to do this .
Whether you have met the similar case when dealing with problems using the external mesh (using the filemesh or filemeshgenerator)? I want to know whether is there some error when installing the moose the my HPC?

[GiudGiud]

Whether you have met the similar case when dealing with problems using the external mesh (using the filemesh or filemeshgenerator)?

Never had this issue. Splitting can be slow if done in parallel but loading splits has always been fine for me

I want to know whether is there some error when installing the moose the my HPC?

I dont know if it's that.