Build CUDA aware MOOSE - Issues to buils PETSc - SUPERLU_DIST

[DanoLV]

Check these boxes if you have followed the posting rules.

• Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (avoid screenshots if possible, triple back quotes before/after pasted text, etc)

Issue or question about MOOSE

Hello.
I have been trying to build MOOSE with CUDA aware OpenMPI an PETSc, but my efforts have been unsuccessful until now.
OpenMPI is CUDa aware in my build, but when I try to build PETSc it fails to make SUPERLU_DIST.

Configure_petsc.sh is been called wth the options:

python3 ./configure --with-64-bit-indices
--with-cxx-dialect=C++17
--with-debugging=no
--with-fortran-bindings=0
--with-mpi=1
--with-openmp=1
--with-cuda=1
--with-shared-libraries=1
--with-sowing=0
--download-fblaslapack=1
--download-hypre=1
--download-metis=1
--download-mumps=1
--download-ptscotch=1
--download-parmetis=1
--download-scalapack=1
--download-slepc=1
--download-strumpack=1
--download-superlu_dist=1
$HDF5_STR
$MUMPS_ARM_STR
$MAKE_NP_STR
"$@"

I atach the PETSc config.log.
configure.log

Thank in advance.

Diagnostics for MOOSE installation issues

[Optional] Output of the diagnostics scripts. To run the script: cd ~/projects/moose/scripts; ./diagnostics.sh

[GiudGiud]

Hello

You could remove superlu-dist by manually editing the update and rebuild petsc script

Guillaume

[lindsayad]

@balay I don't know too much about this, but looking in the configure.log, I see

/home/dano/discos/datos/proyects/moose/petsc/arch-moose/externalpackages/git.superlu_dist/SRC/pdgstrs_lsum_cuda.cu(1746): error: no instance of overloaded function "atomicAdd" matches the argument list

My top google hit for this was this stack overflow post which suggests this error might be fixable with flags. Have you run into this before?

[lindsayad]

I think that any tests using boomeramg will likely fail since boomeramg always assumes the vec and mat will be on the device but by default we just create those on the host

[Traiwit]

Hi @DanoLV

could you please share your update_and_rebuild_petsc.sh and configure_petsc.sh? i'm having hard time running update_and_rebuild_petsc.sh as well

also did you install CUDA 12.5, HDF5 1.14.4, UCX 1.16.0, and OpenMPI 5.0.3 to your linux or conda environment?

I'm super new to this compile PETSC and GPU stuff, I kept running into error after error, could you please guide me?

Thank you for you help!

Kind regards,
Traiwit

[lindsayad]

They gave a pretty detailed configure line. Browsing it, I believe the additional arguments you would pass to scripts/update_and_rebuild_petsc.sh if you want to mimic their configuration are:

scripts/update_and_rebuild_petsc.sh --with-cxx-dialect=C++17 --with-cuda=1 --with-cuda-dialect=cxx17 -CUDAOPTFLAGS="" -Wno-deprecated-gpu-targets --download-kokkos --download-kokkos-kernels

[lindsayad]

make sure that you've configured openmpi with --with-cuda

[balay]

/home/dano/discos/datos/proyects/moose/petsc/arch-moose/externalpackages/git.superlu_dist/SRC/pdgstrs_lsum_cuda.cu(1746): error: no instance of overloaded function "atomicAdd" matches the argument list

This issue was fixed by https://gitlab.com/petsc/petsc/-/merge_requests/7288/diffs?commit_id=cdfa477dea3fec089d157123d8a6f983b5ff4346

but yeah - generall - best to use latest version for GPU usage