Configuration
By default, MPI.jl will download and link against the following MPI implementations:
- Microsoft MPI on Windows
- MPICH on all other platforms
This is suitable for most single-node use cases, but for larger systems, such as HPC clusters or multi-GPU machines, you will probably want to configure against a system-provided MPI implementation in order to exploit features such as fast network interfaces and CUDA-aware or ROCm-aware MPI interfaces.
The MPIPreferences.jl package allows the user to choose which MPI implementation to use in MPI.jl. It uses Preferences.jl to configure the MPI backend for each project separately. This provides a single source of truth that can be used for JLL packages (Julia packages providing C libraries) that link against MPI. It can be installed by
julia --project -e 'using Pkg; Pkg.add("MPIPreferences")'
The way how MPI.jl is configured has changed with MPI.jl v0.20. See Migration from MPI.jl v0.19 or earlier for more information on how to migrate your configuration from earlier MPI.jl versions.
Using a system-provided MPI backend
Requirements
MPI.jl requires a shared library installation of a C MPI library, supporting the MPI 3.0 standard or later. The following MPI implementations should work out-of-the-box with MPI.jl:
- Open MPI
- MPICH (v3.1 or later)
- Intel MPI
- Microsoft MPI
- IBM Spectrum MPI
- MVAPICH
- Cray MPICH
- Fujitsu MPI
- HPE MPT/HMPT
Configuration
Run MPIPreferences.use_system_binary()
. This will attempt to locate and to identify any available MPI implementation, and create a file called LocalPreferences.toml
adjacent to the current Project.toml
.
julia --project -e 'using MPIPreferences; MPIPreferences.use_system_binary()'
If the implementation is changed, you will need to call this function again. See the MPIPreferences.use_system_binary
documentation for specific options.
You can copy LocalPreferences.toml
to a different project folder, but you must list MPIPreferences
in the [extras]
or [deps]
section of the Project.toml
for the settings to take effect.
Due to a bug in Julia (until v1.6.5
and v1.7.1
), getting preferences from transitive dependencies is broken (Preferences.jl#24). To fix this update your version of Julia, or add MPIPreferences
as a direct dependency to your project.
Notes to HPC cluster administrators
Preferences are merged across the Julia load path, such that it is feasible to provide a module file that appends a path to JULIA_LOAD_PATH
variable that contains system-wide preferences. The steps are as follows:
Run
MPIPreferences.use_system_binary()
, which will generate a fileLocalPreferences.toml
containing something like the following:[MPIPreferences] _format = "1.0" abi = "OpenMPI" binary = "system" libmpi = "/software/mpi/lib/libmpi.so" mpiexec = "/software/mpi/bin/mpiexec"
Create a file called
Project.toml
orJuliaProject.toml
in a central location, for example/software/mpi/julia
or in the same directory as the MPI library module, and add the following contents:[extras] MPIPreferences = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267" [preferences.MPIPreferences] _format = "1.0" abi = "OpenMPI" binary = "system" libmpi = "/software/mpi/lib/libmpi.so" mpiexec = "/software/mpi/bin/mpiexec"
updating the contents of the
[preferences.MPIPreferences]
section match those of the[MPIPreferences]
inLocalPreferences.toml
.Append the directory containing the file to the
JULIA_LOAD_PATH
environment variable, with a colon (:
) separator.If this variable is not already set, it should be prefixed with a colon to ensure correct behavior of the Julia load path, e.g.
JULIA_LOAD_PATH=":/software/mpi/julia"
. If using environment modules, this can be achieved withappend-path -d {} JULIA_LOAD_PATH :/software/mpi/julia
in the corresponding module file (preferably the module file for the MPI installation or for Julia).
The user can still provide differing MPI configurations for each Julia project that will take precedent by modifying the local
Project.toml
or by providing aLocalPreferences.toml
file.
Using an alternative JLL-provided MPI library
The following MPI implementations are provided as JLL packages and automatically obtained when installing MPI.jl:
MicrosoftMPI_jll
: Microsoft MPI Default for WindowsMPICH_jll
: MPICH. Default for all other systemsOpenMPI_jll
: Open MPIMPItrampoline_jll
: MPItrampoline: an MPI forwarding layer.
Call MPIPreferences.use_jll_binary
, for example
julia --project -e 'using MPIPreferences; MPIPreferences.use_jll_binary("MPItrampoline_jll")'
If you omit the JLL binary name, the default is selected for the respective operating system.
Configuration of the MPI.jl testsuite
Testing against a different MPI implementation
The LocalPreferences.toml
must be located within the test
folder, you can either create it in place or copy it into place.
~/MPI> julia --project=test
julia> using MPIPreferences
julia> MPIPreferences.use_system_binary()
~/MPI> rm test/Manifest.toml
~/MPI> julia --project
(MPI) pkg> test
Environment variables
The test suite can also be modified by the following variables:
JULIA_MPI_TEST_NPROCS
: How many ranks to use within the testsJULIA_MPI_TEST_ARRAYTYPE
: Set toCuArray
orROCArray
to test the CUDA-aware interface withCUDA.CuArray
or the ROCm-aware interface withAMDGPU.ROCArray
or buffers.JULIA_MPI_TEST_BINARY
: Check that the specified MPI binary is used for the testsJULIA_MPI_TEST_ABI
: Check that the specified MPI ABI is used for the tests
Migration from MPI.jl v0.19 or earlier
For MPI.jl v0.20, environment variables were used to configure which MPI library to use. These have been removed and no longer have any effect. The following subsections explain how to the same effects can be achieved with v0.20 or later.
Please refer to Notes to HPC cluster administrators if you want to migrate your MPI.jl preferences on a cluster with a centrally managed MPI.jl configuration.
JULIA_MPI_BINARY
Use MPIPreferences.use_system_binary
to use a system-provided MPI binary as described here. To switch back or select a different JLL-provided MPI binary, use MPIPreferences.use_jll_binary
as described here.
JULIA_MPI_PATH
Removed without replacement.
JULIA_MPI_LIBRARY
Use MPIPreferences.use_system_binary
with keyword argument library_names
to specify possible, non-standard library names. Alternatively, you can also specify the full path to the library.
JULIA_MPI_ABI
Use MPIPreferences.use_system_binary
with keyword argument abi
to specify which ABI to use. See MPIPreferences.abi
for possible values.
JULIA_MPIEXEC
Use MPIPreferences.use_system_binary
with keyword argument mpiexec
to specify the MPI launcher executable.
JULIA_MPIEXEC_ARGS
Use MPIPreferences.use_system_binary
with keyword argument mpiexec
, and pass a Cmd
object to set the MPI launcher executable and to include specific command line options.
JULIA_MPI_INCLUDE_PATH
Removed without replacement. Automatic generation of a constants file for unknown MPI ABIs is not supported anymore. See also #574.
JULIA_MPI_CFLAGS
Removed without replacement. Automatic generation of a constants file for unknown MPI ABIs is not supported anymore. See also #574.
JULIA_MPICC
Removed without replacement. Automatic generation of a constants file for unknown MPI ABIs is not supported anymore. See also #574.