deisa-dask v0.6.1 : Fixing Supporting for Per-Array MPI Communicators
Supporting Arrays That Don’t Live Everywhere
When deisa-dask was first designed, it assumed that every distributed array involved the same set of MPI processes. That is true for many simulation codes, where each rank owns a piece of every field. Real applications, however, are often more heterogeneous.
Some variables may only exist on a subset of MPI ranks. Others may be generated by a single process, or only by a particular computational component. Treating every array as if it were distributed across the entire communicator unnecessarily restricts these use cases.
The latest update removes this limitation by allowing each array to have its own communication group.
The problem
Consider an MPI application running on four ranks.
temperature
ββββββββ¬βββββββ¬βββββββ¬βββββββ
β R0 β R1 β R2 β R3 β
ββββββββ΄βββββββ΄βββββββ΄βββββββ
density
ββββββββ¬βββββββ¬βββββββ¬βββββββ
β β R1 β β β
ββββββββ΄βββββββ΄βββββββ΄βββββββThe temperature field is distributed over every rank, while density only exists on rank 1.
Previous versions of deisa-dask assumed that every rank participated in every array. This assumption caused problems when using MPI, because communicator management and collective operations still expected all ranks to behave identically.
Per-array communicators
The bridge now creates a dedicated MPI sub-communicator for every array. Ranks that own part of an array join its communicator, while all other ranks are excluded. Conceptually, communication now looks like this:
World communicator
βββ temperature communicator
β βββ Rank 0
β βββ Rank 1
β βββ Rank 2
β βββ Rank 3
β
βββ density communicator
βββ Rank 1This has two important consequences.
First, collective MPI operations are only performed by the ranks that actually participate in an array.
Second, metadata exchanges, such as synchronizing the list of available Dask workers are now broadcast within the array’s communicator instead of the global communicator. This prevents unrelated MPI ranks from participating in communications they do not need.
Better compatibility with MPI
Implementing this correctly required handling MPI_Comm_split in a portable way.
deisa-dask now uses the MPI constants provided by mpi4py whenever they are available, while still providing a fallback
implementation for environments where mpi4py is not installed.
This makes communicator creation behave correctly across different MPI implementations while preserving compatibility with DEISA’s testing infrastructure.
Improved testing
To ensure this behavior remains reliable, the MPI test suite has been extended with a new scenario where not every rank contributes to every array.
In the new test:
- every MPI rank publishes the
temperaturefield, - only one rank publishes the
densityfield, - callback execution verifies that the distributed
temperaturearray is reconstructed correctly.
This reproduces the kind of heterogeneous data distribution commonly found in multiphysics simulations.
The MPI test infrastructure itself also received a few improvements:
pytest.inihas been moved to the repository root, making pytest configuration more consistent.- The GitHub Actions workflow now installs
mpi4pyand Open MPI together throughpip, ensuring that both packages are built against the same MPI implementation. This avoids version mismatches that can occur when mixing system MPI packages with Python wheels.
Conclusion
While this change is largely invisible from the API, it makes deisa-dask considerably more flexible.
Applications are no longer required to artificially distribute every array across every MPI process simply to satisfy the communication layer. Instead, each array naturally involves only the ranks that actually own data.
This opens the door to more heterogeneous simulation workflows, component-based applications, adaptive meshes, and producer/consumer patterns where different variables are generated by different subsets of processes.
From the user’s perspective, nothing changes: arrays are still sent with Bridge.send() and received as Dask arrays.
Under the hood, however, the bridge now constructs the appropriate communication topology automatically for each array.
Full Changelog: https://github.com/deisa-project/deisa-dask/compare/v0.6.0...v0.6.1