deisa-dask v0.6.2 : Fixing Per-Array MPI Communicators Creation Order
Fixing Array Sub-Communicator Creation in deisa-dask
One of the design goals of deisa-dask is to make distributed data exchange transparent, even when different MPI ranks publish different arrays. Internally, this requires the bridge to create a dedicated MPI sub-communicator for every global array.
A small assumption in that setup process recently turned into a subtle bug. This update fixes it.
The Problem
When the bridge starts, each MPI rank advertises the arrays it owns. The bridge then gathers this information to build the global set of array names before creating one MPI sub-communicator per array.
Conceptually, the process looks like this:
- Every rank reports its local array names.
- The names are merged into a global set.
- Every rank iterates over the global list.
- A sub-communicator is created for each array.
The problem was hidden in step 2.
The global collection of array names was built using a Python set. While sets are perfect for removing duplicates,
their iteration order is not guaranteed.
After broadcasting this set to every rank, each process could end up iterating over the arrays in a different order.
Since MPI_Comm_split (or its equivalent) is a collective operation, every process must call it in exactly the same
sequence. If rank 0 creates communicators in the order:
temperature
pressure
velocitywhile another rank iterates as:
pressure
velocity
temperaturethe communicator creation sequence no longer matches across processes. This could lead to incorrect communicator assignments or deadlocks during bridge initialization.
The Fix
The solution is intentionally simple.
Instead of storing the broadcasted array names directly, the bridge now sorts them before using them:
self._global_array_names = sorted(global_names)By converting the unordered collection into a deterministic list, every MPI rank now performs communicator creation in exactly the same order.
This guarantees that _setup_array_comms() always executes the same sequence of collective operations on every
participating process.
Importance of the Fix
This bug only appeared in situations where:
- different MPI ranks exposed different subsets of arrays,
- multiple array communicators needed to be created,
- the iteration order of the Python set differed between processes.
Because Python intentionally makes hash ordering unpredictable across processes, relying on a set’s iteration order is unsafe whenever the order controls collective MPI operations.
Sorting the array names removes this source of nondeterminism entirely.
A Small Change with a Big Impact
The actual code change consists of a single line:
- self._global_array_names = global_names
+ self._global_array_names = sorted(global_names)
Sometimes the smallest fixes are the most important. Distributed systems often depend on every participant executing the exact same sequence of collective operations, and even seemingly harmless nondeterminism can break those assumptions.
With this change, communicator creation is now deterministic and reliable regardless of how arrays are distributed across MPI ranks.
Full Changelog: https://github.com/deisa-project/deisa-dask/compare/v0.6.1...v0.6.2