Skip to content
deisa-dask v0.7.1 is here! 🎉 Discover what’s new
deisa-dask v0.7.1 : Bringing Computation Closer to the Simulation

deisa-dask v0.7.1 : Bringing Computation Closer to the Simulation

November 18, 2026·bmartin
bmartin

Introduction

deisa-dask connects MPI-based simulations to the Dask distributed runtime, allowing analysis callbacks to operate on simulation data without requiring the application to manage the Dask execution model.

This integration relies on reconstructing global Dask arrays from data distributed across MPI ranks. That reconstruction depends on knowing where each rank’s local chunk belongs in the global array.

Recent changes address several issues in this part of deisa-dask: a collective communication deadlock during bridge initialization, incorrect reconstruction of uneven chunk distributions, and the need to provide chunk positions explicitly even when the MPI communicator already contains enough information to derive them.

Fixing collective communication during bridge initialization

deisa-dask can prefetch the task branches associated with registered analysis callbacks during Bridge initialization. This keeps branch discovery out of the simulation’s send() path.

The prefetch operation performs one broadcast per array on that array’s communicator. Consequently, every rank participating in an array must enter the same sequence of broadcasts.

Previously, only world rank 0 performed this prefetch. Other ranks would retrieve their task branches lazily when calling send(). This could appear to work when the simulation handled a single shared array: the ranks would eventually meet in the same collective.

With multiple shared arrays, however, the behavior could deadlock. Rank 0 prefetched arrays in sorted order, while other ranks entered the collectives according to the order in which they sent their arrays. If those orders differed, ranks could wait indefinitely on different array communicators.

The prefetch now runs on every rank, after the post-registration barrier. All participating ranks therefore enter the same sorted sequence of collectives before the simulation starts sending data.

A regression test reproduces the problematic scenario with two MPI ranks sharing two arrays and sending them in opposite orders. It checks that initialization completes, that every rank has cached the registered task branches, and that the resulting reductions are correct.

Supporting uneven MPI chunk distributions

Another limitation concerned the reconstruction of global arrays from local MPI chunks.

Previously, the analytics side assumed that all chunks had the same shape. It inferred the number of tiles along each dimension by dividing the global array shape by a single chunk shape. This works for uniform decompositions, but fails when ranks own chunks of different sizes.

For example, consider a one-dimensional array of three elements distributed over two ranks:

  • Rank 0 owns one element.
  • Rank 1 owns two elements.

The global shape is (3,), and the local chunk shapes are (1,) and (2,).

The previous implementation used the root rank’s chunk shape for every gathered piece. The analytics side consequently received two pieces both described as having shape (1,), and could not reconstruct the global array correctly.

Carrying each chunk’s own metadata

The bridge now includes each sender’s chunk shape and dtype alongside its future and chunk position in the gathered event.

This ensures that the analytics side has the information needed to reconstruct each piece independently, rather than relying on the root rank’s local metadata.

Reconstructing arrays from chunk positions

The full-chunk reconstruction path has been rewritten to use the metadata carried by each piece.

Instead of inferring a uniform grid from global_shape and one common chunk shape, the tiler builds the layout from the actual chunk positions and shapes:

  1. Each piece is associated with its own chunk_position.
  2. Its shape and dtype are used to create the corresponding delayed Dask block.
  3. The block is placed at its declared position in the global grid.
  4. The blocks are assembled with da.block, which supports uneven block dimensions.

The resulting Dask array preserves the original chunk sizes. For the example above, its chunks are ((1, 2),) rather than an incorrectly inferred uniform layout.

The implementation also checks for inconsistent metadata. Duplicate positions, missing positions, incompatible block shapes, and mismatches between the covered dimensions and global_shape now raise errors instead of silently producing an incorrect array.

The same approach applies to multidimensional arrays. Tests cover uneven distributions in one, two, and three dimensions, as well as pieces arriving in an order different from their positions in the global array.

Deriving chunk positions from MPI topology

Supporting uneven chunks requires reliable information about where each chunk belongs. A chunk’s position cannot always be inferred from its shape or from the global array dimensions.

deisa-dask now allows chunk_position to be omitted when registering an array with a Bridge. In that case, the bridge attempts to derive the position once, during initialization, when the communicator topology is available.

The derivation follows this precedence:

  1. Explicit position: if the application provides chunk_position, that value remains authoritative.
  2. Cartesian communicator: if the communicator has Cartesian topology, the bridge uses the rank’s Cartesian coordinates.
  3. Uniform grid: otherwise, the bridge derives the rank’s row-major tile index from the uniform grid implied by global_shape and chunk_shape.

For example, with a one-dimensional array of shape (4,), distributed over two ranks with chunks of shape (2,), the bridge can derive positions (0,) and (1,) without requiring the application to provide them.

Cartesian topology is particularly useful for multidimensional decompositions. For example, a two-dimensional array distributed over a Cartesian communicator can use the communicator’s coordinates directly as its chunk positions.

There is an important distinction between deriving a position and guessing one. The uniform-grid fallback is only valid when the global dimensions are evenly divisible by the chunk dimensions and the resulting number of tiles matches the number of participating ranks.

If these conditions are not met, the bridge raises a ValueError asking the application to provide an explicit chunk_position. Uneven distributions therefore remain supported, but their positions must be declared by the application.

This also makes the responsibilities of the two sides clearer: the bridge resolves positions from the application and its communicator topology, while the analytics side consumes those positions to reconstruct the global array. The latter no longer guesses a layout from a common chunk shape.

Testing the changes with MPI

These changes affect collective operations and data distribution, so unit tests alone are not sufficient.

The test suite now includes both simulated communicator tests and real MPI integration tests.

The new scenarios cover:

  • Multiple shared arrays whose send order differs between ranks, reproducing the initialization deadlock.
  • Uneven chunk distributions, including one-dimensional arrays split into chunks of different sizes.
  • Reconstruction of two- and three-dimensional arrays with uneven chunk dimensions.
  • Automatic position derivation from a regular uniform grid.
  • Position derivation from a Cartesian communicator, including a three-rank, non-square topology.
  • Invalid or ambiguous layouts, which must fail with an explicit error rather than being silently reconstructed.

The uneven-distribution integration tests exercise the full-chunk path with precompute=False, ensuring that the reconstruction mechanism itself is tested. Other tests inspect the gathered event metadata directly to verify that each piece carries its own shape, dtype, and position.

Together, these tests check not only that the individual operations work, but also that the metadata exchanged between the simulation and analytics sides remains consistent.

Looking ahead

These changes make deisa-dask more robust when connecting MPI applications to Dask.

The bridge now handles collective branch prefetching consistently across ranks, and the analytics side can reconstruct global arrays without assuming a uniform chunk distribution. At the same time, applications using regular decompositions can rely on their MPI communicator topology to provide chunk positions automatically.

The guiding principle is to make the data layout explicit wherever it matters, while deriving information from the communicator when that derivation is unambiguous. When the layout cannot be established reliably, deisa-dask reports the problem rather than silently making an assumption.

For applications with uneven domain decompositions, this means that local chunks can retain their actual shapes throughout the transfer and reconstruction process, while still being exposed as a global Dask array for analysis.

Full Changelog: https://github.com/deisa-project/deisa-dask/compare/v0.7.0...v0.7.1

Last updated on