Skip to content
deisa-dask v0.7.1 is here! 🎉 Discover what’s new
deisa-dask v0.6.0 : Supporting Per-Array MPI Communicators

deisa-dask v0.6.0 : Supporting Per-Array MPI Communicators

July 6, 2026·bmartin
bmartin

We’re excited to announce the latest update to deisa-dask. This release introduces a significant improvement to the deisa runtime: support for non-distributed arrays, making it easier to combine local and distributed data within the same in situ workflow.

Alongside this new capability, we’ve also launched a dedicated documentation website with a step-by-step getting started guide, API reference, and project documentation.

Supporting Arbitrary MPI Communicators

Previous versions of deisa-dask assumed that every distributed array was partitioned across all MPI ranks participating in the simulation. While this matches many HPC applications, it is not always the most natural way to organize data.

In practice, simulations often produce data with different communication patterns. Some arrays are distributed over the entire application, while others exist only on a subset of processes. For example, data local to a compute group, metadata generated by a dedicated set of ranks, or arrays replicated on a single process.

This release removes that limitation by allowing arrays to be associated with arbitrary MPI communicators. An array can now be:

  • distributed across all MPI ranks,
  • distributed across a subset of MPI ranks,
  • or completely local to a single rank.

From the analysis point of view, these arrays are exposed through the same deisa-dask API and can be consumed transparently by Dask workflows. This provides much greater flexibility when coupling simulations and analytics while avoiding unnecessary communication or data redistribution.

Supporting More HPC Workflows

Supporting arbitrary communicators allows simulations to expose data using the decomposition that best matches the underlying algorithm, rather than forcing every array into a global distribution.

For example:

  • large simulation fields can remain distributed across the entire MPI application,
  • domain-specific data structures can be distributed only within dedicated process groups,
  • small configuration tables or diagnostics can remain local to a single rank.

By preserving the natural ownership of each dataset, deisa-dask simplifies simulation instrumentation, reduces unnecessary communication, and enables more expressive in situ workflows.

Implementation

Adding support for local arrays required a substantial refactoring of the communication layer inside deisa-dask.

The bridge responsible for transferring simulation data to Dask now distinguishes between distributed and local objects and selects the appropriate communication strategy for each. Metadata handling has also been updated so that the analysis side can reconstruct arrays correctly regardless of how they were produced.

Because this change affects the core runtime, extensive testing was added to validate the new behavior across multiple execution configurations. New tests cover:

  • distributed and non-distributed arrays,
  • per-array communicators,
  • MPI execution,
  • fake communicators used during development,
  • multiple worker configurations.

These additions significantly strengthen the reliability of the deisa runtime.

A New Documentation Website

This release also introduces a brand new documentation website.

The new site includes:

  • a Getting Started tutorial,
  • installation instructions,
  • API documentation generated directly from the source,
  • project overview,
  • release notes,
  • examples demonstrating deisa-dask in action.

Our goal is to make it easier for new users to go from installation to their first in situ analysis in just a few minutes while providing comprehensive documentation for advanced users.

Looking Forward

This release continues our effort to make deisa-dask a flexible and easy-to-use platform for in situ analytics on HPC systems.

Supporting both distributed and local arrays removes an important limitation of previous releases and brings the programming model closer to the way scientific applications naturally organize their data.

As always, feedback, bug reports, and contributions are welcome. We hope these improvements make it even easier to integrate Dask analytics into your simulation workflows.

Full Changelog: https://github.com/deisa-project/deisa-dask/compare/v0.5.1...v0.6.0

Last updated on