deisa-dask v0.5.1
deisa-dask v0.5.1 is a patch release that introduces the following changes:
- In CI, unit tests now run in parallel using pytest-xdist
- When sending data using the Bridge, the chunk position is now read from
arrays_metadata codemeta.jsonis added to the repository root to be referenced in the NumPEx software cataloguesetup.pyis replaced bypyproject.toml- Bugfix: data could go missing (garbage collected by the scheduler) in certain cases
It also includes contributions from 1 new contributors and addresses long-standing community requests.
Bugfix
We’re happy to announce the release of deisa-dask 0.5.1, a maintenance release that fixes a critical issue affecting workflows that expose multiple arrays from a simulation callback.
The Problem
deisa-dask allows simulations to expose distributed data directly to Dask, enabling in situ analytics without writing intermediate files.
In some configurations, however, users could observe missing or incomplete data when multiple arrays were transferred during the same call to Bridge.send().
The root cause was related to the lifecycle management of Dask objects. Under certain circumstances, arrays created by deisa-dask could be garbage collected by the Dask scheduler before the analytics workflow had a chance to consume them. This issue was more likely to appear when multiple arrays were published concurrently and when the scheduler aggressively reclaimed unused objects.
The Fix
Version 0.5.1 addresses this issue by explicitly persisting generated Dask arrays using client.persist().
This ensures that arrays remain alive on the Dask cluster until they are consumed by the analysis workflow, preventing premature garbage collection and eliminating the possibility of missing data.
In addition to the runtime fix, the release strengthens the test suite with:
- Parametric tests covering different numbers of Dask workers.
- Improved handling of intermittent test failures.
- Additional exception handling in tests.
- More robust synchronization logic for worker-dependent scenarios.
Impact on Users
If your workflow transfers a single array per callback, you may never have encountered this issue. However, users exposing multiple arrays simultaneously, such as coupled analytics pipelines, multi-field simulations, or machine-learning workflows, should upgrade to 0.5.1.
The upgrade requires no API changes and is fully backward compatible with deisa-dask 0.5.0.
Additional Improvements
This release also includes several infrastructure and maintenance updates:
- Improved test execution through parallelized testing.
- Better handling of chunk position metadata.
- Migration from setup.py to a modern pyproject.toml build system.
- Addition of project metadata through codemeta.json.
Upgrading
Upgrade using pip
pip install --upgrade deisa-daskUpgrade using Spack
deisa-dask is packaged in the official spack-packages repository. The latest release is found on the develop branch.
spack repo update # update the repo that is tracking the develop branch
spack versions py-deisa-dask # check that version 0.5.1 is found
spack install py-deisa-daskUpgrade using Guix
deisa-dask is also packaged for Guix in the guix-science channel. Your Guix setup must be configured to use the guix-science channel.
guix pull # update guix and it's channelsWe recommend all users running deisa-dask 0.5.0 to upgrade to 0.5.1 to benefit from the improved reliability of data transfers and stronger test coverage.
Contributors
This release was made possible by contributions from 1 new contributors:
- @abhinav-jay -
codemeta.jsonand this wonderful website
Full Changelog: https://github.com/deisa-project/deisa-dask/compare/v0.5.0...v0.5.1