deisa-dask v0.7.0 : Bringing Computation Closer to the Simulation
We are pleased to announce deisa-dask 0.7.0, an important release introducing new capabilities for in-situ analytics in HPC simulations.
This release focuses on an important challenge in simulation-analytics coupling: moving large amounts of simulation data to an analytics infrastructure can become expensive, even when the analysis itself is relatively simple. Many operations, such as computing sums, means, or variances, do not actually require the complete dataset to leave the simulation processes.
With version 0.7.0, deisa-dask takes a step towards bringing computation closer to where the data is produced. It introduces automatic bridge-side precomputation, a new mergeable PCA implementation, and an improved benchmarking infrastructure to track performance as the project evolves.
Bridge-side precomputation: moving computation closer to the data
Until now, deisa-dask primarily relied on transferring simulation data from MPI processes to Dask workers, where registered callbacks performed the analysis.
This approach offers considerable flexibility: users can express their analyses using familiar Dask Array operations, without having to manage the underlying MPI communication.
However, transferring complete data chunks is not always necessary. Consider a simple global reduction:
@deisa.register("temperature")
def callback(window):
temperature = window[-1]
total = temperature.sum().compute()To compute this sum, each MPI process only needs to compute a local sum over its own data. The analytics side then combines these partial results into the global value.
Transferring the complete temperature field to the Dask cluster just to compute a scalar is wasteful.
Automatic local reduction
Version 0.7.0 introduces a mechanism that identifies supported reduction operations in registered callbacks and moves their execution to the bridge side.
During callback registration, deisa-dask analyzes the callback and inspects its Dask task graphs to identify operations that can be executed independently on each simulation chunk.
Currently supported operations include:
summeanvarstdminmaxprod
Instead of transferring the original chunks, each bridge executes the corresponding operation on its local NumPy array and sends only the resulting partial value to the analytics side.
The partial results are then combined to reconstruct the global reduction result.
The resulting data movement looks like this:
Simulation (MPI ranks)
Rank 0 Rank 1 Rank 2
| | |
v v v
Local sum Local sum Local sum
| | |
v v v
Partial Partial Partial
| | |
+------------+------------+
|
v
Dask analytics side
|
v
Global reduction
|
v
User callbackThis changes the amount of data transferred from the size of the original simulation field to the size of its reduction results.
For large simulation fields, the difference can be substantial.
Keeping the familiar Dask interface
An important design objective was to introduce this optimization without requiring users to rewrite their analysis code.
The callback still expresses reductions using the familiar Dask Array API. The precomputation mechanism analyzes the callback at registration time and arranges for the corresponding operations to be executed on the bridges.
This is particularly interesting because the optimization is transparent for supported callback patterns: the user continues to work with the same high-level operations while deisa-dask changes where the computation takes place.
Explicit control over precomputation
Not every analysis can be reduced to independent operations on local chunks.
For example, plotting, arbitrary transformations, and algorithms that require access to the complete array still need the original data to reach the Dask workers.
For these cases, callbacks can explicitly disable precomputation:
@deisa.register("temperature", precompute=False)
def callback(window):
temperature = window[-1]
# Full array is available on the Dask side.
result = temperature.compute()The precompute parameter defaults to True. When set to False, deisa-dask skips callback analysis and uses the
traditional full-chunk scatter mechanism.
The new mechanism also validates callback compatibility during registration. Unsupported patterns, such as raw data access, cross-array expressions, or reductions depending on other reductions, are rejected rather than silently producing incorrect results.
This makes the optimization explicit and predictable while preserving the flexibility of the original execution model.
Mergeable PCA: distributed dimensionality reduction through compact summaries
The second major addition in version 0.7.0 is a new module: deisa.dask.mergeable_pca.
Principal Component Analysis (PCA) is a common technique in scientific computing, particularly for exploring large simulation datasets, identifying dominant structures, and reducing the dimensionality of physical fields.
However, conventional distributed PCA implementations often require substantial data movement or intermediate communication between workers.
deisa-dask introduces a different approach based on mergeable PCA summaries.
From local PCA to global PCA
The central idea is that each simulation rank can independently compute a compact summary of its local data.
A summary contains:
- The number of samples.
- The local mean.
- The principal components.
- The associated singular values.
Two summaries can then be merged to obtain the summary corresponding to the union of their datasets, without transferring the original samples to the merge operation.
The implementation uses the classical covariance merge formulation of Chan, Golub, and LeVeque (1979), together with a merge-and-truncate tree strategy.
Conceptually:
Simulation data blocks
Block 0 Block 1 Block 2 Block 3
| | | |
v v v v
Local PCA Local PCA Local PCA Local PCA
| | | |
v v v v
Summary 0 Summary 1 Summary 2 Summary 3
| | | |
+------+-----+ +------+-----+
| |
v v
Merge PCA Merge PCA
| |
+------------+------------+
|
v
Merge PCA
|
v
Global PCA summary
|
v
Final componentsOnly compact summaries need to travel between merge operations. The original samples remain at their source.
With full local rank, the resulting PCA is mathematically equivalent to batch PCA, up to the usual numerical precision and sign ambiguity of singular vectors.
Controlling the communication-accuracy trade-off
The implementation also introduces a local_rank parameter.
By default, local summaries retain their full rank, allowing the merge to reproduce the exact batch result.
However, users can limit the rank retained by each local summary:
from deisa.dask import MergeablePCA
pca = MergeablePCA(
n_components=10,
local_rank=20,
).fit(data)Reducing local_rank decreases the size of the summaries exchanged during merging. The trade-off is that information
discarded during local decomposition cannot be recovered later, potentially introducing differences in the final
principal subspace.
This parameter therefore provides a way to explore the trade-off between communication volume and approximation accuracy.
The implementation includes tests that verify the expected behavior across different local ranks, including the exact full-rank case.
Supporting multidimensional simulation data
Scientific simulation data is not always naturally represented as a two-dimensional matrix.
For example, a gyrokinetic simulation may produce a field distributed over several spatial and velocity dimensions. Applying PCA requires deciding which dimensions represent samples and which represent features.
MergeablePCA provides an axis policy to handle this situation.
Users can specify axis names and explicitly identify feature and sample axes:
pca = MergeablePCA(
n_components=5,
axis_names=("species", "tor1", "tor2", "tor3", "vpar", "mu"),
feature_axes=("vpar", "mu"),
).fit(data)In this example, the velocity dimensions form the feature space, while the remaining dimensions are flattened into samples.
The implementation can also infer the default layout from the Dask chunk structure: axes that are split across chunks are treated as sample axes, while axes left whole are treated as feature axes.
Importantly, feature dimensions must remain complete within each Dask chunk. If a feature axis is split across chunks, the estimator raises an informative error rather than silently changing the mathematical problem.
A real Dask task graph
The implementation is built around Dask and exposes the computation as a genuine task graph.
The _fit_dask_delayed() method constructs the PCA reduction tree without executing it. Users can inspect and visualize
the graph before triggering computation.

The task graph illustrates how MergeablePCA distributes the computation across 16 bridges. Each bridge first computes
a local PCA summary from its own data block, producing 16 independent local_pca tasks. These summaries are then
combined through a balanced binary reduction tree, with merge_pca tasks progressively merging pairs of summaries until
a single global PCA summary remains. Crucially, the merge operations only exchange compact summaries rather than the
original data, keeping the raw samples local to each bridge. The tree contains 15 merge operations and has a logarithmic
depth, allowing the global result to be assembled efficiently as the number of bridges grows.
This is useful for understanding the execution structure and evaluating how the reduction scales with the number of input blocks.
The estimator also provides a scikit-learn-style interface, including fit, fit_transform, transform, and
inverse_transform.
PCA integrated with bridge-side precomputation
Perhaps the most interesting aspect is that mergeable PCA is also integrated into the new bridge-side precomputation mechanism.
A callback can request PCA directly on a registered array:
from deisa.dask import MergeablePCA
@deisa.register("temperature")
def callback(window):
temperature = window[-1]
pca = MergeablePCA(
n_components=5,
local_rank=20,
).fit(temperature)deisa-dask recognizes this estimator operation during callback analysis.
Each bridge performs a local PCA on its own NumPy chunk and sends a mergeable summary rather than the original data. The analytics side then merges these summaries and constructs the final PCA result.
An important detail is that the bridge does not truncate the summary to n_components. The complete local summary
required for merging is preserved, and the requested number of components is applied only after the global merge.
This integration illustrates the broader potential of bridge-side computation: it is not limited to simple scalar reductions, but can also accommodate more sophisticated distributed algorithms when their intermediate results can be represented by compact mergeable summaries.
Performance tracking with Bencher
Optimizing data movement and callback execution is only useful if performance improvements can be measured and regressions detected.
Version 0.7.0 introduces continuous performance benchmarking using Bencher.
The benchmark infrastructure now tracks the time between Bridge.send() and the beginning of the corresponding
analytics callback.
Rather than measuring only individual communication operations, this metric captures the end-to-end latency experienced by the application.
The benchmark suite exercises different numbers of MPI bridges and records latency statistics, including the median and higher percentiles.
Two complementary workflows have been introduced:
- Benchmarks run on the main branch to establish and update reference performance.
- Pull requests are benchmarked against their target branch to identify potential performance regressions.
Benchmark results are uploaded to Bencher and associated with the corresponding GitHub commits and pull requests. Configured thresholds can flag regressions automatically.
This provides a foundation for making performance a continuous part of the development process rather than something evaluated only during dedicated experiments.
A more complete project documentation
This release also includes a significant improvement to the project’s documentation and developer experience.
A new README introduces deisa-dask, explains its architecture, and provides a complete getting-started example.
The example illustrates how to connect an MPI simulation to a Dask analytics process, send simulation fields through a
Bridge, and register callbacks using Deisa.
A new CONTRIBUTING.md document also describes the repository structure, development environment, coding conventions,
testing procedures, and pull-request workflow.
These additions should make it easier for new users and contributors to understand the project and start experimenting with it.
Other improvements
Several additional changes accompany the main features:
- MPI dependency:
mpi4pyis now a required dependency rather than an optional installation extra. - Bridge robustness: asynchronous scatter operations now handle calls from threads that already have a running event loop, avoiding teardown errors.
- Callback lifecycle: fixes address cases where initial callbacks could be missed because callback execution was not properly synchronized.
- Object cleanup: exceptions raised during
Deisadestruction are handled more safely. - Input compatibility:
Bridge.send()now accepts zero-dimensional NumPy arrays and one-dimensional arrays of size one as timestep arguments. This avoids errors when then timestep argument is not explicitly cast toint. - Testing: the test suite has been extended to cover bridge-side precomputation, PCA merging, MPI integration, and worker memory consumption.
Looking ahead
deisa-dask 0.7.0 represents an important evolution of the project.
The introduction of bridge-side precomputation changes the way simulation data can be consumed by analytics applications. Instead of always transferring complete fields to Dask workers, computation can now take place directly on the MPI ranks that own the data.
Mergeable PCA demonstrates how this principle can be extended beyond simple reductions, enabling more advanced analytics while keeping communication focused on compact intermediate results.
Together, these developments move deisa-dask towards a more flexible model for in-situ analytics, where the placement of computation can adapt to the structure of the analysis and the distribution of simulation data.
We look forward to exploring further applications of this approach and continuing to improve the scalability and performance of deisa-dask.
Full Changelog: https://github.com/deisa-project/deisa-dask/compare/v0.6.3...v0.7.0