<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>deisa-dask – Blog</title><link>https://deisa-project.github.io/deisa-dask/blog/</link><description>Recent content in Blog on deisa-dask</description><generator>Hugo -- gohugo.io</generator><language>en</language><atom:link href="https://deisa-project.github.io/deisa-dask/blog/index.xml" rel="self" type="application/rss+xml"/><item><title>deisa-dask v0.7.1 : Bringing Computation Closer to the Simulation</title><link>https://deisa-project.github.io/deisa-dask/blog/0.7.1/</link><pubDate>Wed, 18 Nov 2026 00:00:00 +0000</pubDate><guid>https://deisa-project.github.io/deisa-dask/blog/0.7.1/</guid><description>
&lt;h2&gt;Introduction&lt;span class="hx:absolute hx:-mt-20" id="introduction"&gt;&lt;/span&gt;
&lt;a href="#introduction" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;deisa-dask connects MPI-based simulations to the Dask distributed runtime, allowing analysis callbacks to operate on
simulation data without requiring the application to manage the Dask execution model.&lt;/p&gt;
&lt;p&gt;This integration relies on reconstructing global Dask arrays from data distributed across MPI ranks. That reconstruction
depends on knowing where each rank&amp;rsquo;s local chunk belongs in the global array.&lt;/p&gt;
&lt;p&gt;Recent changes address several issues in this part of deisa-dask: a collective communication deadlock during bridge
initialization, incorrect reconstruction of uneven chunk distributions, and the need to provide chunk positions
explicitly even when the MPI communicator already contains enough information to derive them.&lt;/p&gt;
&lt;h2&gt;Fixing collective communication during bridge initialization&lt;span class="hx:absolute hx:-mt-20" id="fixing-collective-communication-during-bridge-initialization"&gt;&lt;/span&gt;
&lt;a href="#fixing-collective-communication-during-bridge-initialization" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;deisa-dask can prefetch the task branches associated with registered analysis callbacks during &lt;code&gt;Bridge&lt;/code&gt; initialization.
This keeps branch discovery out of the simulation&amp;rsquo;s &lt;code&gt;send()&lt;/code&gt; path.&lt;/p&gt;
&lt;p&gt;The prefetch operation performs one broadcast per array on that array&amp;rsquo;s communicator. Consequently, &lt;strong&gt;every rank
participating in an array must enter the same sequence of broadcasts&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Previously, only world rank 0 performed this prefetch. Other ranks would retrieve their task branches lazily when
calling &lt;code&gt;send()&lt;/code&gt;. This could appear to work when the simulation handled a single shared array: the ranks would
eventually meet in the same collective.&lt;/p&gt;
&lt;p&gt;With multiple shared arrays, however, the behavior could deadlock. Rank 0 prefetched arrays in sorted order, while other
ranks entered the collectives according to the order in which they sent their arrays. If those orders differed, ranks
could wait indefinitely on different array communicators.&lt;/p&gt;
&lt;p&gt;The prefetch now runs on every rank, after the post-registration barrier. All participating ranks therefore enter the
same sorted sequence of collectives before the simulation starts sending data.&lt;/p&gt;
&lt;p&gt;A regression test reproduces the problematic scenario with two MPI ranks sharing two arrays and sending them in opposite
orders. It checks that initialization completes, that every rank has cached the registered task branches, and that the
resulting reductions are correct.&lt;/p&gt;
&lt;h2&gt;Supporting uneven MPI chunk distributions&lt;span class="hx:absolute hx:-mt-20" id="supporting-uneven-mpi-chunk-distributions"&gt;&lt;/span&gt;
&lt;a href="#supporting-uneven-mpi-chunk-distributions" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Another limitation concerned the reconstruction of global arrays from local MPI chunks.&lt;/p&gt;
&lt;p&gt;Previously, the analytics side assumed that all chunks had the same shape. It inferred the number of tiles along each
dimension by dividing the global array shape by a single chunk shape. This works for uniform decompositions, but fails
when ranks own chunks of different sizes.&lt;/p&gt;
&lt;p&gt;For example, consider a one-dimensional array of three elements distributed over two ranks:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Rank 0 owns one element.&lt;/li&gt;
&lt;li&gt;Rank 1 owns two elements.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The global shape is &lt;code&gt;(3,)&lt;/code&gt;, and the local chunk shapes are &lt;code&gt;(1,)&lt;/code&gt; and &lt;code&gt;(2,)&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The previous implementation used the root rank&amp;rsquo;s chunk shape for every gathered piece. The analytics side consequently
received two pieces both described as having shape &lt;code&gt;(1,)&lt;/code&gt;, and could not reconstruct the global array correctly.&lt;/p&gt;
&lt;h3&gt;Carrying each chunk&amp;rsquo;s own metadata&lt;span class="hx:absolute hx:-mt-20" id="carrying-each-chunks-own-metadata"&gt;&lt;/span&gt;
&lt;a href="#carrying-each-chunks-own-metadata" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;The bridge now includes each sender&amp;rsquo;s chunk shape and dtype alongside its future and chunk position in the gathered
event.&lt;/p&gt;
&lt;p&gt;This ensures that the analytics side has the information needed to reconstruct each piece independently, rather than
relying on the root rank&amp;rsquo;s local metadata.&lt;/p&gt;
&lt;h3&gt;Reconstructing arrays from chunk positions&lt;span class="hx:absolute hx:-mt-20" id="reconstructing-arrays-from-chunk-positions"&gt;&lt;/span&gt;
&lt;a href="#reconstructing-arrays-from-chunk-positions" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;The full-chunk reconstruction path has been rewritten to use the metadata carried by each piece.&lt;/p&gt;
&lt;p&gt;Instead of inferring a uniform grid from &lt;code&gt;global_shape&lt;/code&gt; and one common chunk shape, the tiler builds the layout from the
actual chunk positions and shapes:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Each piece is associated with its own &lt;code&gt;chunk_position&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Its shape and dtype are used to create the corresponding delayed Dask block.&lt;/li&gt;
&lt;li&gt;The block is placed at its declared position in the global grid.&lt;/li&gt;
&lt;li&gt;The blocks are assembled with &lt;code&gt;da.block&lt;/code&gt;, which supports uneven block dimensions.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The resulting Dask array preserves the original chunk sizes. For the example above, its chunks are &lt;code&gt;((1, 2),)&lt;/code&gt; rather
than an incorrectly inferred uniform layout.&lt;/p&gt;
&lt;p&gt;The implementation also checks for inconsistent metadata. Duplicate positions, missing positions, incompatible block
shapes, and mismatches between the covered dimensions and &lt;code&gt;global_shape&lt;/code&gt; now raise errors instead of silently producing
an incorrect array.&lt;/p&gt;
&lt;p&gt;The same approach applies to multidimensional arrays. Tests cover uneven distributions in one, two, and three
dimensions, as well as pieces arriving in an order different from their positions in the global array.&lt;/p&gt;
&lt;h2&gt;Deriving chunk positions from MPI topology&lt;span class="hx:absolute hx:-mt-20" id="deriving-chunk-positions-from-mpi-topology"&gt;&lt;/span&gt;
&lt;a href="#deriving-chunk-positions-from-mpi-topology" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Supporting uneven chunks requires reliable information about where each chunk belongs. A chunk&amp;rsquo;s position cannot always
be inferred from its shape or from the global array dimensions.&lt;/p&gt;
&lt;p&gt;deisa-dask now allows &lt;code&gt;chunk_position&lt;/code&gt; to be omitted when registering an array with a &lt;code&gt;Bridge&lt;/code&gt;. In that case, the bridge
attempts to derive the position once, during initialization, when the communicator topology is available.&lt;/p&gt;
&lt;p&gt;The derivation follows this precedence:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Explicit position:&lt;/strong&gt; if the application provides &lt;code&gt;chunk_position&lt;/code&gt;, that value remains authoritative.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Cartesian communicator:&lt;/strong&gt; if the communicator has Cartesian topology, the bridge uses the rank&amp;rsquo;s Cartesian
coordinates.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Uniform grid:&lt;/strong&gt; otherwise, the bridge derives the rank&amp;rsquo;s row-major tile index from the uniform grid implied by
&lt;code&gt;global_shape&lt;/code&gt; and &lt;code&gt;chunk_shape&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;For example, with a one-dimensional array of shape &lt;code&gt;(4,)&lt;/code&gt;, distributed over two ranks with chunks of shape &lt;code&gt;(2,)&lt;/code&gt;, the
bridge can derive positions &lt;code&gt;(0,)&lt;/code&gt; and &lt;code&gt;(1,)&lt;/code&gt; without requiring the application to provide them.&lt;/p&gt;
&lt;p&gt;Cartesian topology is particularly useful for multidimensional decompositions. For example, a two-dimensional array
distributed over a Cartesian communicator can use the communicator&amp;rsquo;s coordinates directly as its chunk positions.&lt;/p&gt;
&lt;p&gt;There is an important distinction between deriving a position and guessing one. The uniform-grid fallback is only valid
when the global dimensions are evenly divisible by the chunk dimensions and the resulting number of tiles matches the
number of participating ranks.&lt;/p&gt;
&lt;p&gt;If these conditions are not met, the bridge raises a &lt;code&gt;ValueError&lt;/code&gt; asking the application to provide an explicit
&lt;code&gt;chunk_position&lt;/code&gt;. Uneven distributions therefore remain supported, but their positions must be declared by the
application.&lt;/p&gt;
&lt;p&gt;This also makes the responsibilities of the two sides clearer: the bridge resolves positions from the application and
its communicator topology, while the analytics side consumes those positions to reconstruct the global array. The latter
no longer guesses a layout from a common chunk shape.&lt;/p&gt;
&lt;h2&gt;Testing the changes with MPI&lt;span class="hx:absolute hx:-mt-20" id="testing-the-changes-with-mpi"&gt;&lt;/span&gt;
&lt;a href="#testing-the-changes-with-mpi" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;These changes affect collective operations and data distribution, so unit tests alone are not sufficient.&lt;/p&gt;
&lt;p&gt;The test suite now includes both simulated communicator tests and real MPI integration tests.&lt;/p&gt;
&lt;p&gt;The new scenarios cover:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Multiple shared arrays whose send order differs between ranks, reproducing the initialization deadlock.&lt;/li&gt;
&lt;li&gt;Uneven chunk distributions, including one-dimensional arrays split into chunks of different sizes.&lt;/li&gt;
&lt;li&gt;Reconstruction of two- and three-dimensional arrays with uneven chunk dimensions.&lt;/li&gt;
&lt;li&gt;Automatic position derivation from a regular uniform grid.&lt;/li&gt;
&lt;li&gt;Position derivation from a Cartesian communicator, including a three-rank, non-square topology.&lt;/li&gt;
&lt;li&gt;Invalid or ambiguous layouts, which must fail with an explicit error rather than being silently reconstructed.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The uneven-distribution integration tests exercise the full-chunk path with &lt;code&gt;precompute=False&lt;/code&gt;, ensuring that the
reconstruction mechanism itself is tested. Other tests inspect the gathered event metadata directly to verify that each
piece carries its own shape, dtype, and position.&lt;/p&gt;
&lt;p&gt;Together, these tests check not only that the individual operations work, but also that the metadata exchanged between
the simulation and analytics sides remains consistent.&lt;/p&gt;
&lt;h2&gt;Looking ahead&lt;span class="hx:absolute hx:-mt-20" id="looking-ahead"&gt;&lt;/span&gt;
&lt;a href="#looking-ahead" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;These changes make deisa-dask more robust when connecting MPI applications to Dask.&lt;/p&gt;
&lt;p&gt;The bridge now handles collective branch prefetching consistently across ranks, and the analytics side can reconstruct
global arrays without assuming a uniform chunk distribution. At the same time, applications using regular decompositions
can rely on their MPI communicator topology to provide chunk positions automatically.&lt;/p&gt;
&lt;p&gt;The guiding principle is to make the data layout explicit wherever it matters, while deriving information from the
communicator when that derivation is unambiguous. When the layout cannot be established reliably, deisa-dask reports the
problem rather than silently making an assumption.&lt;/p&gt;
&lt;p&gt;For applications with uneven domain decompositions, this means that local chunks can retain their actual shapes
throughout the transfer and reconstruction process, while still being exposed as a global Dask array for analysis.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Full Changelog&lt;/strong&gt;: &lt;a
href="https://github.com/deisa-project/deisa-dask/compare/v0.7.0...v0.7.1"target="_blank" rel="noopener"&gt;https://github.com/deisa-project/deisa-dask/compare/v0.7.0...v0.7.1&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt;&lt;/p&gt;</description></item><item><title>deisa-dask v0.7.0 : Bringing Computation Closer to the Simulation</title><link>https://deisa-project.github.io/deisa-dask/blog/0.7.0/</link><pubDate>Fri, 06 Nov 2026 00:00:00 +0000</pubDate><guid>https://deisa-project.github.io/deisa-dask/blog/0.7.0/</guid><description>
&lt;p&gt;We are pleased to announce &lt;strong&gt;deisa-dask 0.7.0&lt;/strong&gt;, an important release introducing new capabilities for in-situ analytics
in HPC simulations.&lt;/p&gt;
&lt;p&gt;This release focuses on an important challenge in simulation-analytics coupling: moving large amounts of simulation data
to an analytics infrastructure can become expensive, even when the analysis itself is relatively simple. Many
operations, such as computing sums, means, or variances, do not actually require the complete dataset to leave the
simulation processes.&lt;/p&gt;
&lt;p&gt;With version 0.7.0, deisa-dask takes a step towards bringing computation closer to where the data is produced. It
introduces automatic bridge-side precomputation, a new mergeable PCA implementation, and an improved benchmarking
infrastructure to track performance as the project evolves.&lt;/p&gt;
&lt;h2&gt;Bridge-side precomputation: moving computation closer to the data&lt;span class="hx:absolute hx:-mt-20" id="bridge-side-precomputation-moving-computation-closer-to-the-data"&gt;&lt;/span&gt;
&lt;a href="#bridge-side-precomputation-moving-computation-closer-to-the-data" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Until now, deisa-dask primarily relied on transferring simulation data from MPI processes to Dask workers, where
registered callbacks performed the analysis.&lt;/p&gt;
&lt;p&gt;This approach offers considerable flexibility: users can express their analyses using familiar Dask Array operations,
without having to manage the underlying MPI communication.&lt;/p&gt;
&lt;p&gt;However, transferring complete data chunks is not always necessary. Consider a simple global reduction:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@deisa.register&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;temperature&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;callback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;temperature&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;To compute this sum, each MPI process only needs to compute a local sum over its own data. The analytics side then
combines these partial results into the global value.&lt;/p&gt;
&lt;p&gt;Transferring the complete temperature field to the Dask cluster just to compute a scalar is wasteful.&lt;/p&gt;
&lt;h3&gt;Automatic local reduction&lt;span class="hx:absolute hx:-mt-20" id="automatic-local-reduction"&gt;&lt;/span&gt;
&lt;a href="#automatic-local-reduction" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;Version 0.7.0 introduces a mechanism that identifies supported reduction operations in registered callbacks and moves
their execution to the bridge side.&lt;/p&gt;
&lt;p&gt;During callback registration, deisa-dask analyzes the callback and inspects its Dask task graphs to identify operations
that can be executed independently on each simulation chunk.&lt;/p&gt;
&lt;p&gt;Currently supported operations include:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;sum&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;mean&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;var&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;std&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;min&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;max&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;prod&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Instead of transferring the original chunks, each bridge executes the corresponding operation on its local NumPy array
and sends only the resulting partial value to the analytics side.&lt;/p&gt;
&lt;p&gt;The partial results are then combined to reconstruct the global reduction result.&lt;/p&gt;
&lt;p&gt;The resulting data movement looks like this:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Simulation (MPI ranks)
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Rank 0 Rank 1 Rank 2
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | | |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v v v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Local sum Local sum Local sum
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | | |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v v v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Partial Partial Partial
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | | |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; +------------+------------+
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Dask analytics side
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Global reduction
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; User callback&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;This changes the amount of data transferred from the size of the original simulation field to the size of its reduction
results.&lt;/p&gt;
&lt;p&gt;For large simulation fields, the difference can be substantial.&lt;/p&gt;
&lt;h3&gt;Keeping the familiar Dask interface&lt;span class="hx:absolute hx:-mt-20" id="keeping-the-familiar-dask-interface"&gt;&lt;/span&gt;
&lt;a href="#keeping-the-familiar-dask-interface" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;An important design objective was to introduce this optimization without requiring users to rewrite their analysis code.&lt;/p&gt;
&lt;p&gt;The callback still expresses reductions using the familiar Dask Array API. The precomputation mechanism analyzes the
callback at registration time and arranges for the corresponding operations to be executed on the bridges.&lt;/p&gt;
&lt;p&gt;This is particularly interesting because the optimization is &lt;strong&gt;transparent&lt;/strong&gt; for supported callback patterns: the user
continues to work with the same high-level operations while deisa-dask changes where the computation takes place.&lt;/p&gt;
&lt;h3&gt;Explicit control over precomputation&lt;span class="hx:absolute hx:-mt-20" id="explicit-control-over-precomputation"&gt;&lt;/span&gt;
&lt;a href="#explicit-control-over-precomputation" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;Not every analysis can be reduced to independent operations on local chunks.&lt;/p&gt;
&lt;p&gt;For example, plotting, arbitrary transformations, and algorithms that require access to the complete array still need
the original data to reach the Dask workers.&lt;/p&gt;
&lt;p&gt;For these cases, callbacks can explicitly disable precomputation:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@deisa.register&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;temperature&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;precompute&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;callback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;temperature&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Full array is available on the Dask side.&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;The &lt;code&gt;precompute&lt;/code&gt; parameter defaults to &lt;code&gt;True&lt;/code&gt;. When set to &lt;code&gt;False&lt;/code&gt;, deisa-dask skips callback analysis and uses the
traditional full-chunk scatter mechanism.&lt;/p&gt;
&lt;p&gt;The new mechanism also validates callback compatibility during registration. Unsupported patterns, such as raw data
access, cross-array expressions, or reductions depending on other reductions, are rejected rather than silently
producing incorrect results.&lt;/p&gt;
&lt;p&gt;This makes the optimization explicit and predictable while preserving the flexibility of the original execution model.&lt;/p&gt;
&lt;h2&gt;Mergeable PCA: distributed dimensionality reduction through compact summaries&lt;span class="hx:absolute hx:-mt-20" id="mergeable-pca-distributed-dimensionality-reduction-through-compact-summaries"&gt;&lt;/span&gt;
&lt;a href="#mergeable-pca-distributed-dimensionality-reduction-through-compact-summaries" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The second major addition in version 0.7.0 is a new module: &lt;code&gt;deisa.dask.mergeable_pca&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Principal Component Analysis (PCA) is a common technique in scientific computing, particularly for exploring large
simulation datasets, identifying dominant structures, and reducing the dimensionality of physical fields.&lt;/p&gt;
&lt;p&gt;However, conventional distributed PCA implementations often require substantial data movement or intermediate
communication between workers.&lt;/p&gt;
&lt;p&gt;deisa-dask introduces a different approach based on mergeable PCA summaries.&lt;/p&gt;
&lt;h3&gt;From local PCA to global PCA&lt;span class="hx:absolute hx:-mt-20" id="from-local-pca-to-global-pca"&gt;&lt;/span&gt;
&lt;a href="#from-local-pca-to-global-pca" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;The central idea is that each simulation rank can independently compute a compact summary of its local data.&lt;/p&gt;
&lt;p&gt;A summary contains:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The number of samples.&lt;/li&gt;
&lt;li&gt;The local mean.&lt;/li&gt;
&lt;li&gt;The principal components.&lt;/li&gt;
&lt;li&gt;The associated singular values.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Two summaries can then be merged to obtain the summary corresponding to the union of their datasets, without
transferring the original samples to the merge operation.&lt;/p&gt;
&lt;p&gt;The implementation uses the classical covariance merge formulation of Chan, Golub, and LeVeque (1979), together with a
merge-and-truncate tree strategy.&lt;/p&gt;
&lt;p&gt;Conceptually:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Simulation data blocks
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Block 0 Block 1 Block 2 Block 3
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | | | |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v v v v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Local PCA Local PCA Local PCA Local PCA
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | | | |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v v v v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Summary 0 Summary 1 Summary 2 Summary 3
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | | | |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; +------+-----+ +------+-----+
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Merge PCA Merge PCA
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; | |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; +------------+------------+
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Merge PCA
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Global PCA summary
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; |
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; v
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Final components&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Only compact summaries need to travel between merge operations. The original samples remain at their source.&lt;/p&gt;
&lt;p&gt;With full local rank, the resulting PCA is mathematically equivalent to batch PCA, up to the usual numerical precision
and sign ambiguity of singular vectors.&lt;/p&gt;
&lt;h3&gt;Controlling the communication-accuracy trade-off&lt;span class="hx:absolute hx:-mt-20" id="controlling-the-communication-accuracy-trade-off"&gt;&lt;/span&gt;
&lt;a href="#controlling-the-communication-accuracy-trade-off" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;The implementation also introduces a &lt;code&gt;local_rank&lt;/code&gt; parameter.&lt;/p&gt;
&lt;p&gt;By default, local summaries retain their full rank, allowing the merge to reproduce the exact batch result.&lt;/p&gt;
&lt;p&gt;However, users can limit the rank retained by each local summary:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;deisa.dask&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;MergeablePCA&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;pca&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;MergeablePCA&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;n_components&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;local_rank&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Reducing &lt;code&gt;local_rank&lt;/code&gt; decreases the size of the summaries exchanged during merging. The trade-off is that information
discarded during local decomposition cannot be recovered later, potentially introducing differences in the final
principal subspace.&lt;/p&gt;
&lt;p&gt;This parameter therefore provides a way to explore the trade-off between communication volume and approximation
accuracy.&lt;/p&gt;
&lt;p&gt;The implementation includes tests that verify the expected behavior across different local ranks, including the exact
full-rank case.&lt;/p&gt;
&lt;h3&gt;Supporting multidimensional simulation data&lt;span class="hx:absolute hx:-mt-20" id="supporting-multidimensional-simulation-data"&gt;&lt;/span&gt;
&lt;a href="#supporting-multidimensional-simulation-data" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;Scientific simulation data is not always naturally represented as a two-dimensional matrix.&lt;/p&gt;
&lt;p&gt;For example, a gyrokinetic simulation may produce a field distributed over several spatial and velocity dimensions.
Applying PCA requires deciding which dimensions represent samples and which represent features.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;MergeablePCA&lt;/code&gt; provides an axis policy to handle this situation.&lt;/p&gt;
&lt;p&gt;Users can specify axis names and explicitly identify feature and sample axes:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;pca&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;MergeablePCA&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;n_components&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;axis_names&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;species&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tor1&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tor2&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tor3&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;vpar&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;mu&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;feature_axes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;vpar&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;mu&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;In this example, the velocity dimensions form the feature space, while the remaining dimensions are flattened into
samples.&lt;/p&gt;
&lt;p&gt;The implementation can also infer the default layout from the Dask chunk structure: axes that are split across chunks
are treated as sample axes, while axes left whole are treated as feature axes.&lt;/p&gt;
&lt;p&gt;Importantly, feature dimensions must remain complete within each Dask chunk. If a feature axis is split across chunks,
the estimator raises an informative error rather than silently changing the mathematical problem.&lt;/p&gt;
&lt;h3&gt;A real Dask task graph&lt;span class="hx:absolute hx:-mt-20" id="a-real-dask-task-graph"&gt;&lt;/span&gt;
&lt;a href="#a-real-dask-task-graph" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;The implementation is built around Dask and exposes the computation as a genuine task graph.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;_fit_dask_delayed()&lt;/code&gt; method constructs the PCA reduction tree without executing it. Users can inspect and visualize
the graph before triggering computation.&lt;/p&gt;
&lt;p&gt;&lt;img src="https://deisa-project.github.io/deisa-dask/images/mergeable-pca.png" alt="MergeablePCA task graph" data-zoomable loading="lazy" /&gt;&lt;/p&gt;
&lt;p&gt;The task graph illustrates how &lt;code&gt;MergeablePCA&lt;/code&gt; distributes the computation across 16 bridges. Each bridge first computes
a local PCA summary from its own data block, producing 16 independent &lt;code&gt;local_pca&lt;/code&gt; tasks. These summaries are then
combined through a balanced binary reduction tree, with &lt;code&gt;merge_pca&lt;/code&gt; tasks progressively merging pairs of summaries until
a single global PCA summary remains. Crucially, the merge operations only exchange compact summaries rather than the
original data, keeping the raw samples local to each bridge. The tree contains 15 merge operations and has a logarithmic
depth, allowing the global result to be assembled efficiently as the number of bridges grows.&lt;/p&gt;
&lt;p&gt;This is useful for understanding the execution structure and evaluating how the reduction scales with the number of
input blocks.&lt;/p&gt;
&lt;p&gt;The estimator also provides a scikit-learn-style interface, including &lt;code&gt;fit&lt;/code&gt;, &lt;code&gt;fit_transform&lt;/code&gt;, &lt;code&gt;transform&lt;/code&gt;, and
&lt;code&gt;inverse_transform&lt;/code&gt;.&lt;/p&gt;
&lt;h3&gt;PCA integrated with bridge-side precomputation&lt;span class="hx:absolute hx:-mt-20" id="pca-integrated-with-bridge-side-precomputation"&gt;&lt;/span&gt;
&lt;a href="#pca-integrated-with-bridge-side-precomputation" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;Perhaps the most interesting aspect is that mergeable PCA is also integrated into the new bridge-side precomputation
mechanism.&lt;/p&gt;
&lt;p&gt;A callback can request PCA directly on a registered array:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;deisa.dask&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;MergeablePCA&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@deisa.register&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;temperature&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;callback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;temperature&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;pca&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;MergeablePCA&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;n_components&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;local_rank&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;deisa-dask recognizes this estimator operation during callback analysis.&lt;/p&gt;
&lt;p&gt;Each bridge performs a local PCA on its own NumPy chunk and sends a mergeable summary rather than the original data. The
analytics side then merges these summaries and constructs the final PCA result.&lt;/p&gt;
&lt;p&gt;An important detail is that the bridge does not truncate the summary to &lt;code&gt;n_components&lt;/code&gt;. The complete local summary
required for merging is preserved, and the requested number of components is applied only after the global merge.&lt;/p&gt;
&lt;p&gt;This integration illustrates the broader potential of bridge-side computation: it is not limited to simple scalar
reductions, but can also accommodate more sophisticated distributed algorithms when their intermediate results can be
represented by compact mergeable summaries.&lt;/p&gt;
&lt;h2&gt;Performance tracking with Bencher&lt;span class="hx:absolute hx:-mt-20" id="performance-tracking-with-bencher"&gt;&lt;/span&gt;
&lt;a href="#performance-tracking-with-bencher" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Optimizing data movement and callback execution is only useful if performance improvements can be measured and
regressions detected.&lt;/p&gt;
&lt;p&gt;Version 0.7.0 introduces continuous performance benchmarking using &lt;a
href="https://bencher.dev/"target="_blank" rel="noopener"&gt;Bencher&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The benchmark infrastructure now tracks the time between &lt;code&gt;Bridge.send()&lt;/code&gt; and the beginning of the corresponding
analytics callback.&lt;/p&gt;
&lt;p&gt;Rather than measuring only individual communication operations, this metric captures the end-to-end latency experienced
by the application.&lt;/p&gt;
&lt;p&gt;The benchmark suite exercises different numbers of MPI bridges and records latency statistics, including the median and
higher percentiles.&lt;/p&gt;
&lt;p&gt;Two complementary workflows have been introduced:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Benchmarks run on the main branch to establish and update reference performance.&lt;/li&gt;
&lt;li&gt;Pull requests are benchmarked against their target branch to identify potential performance regressions.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Benchmark results are uploaded to Bencher and associated with the corresponding GitHub commits and pull requests.
Configured thresholds can flag regressions automatically.&lt;/p&gt;
&lt;p&gt;This provides a foundation for making performance a continuous part of the development process rather than something
evaluated only during dedicated experiments.&lt;/p&gt;
&lt;h2&gt;A more complete project documentation&lt;span class="hx:absolute hx:-mt-20" id="a-more-complete-project-documentation"&gt;&lt;/span&gt;
&lt;a href="#a-more-complete-project-documentation" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This release also includes a significant improvement to the project&amp;rsquo;s documentation and developer experience.&lt;/p&gt;
&lt;p&gt;A new README introduces deisa-dask, explains its architecture, and provides a complete getting-started example.&lt;/p&gt;
&lt;p&gt;The example illustrates how to connect an MPI simulation to a Dask analytics process, send simulation fields through a
&lt;code&gt;Bridge&lt;/code&gt;, and register callbacks using &lt;code&gt;Deisa&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;A new &lt;code&gt;CONTRIBUTING.md&lt;/code&gt; document also describes the repository structure, development environment, coding conventions,
testing procedures, and pull-request workflow.&lt;/p&gt;
&lt;p&gt;These additions should make it easier for new users and contributors to understand the project and start experimenting
with it.&lt;/p&gt;
&lt;h2&gt;Other improvements&lt;span class="hx:absolute hx:-mt-20" id="other-improvements"&gt;&lt;/span&gt;
&lt;a href="#other-improvements" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Several additional changes accompany the main features:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;MPI dependency:&lt;/strong&gt; &lt;code&gt;mpi4py&lt;/code&gt; is now a required dependency rather than an optional installation extra.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Bridge robustness:&lt;/strong&gt; asynchronous scatter operations now handle calls from threads that already have a running event
loop, avoiding teardown errors.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Callback lifecycle:&lt;/strong&gt; fixes address cases where initial callbacks could be missed because callback execution was not
properly synchronized.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Object cleanup:&lt;/strong&gt; exceptions raised during &lt;code&gt;Deisa&lt;/code&gt; destruction are handled more safely.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Input compatibility:&lt;/strong&gt; &lt;code&gt;Bridge.send()&lt;/code&gt; now accepts zero-dimensional NumPy arrays and one-dimensional arrays of size
one as timestep arguments. This avoids errors when then timestep argument is not explicitly cast to &lt;code&gt;int&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Testing:&lt;/strong&gt; the test suite has been extended to cover bridge-side precomputation, PCA merging, MPI integration, and
worker memory consumption.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Looking ahead&lt;span class="hx:absolute hx:-mt-20" id="looking-ahead"&gt;&lt;/span&gt;
&lt;a href="#looking-ahead" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;deisa-dask 0.7.0 represents an important evolution of the project.&lt;/p&gt;
&lt;p&gt;The introduction of bridge-side precomputation changes the way simulation data can be consumed by analytics
applications. Instead of always transferring complete fields to Dask workers, computation can now take place directly on
the MPI ranks that own the data.&lt;/p&gt;
&lt;p&gt;Mergeable PCA demonstrates how this principle can be extended beyond simple reductions, enabling more advanced analytics
while keeping communication focused on compact intermediate results.&lt;/p&gt;
&lt;p&gt;Together, these developments move deisa-dask towards a more flexible model for in-situ analytics, where the placement of
computation can adapt to the structure of the analysis and the distribution of simulation data.&lt;/p&gt;
&lt;p&gt;We look forward to exploring further applications of this approach and continuing to improve the scalability and
performance of deisa-dask.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Full Changelog&lt;/strong&gt;: &lt;a
href="https://github.com/deisa-project/deisa-dask/compare/v0.6.3...v0.7.0"target="_blank" rel="noopener"&gt;https://github.com/deisa-project/deisa-dask/compare/v0.6.3...v0.7.0&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt;&lt;/p&gt;</description></item><item><title>deisa-dask v0.6.3 : Fixing Per-Array MPI Communicators Creation Order</title><link>https://deisa-project.github.io/deisa-dask/blog/0.6.3/</link><pubDate>Wed, 29 Jul 2026 00:00:00 +0000</pubDate><guid>https://deisa-project.github.io/deisa-dask/blog/0.6.3/</guid><description>
&lt;h1&gt;deisa-dask 0.6.3: Improved Reliability, Better MPI Support, and New Performance Benchmarks&lt;/h1&gt;&lt;p&gt;We&amp;rsquo;re pleased to announce &lt;strong&gt;deisa-dask 0.6.3&lt;/strong&gt;, a release focused on improving robustness and developer experience.
While this release does not introduce major API changes, it significantly strengthens the runtime by improving callback
execution, fixing several concurrency issues, expanding MPI compatibility, and introducing new benchmarking tools.&lt;/p&gt;
&lt;h2&gt;More reliable callback execution&lt;span class="hx:absolute hx:-mt-20" id="more-reliable-callback-execution"&gt;&lt;/span&gt;
&lt;a href="#more-reliable-callback-execution" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;One of the primary goals of this release was to make callback execution more predictable.&lt;/p&gt;
&lt;h3&gt;&lt;code&gt;execute_callbacks()&lt;/code&gt; now waits for completion&lt;span class="hx:absolute hx:-mt-20" id="execute_callbacks-now-waits-for-completion"&gt;&lt;/span&gt;
&lt;a href="#execute_callbacks-now-waits-for-completion" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;In previous versions, &lt;code&gt;execute_callbacks()&lt;/code&gt; could return before all registered callbacks had finished executing. This
made it difficult to reason about application state after callback execution, especially when callbacks performed
asynchronous work.&lt;/p&gt;
&lt;p&gt;With 0.6.3, &lt;code&gt;execute_callbacks()&lt;/code&gt; now blocks until every callback has completed. This provides a much clearer execution
model: once the function returns, all callbacks associated with the current data have finished.&lt;/p&gt;
&lt;p&gt;For users, this means fewer synchronization concerns and more deterministic behavior when integrating deisa-dask into
larger workflows.&lt;/p&gt;
&lt;h3&gt;Detecting invalid timestep ordering&lt;span class="hx:absolute hx:-mt-20" id="detecting-invalid-timestep-ordering"&gt;&lt;/span&gt;
&lt;a href="#detecting-invalid-timestep-ordering" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;Callbacks now verify that incoming iterations are &lt;strong&gt;monotonically increasing&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;If an older timestep is accidentally delivered after a newer one, deisa-dask raises an exception instead of silently
accepting inconsistent data.&lt;/p&gt;
&lt;p&gt;This additional validation helps detect application bugs early and prevents sliding windows from entering inconsistent
states.&lt;/p&gt;
&lt;h2&gt;Improved internal robustness&lt;span class="hx:absolute hx:-mt-20" id="improved-internal-robustness"&gt;&lt;/span&gt;
&lt;a href="#improved-internal-robustness" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Several internal synchronization issues have been resolved in this release.&lt;/p&gt;
&lt;p&gt;A concurrency issue affecting the internal window management has been fixed, making callback execution more reliable
under concurrent workloads. In addition, callback metadata now consistently reports the correct iteration associated
with each received array, eliminating cases where iteration information could become inconsistent.&lt;/p&gt;
&lt;p&gt;Although these improvements are largely invisible to users, they increase the reliability of long-running simulations
and streaming workflows.&lt;/p&gt;
&lt;h2&gt;Better MPI compatibility&lt;span class="hx:absolute hx:-mt-20" id="better-mpi-compatibility"&gt;&lt;/span&gt;
&lt;a href="#better-mpi-compatibility" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;deisa-dask now continuously tests its MPI backend against &lt;strong&gt;both OpenMPI and MPICH&lt;/strong&gt; in continuous integration.&lt;/p&gt;
&lt;p&gt;Supporting multiple MPI implementations is important because behavior can differ subtly between libraries. Running the
test suite against both implementations helps ensure that new features remain portable across HPC systems without
requiring users to modify their applications.&lt;/p&gt;
&lt;h2&gt;Measuring callback latency&lt;span class="hx:absolute hx:-mt-20" id="measuring-callback-latency"&gt;&lt;/span&gt;
&lt;a href="#measuring-callback-latency" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This release introduces a new benchmark that measures one of the most important performance metrics for in-situ
analysis: &lt;strong&gt;the time between a simulation sending data and a callback beginning execution&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Rather than measuring only communication throughput, this benchmark captures the end-to-end latency experienced by
applications:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Bridge.send()
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; │
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ▼
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Communication
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; │
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ▼
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Data reconstruction
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; │
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ▼
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Callback execution begins&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;This provides a more representative view of the responsiveness of the complete deisa-dask pipeline and establishes a
foundation for tracking latency improvements in future releases.&lt;/p&gt;
&lt;p&gt;Additional benchmark scripts have also been added for studying client connection overhead and communication latency
under different configurations.&lt;/p&gt;
&lt;h2&gt;A cleaner development workflow&lt;span class="hx:absolute hx:-mt-20" id="a-cleaner-development-workflow"&gt;&lt;/span&gt;
&lt;a href="#a-cleaner-development-workflow" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This release also improves the development experience.&lt;/p&gt;
&lt;p&gt;Code formatting and style are now standardized using &lt;strong&gt;Ruff&lt;/strong&gt;, with automated formatting and linting integrated into the
continuous integration pipeline. This helps keep the codebase consistent while catching common issues before they reach
the main branch.&lt;/p&gt;
&lt;h2&gt;Looking ahead&lt;span class="hx:absolute hx:-mt-20" id="looking-ahead"&gt;&lt;/span&gt;
&lt;a href="#looking-ahead" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Although 0.6.3 is primarily a maintenance release, it strengthens many of the core components of deisa-dask:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;callback execution is now deterministic,&lt;/li&gt;
&lt;li&gt;iteration ordering is validated automatically,&lt;/li&gt;
&lt;li&gt;concurrency issues have been resolved,&lt;/li&gt;
&lt;li&gt;MPI compatibility is tested across multiple implementations,&lt;/li&gt;
&lt;li&gt;and new benchmarking tools make it easier to evaluate end-to-end performance.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Together, these improvements make deisa-dask a more reliable platform for developing in-situ analytics workflows while
providing a stronger foundation for future performance and scalability improvements.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Full Changelog&lt;/strong&gt;: &lt;a
href="https://github.com/deisa-project/deisa-dask/compare/v0.6.2...v0.6.3"target="_blank" rel="noopener"&gt;https://github.com/deisa-project/deisa-dask/compare/v0.6.2...v0.6.3&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt;&lt;/p&gt;</description></item><item><title>deisa-dask v0.6.2 : Fixing Per-Array MPI Communicators Creation Order</title><link>https://deisa-project.github.io/deisa-dask/blog/0.6.2/</link><pubDate>Thu, 09 Jul 2026 00:00:00 +0000</pubDate><guid>https://deisa-project.github.io/deisa-dask/blog/0.6.2/</guid><description>
&lt;h1&gt;Fixing Array Sub-Communicator Creation in deisa-dask&lt;/h1&gt;&lt;p&gt;One of the design goals of deisa-dask is to make distributed data exchange transparent, even when different MPI
ranks publish different arrays.
Internally, this requires the bridge to create a dedicated MPI sub-communicator for every global array.&lt;/p&gt;
&lt;p&gt;A small assumption in that setup process recently turned into a subtle bug. This update fixes it.&lt;/p&gt;
&lt;h2&gt;The Problem&lt;span class="hx:absolute hx:-mt-20" id="the-problem"&gt;&lt;/span&gt;
&lt;a href="#the-problem" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;When the bridge starts, each MPI rank advertises the arrays it owns. The bridge then gathers this information to build
the global set of array names before creating one MPI sub-communicator per array.&lt;/p&gt;
&lt;p&gt;Conceptually, the process looks like this:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Every rank reports its local array names.&lt;/li&gt;
&lt;li&gt;The names are merged into a global set.&lt;/li&gt;
&lt;li&gt;Every rank iterates over the global list.&lt;/li&gt;
&lt;li&gt;A sub-communicator is created for each array.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The problem was hidden in step 2.&lt;/p&gt;
&lt;p&gt;The global collection of array names was built using a Python &lt;code&gt;set&lt;/code&gt;. While sets are perfect for removing duplicates,
&lt;strong&gt;their iteration order is not guaranteed&lt;/strong&gt;.
After broadcasting this set to every rank, each process could end up iterating over the arrays in a different order.&lt;/p&gt;
&lt;p&gt;Since &lt;code&gt;MPI_Comm_split&lt;/code&gt; (or its equivalent) is a collective operation, &lt;strong&gt;every process must call it in exactly the same
sequence&lt;/strong&gt;. If rank 0 creates communicators in the order:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;temperature
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;pressure
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;velocity&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;while another rank iterates as:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;pressure
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;velocity
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;temperature&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;the communicator creation sequence no longer matches across processes.
This could lead to incorrect communicator assignments or deadlocks during bridge initialization.&lt;/p&gt;
&lt;h2&gt;The Fix&lt;span class="hx:absolute hx:-mt-20" id="the-fix"&gt;&lt;/span&gt;
&lt;a href="#the-fix" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The solution is intentionally simple.&lt;/p&gt;
&lt;p&gt;Instead of storing the broadcasted array names directly, the bridge now sorts them before using them:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_global_array_names&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;global_names&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;By converting the unordered collection into a deterministic list, every MPI rank now performs communicator creation in
exactly the same order.&lt;/p&gt;
&lt;p&gt;This guarantees that &lt;code&gt;_setup_array_comms()&lt;/code&gt; always executes the same sequence of collective operations on every
participating process.&lt;/p&gt;
&lt;h2&gt;Importance of the Fix&lt;span class="hx:absolute hx:-mt-20" id="importance-of-the-fix"&gt;&lt;/span&gt;
&lt;a href="#importance-of-the-fix" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This bug only appeared in situations where:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;different MPI ranks exposed different subsets of arrays,&lt;/li&gt;
&lt;li&gt;multiple array communicators needed to be created,&lt;/li&gt;
&lt;li&gt;the iteration order of the Python set differed between processes.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Because Python intentionally makes hash ordering unpredictable across processes, relying on a set&amp;rsquo;s iteration order is
unsafe whenever the order controls collective MPI operations.&lt;/p&gt;
&lt;p&gt;Sorting the array names removes this source of nondeterminism entirely.&lt;/p&gt;
&lt;h2&gt;A Small Change with a Big Impact&lt;span class="hx:absolute hx:-mt-20" id="a-small-change-with-a-big-impact"&gt;&lt;/span&gt;
&lt;a href="#a-small-change-with-a-big-impact" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The actual code change consists of a single line:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-diff" data-lang="diff"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="gd"&gt;- self._global_array_names = global_names
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="gi"&gt;+ self._global_array_names = sorted(global_names)
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Sometimes the smallest fixes are the most important. Distributed systems often depend on every participant executing the
exact same sequence of collective operations, and even seemingly harmless nondeterminism can break those assumptions.&lt;/p&gt;
&lt;p&gt;With this change, communicator creation is now deterministic and reliable regardless of how arrays are distributed
across MPI ranks.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Full Changelog&lt;/strong&gt;: &lt;a
href="https://github.com/deisa-project/deisa-dask/compare/v0.6.1...v0.6.2"target="_blank" rel="noopener"&gt;https://github.com/deisa-project/deisa-dask/compare/v0.6.1...v0.6.2&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt;&lt;/p&gt;</description></item><item><title>deisa-dask v0.6.1 : Fixing Supporting for Per-Array MPI Communicators</title><link>https://deisa-project.github.io/deisa-dask/blog/0.6.1/</link><pubDate>Wed, 08 Jul 2026 00:00:00 +0000</pubDate><guid>https://deisa-project.github.io/deisa-dask/blog/0.6.1/</guid><description>
&lt;h1&gt;Supporting Arrays That Don&amp;rsquo;t Live Everywhere&lt;/h1&gt;&lt;p&gt;When deisa-dask was first designed, it assumed that every distributed array involved the same set of MPI processes.
That is true for many simulation codes, where each rank owns a piece of every field.
Real applications, however, are often more heterogeneous.&lt;/p&gt;
&lt;p&gt;Some variables may only exist on a subset of MPI ranks.
Others may be generated by a single process, or only by a particular computational component.
Treating every array as if it were distributed across the entire communicator unnecessarily restricts these use cases.&lt;/p&gt;
&lt;p&gt;The latest update removes this limitation by allowing each array to have its own communication group.&lt;/p&gt;
&lt;h2&gt;The problem&lt;span class="hx:absolute hx:-mt-20" id="the-problem"&gt;&lt;/span&gt;
&lt;a href="#the-problem" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Consider an MPI application running on four ranks.&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;pre&gt;&lt;code&gt;temperature
┌──────┬──────┬──────┬──────┐
│ R0 │ R1 │ R2 │ R3 │
└──────┴──────┴──────┴──────┘
density
┌──────┬──────┬──────┬──────┐
│ │ R1 │ │ │
└──────┴──────┴──────┴──────┘&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;The &lt;code&gt;temperature&lt;/code&gt; field is distributed over every rank, while &lt;code&gt;density&lt;/code&gt; only exists on rank 1.&lt;/p&gt;
&lt;p&gt;Previous versions of deisa-dask assumed that every rank participated in every array.
This assumption caused problems when using MPI, because communicator management and collective operations still
expected all ranks to behave identically.&lt;/p&gt;
&lt;h2&gt;Per-array communicators&lt;span class="hx:absolute hx:-mt-20" id="per-array-communicators"&gt;&lt;/span&gt;
&lt;a href="#per-array-communicators" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The bridge now creates a dedicated MPI sub-communicator for every array.
Ranks that own part of an array join its communicator, while all other ranks are excluded.
Conceptually, communication now looks like this:&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;pre&gt;&lt;code&gt;World communicator
├── temperature communicator
│ ├── Rank 0
│ ├── Rank 1
│ ├── Rank 2
│ └── Rank 3
│
└── density communicator
└── Rank 1&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;This has two important consequences.&lt;/p&gt;
&lt;p&gt;First, collective MPI operations are only performed by the ranks that actually participate in an array.&lt;/p&gt;
&lt;p&gt;Second, metadata exchanges, such as synchronizing the list of available Dask workers are now broadcast within the
array&amp;rsquo;s communicator instead of the global communicator.
This prevents unrelated MPI ranks from participating in communications they do not need.&lt;/p&gt;
&lt;h2&gt;Better compatibility with MPI&lt;span class="hx:absolute hx:-mt-20" id="better-compatibility-with-mpi"&gt;&lt;/span&gt;
&lt;a href="#better-compatibility-with-mpi" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Implementing this correctly required handling &lt;code&gt;MPI_Comm_split&lt;/code&gt; in a portable way.
deisa-dask now uses the MPI constants provided by &lt;code&gt;mpi4py&lt;/code&gt; whenever they are available, while still providing a fallback
implementation for environments where &lt;code&gt;mpi4py&lt;/code&gt; is not installed.&lt;/p&gt;
&lt;p&gt;This makes communicator creation behave correctly across different MPI implementations while preserving compatibility
with DEISA&amp;rsquo;s testing infrastructure.&lt;/p&gt;
&lt;h2&gt;Improved testing&lt;span class="hx:absolute hx:-mt-20" id="improved-testing"&gt;&lt;/span&gt;
&lt;a href="#improved-testing" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;To ensure this behavior remains reliable, the MPI test suite has been extended with a new scenario where not every rank
contributes to every array.&lt;/p&gt;
&lt;p&gt;In the new test:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;every MPI rank publishes the &lt;code&gt;temperature&lt;/code&gt; field,&lt;/li&gt;
&lt;li&gt;only one rank publishes the &lt;code&gt;density&lt;/code&gt; field,&lt;/li&gt;
&lt;li&gt;callback execution verifies that the distributed &lt;code&gt;temperature&lt;/code&gt; array is reconstructed correctly.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This reproduces the kind of heterogeneous data distribution commonly found in multiphysics simulations.&lt;/p&gt;
&lt;p&gt;The MPI test infrastructure itself also received a few improvements:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;pytest.ini&lt;/code&gt; has been moved to the repository root, making pytest configuration more consistent.&lt;/li&gt;
&lt;li&gt;The GitHub Actions workflow now installs &lt;code&gt;mpi4py&lt;/code&gt; and Open MPI together through &lt;code&gt;pip&lt;/code&gt;, ensuring that both packages are
built against the same MPI implementation. This avoids version mismatches that can occur when mixing system MPI
packages with Python wheels.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Conclusion&lt;span class="hx:absolute hx:-mt-20" id="conclusion"&gt;&lt;/span&gt;
&lt;a href="#conclusion" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;While this change is largely invisible from the API, it makes deisa-dask considerably more flexible.&lt;/p&gt;
&lt;p&gt;Applications are no longer required to artificially distribute every array across every MPI process simply to satisfy
the communication layer.
Instead, each array naturally involves only the ranks that actually own data.&lt;/p&gt;
&lt;p&gt;This opens the door to more heterogeneous simulation workflows, component-based applications, adaptive meshes, and
producer/consumer patterns where different variables are generated by different subsets of processes.&lt;/p&gt;
&lt;p&gt;From the user&amp;rsquo;s perspective, nothing changes: arrays are still sent with &lt;code&gt;Bridge.send()&lt;/code&gt; and received as Dask arrays.
Under the hood, however, the bridge now constructs the appropriate communication topology automatically for each array.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Full Changelog&lt;/strong&gt;: &lt;a
href="https://github.com/deisa-project/deisa-dask/compare/v0.6.0...v0.6.1"target="_blank" rel="noopener"&gt;https://github.com/deisa-project/deisa-dask/compare/v0.6.0...v0.6.1&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt;&lt;/p&gt;</description></item><item><title>deisa-dask v0.6.0 : Supporting Per-Array MPI Communicators</title><link>https://deisa-project.github.io/deisa-dask/blog/0.6.0/</link><pubDate>Mon, 06 Jul 2026 00:00:00 +0000</pubDate><guid>https://deisa-project.github.io/deisa-dask/blog/0.6.0/</guid><description>
&lt;p&gt;We&amp;rsquo;re excited to announce the latest update to &lt;strong&gt;deisa-dask&lt;/strong&gt;.
This release introduces a significant improvement to the deisa runtime: &lt;strong&gt;support for non-distributed arrays&lt;/strong&gt;, making
it easier to combine local and distributed data within the same in situ workflow.&lt;/p&gt;
&lt;p&gt;Alongside this new capability, we&amp;rsquo;ve also launched a dedicated documentation website with a step-by-step getting started
guide, API reference, and project documentation.&lt;/p&gt;
&lt;h2&gt;Supporting Arbitrary MPI Communicators&lt;span class="hx:absolute hx:-mt-20" id="supporting-arbitrary-mpi-communicators"&gt;&lt;/span&gt;
&lt;a href="#supporting-arbitrary-mpi-communicators" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Previous versions of deisa-dask assumed that every distributed array was partitioned across &lt;strong&gt;all MPI ranks&lt;/strong&gt;
participating in the simulation. While this matches many HPC applications, it is not always the most natural way to
organize data.&lt;/p&gt;
&lt;p&gt;In practice, simulations often produce data with different communication patterns. Some arrays are distributed over the
entire application, while others exist only on a subset of processes. For example, data local to a compute group,
metadata generated by a dedicated set of ranks, or arrays replicated on a single process.&lt;/p&gt;
&lt;p&gt;This release removes that limitation by allowing arrays to be associated with &lt;strong&gt;arbitrary MPI communicators&lt;/strong&gt;. An array
can now be:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;distributed across all MPI ranks,&lt;/li&gt;
&lt;li&gt;distributed across a subset of MPI ranks,&lt;/li&gt;
&lt;li&gt;or completely local to a single rank.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;From the analysis point of view, these arrays are exposed through the same deisa-dask API and can be consumed
transparently by Dask workflows. This provides much greater flexibility when coupling simulations and analytics while
avoiding unnecessary communication or data redistribution.&lt;/p&gt;
&lt;h2&gt;Supporting More HPC Workflows&lt;span class="hx:absolute hx:-mt-20" id="supporting-more-hpc-workflows"&gt;&lt;/span&gt;
&lt;a href="#supporting-more-hpc-workflows" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Supporting arbitrary communicators allows simulations to expose data using the decomposition that best matches the
underlying algorithm, rather than forcing every array into a global distribution.&lt;/p&gt;
&lt;p&gt;For example:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;large simulation fields can remain distributed across the entire MPI application,&lt;/li&gt;
&lt;li&gt;domain-specific data structures can be distributed only within dedicated process groups,&lt;/li&gt;
&lt;li&gt;small configuration tables or diagnostics can remain local to a single rank.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;By preserving the natural ownership of each dataset, deisa-dask simplifies simulation instrumentation, reduces
unnecessary communication, and enables more expressive in situ workflows.&lt;/p&gt;
&lt;h2&gt;Implementation&lt;span class="hx:absolute hx:-mt-20" id="implementation"&gt;&lt;/span&gt;
&lt;a href="#implementation" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Adding support for local arrays required a substantial refactoring of the communication layer inside deisa-dask.&lt;/p&gt;
&lt;p&gt;The bridge responsible for transferring simulation data to Dask now distinguishes between distributed and local objects
and selects the appropriate communication strategy for each. Metadata handling has also been updated so that the
analysis side can reconstruct arrays correctly regardless of how they were produced.&lt;/p&gt;
&lt;p&gt;Because this change affects the core runtime, extensive testing was added to validate the new behavior across multiple
execution configurations. New tests cover:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;distributed and non-distributed arrays,&lt;/li&gt;
&lt;li&gt;per-array communicators,&lt;/li&gt;
&lt;li&gt;MPI execution,&lt;/li&gt;
&lt;li&gt;fake communicators used during development,&lt;/li&gt;
&lt;li&gt;multiple worker configurations.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These additions significantly strengthen the reliability of the deisa runtime.&lt;/p&gt;
&lt;h2&gt;A New Documentation Website&lt;span class="hx:absolute hx:-mt-20" id="a-new-documentation-website"&gt;&lt;/span&gt;
&lt;a href="#a-new-documentation-website" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This release also introduces a brand new documentation website.&lt;/p&gt;
&lt;p&gt;The new site includes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;a Getting Started tutorial,&lt;/li&gt;
&lt;li&gt;installation instructions,&lt;/li&gt;
&lt;li&gt;API documentation generated directly from the source,&lt;/li&gt;
&lt;li&gt;project overview,&lt;/li&gt;
&lt;li&gt;release notes,&lt;/li&gt;
&lt;li&gt;examples demonstrating deisa-dask in action.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Our goal is to make it easier for new users to go from installation to their first in situ analysis in just a few
minutes while providing comprehensive documentation for advanced users.&lt;/p&gt;
&lt;h2&gt;Looking Forward&lt;span class="hx:absolute hx:-mt-20" id="looking-forward"&gt;&lt;/span&gt;
&lt;a href="#looking-forward" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This release continues our effort to make deisa-dask a flexible and easy-to-use platform for in situ analytics on HPC
systems.&lt;/p&gt;
&lt;p&gt;Supporting both distributed and local arrays removes an important limitation of previous releases and brings the
programming model closer to the way scientific applications naturally organize their data.&lt;/p&gt;
&lt;p&gt;As always, feedback, bug reports, and contributions are welcome. We hope these improvements make it even easier to
integrate Dask analytics into your simulation workflows.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Full Changelog&lt;/strong&gt;: &lt;a
href="https://github.com/deisa-project/deisa-dask/compare/v0.5.1...v0.6.0"target="_blank" rel="noopener"&gt;https://github.com/deisa-project/deisa-dask/compare/v0.5.1...v0.6.0&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt;&lt;/p&gt;</description></item><item><title>deisa-dask v0.5.1</title><link>https://deisa-project.github.io/deisa-dask/blog/0.5.1/</link><pubDate>Mon, 22 Jun 2026 00:00:00 +0000</pubDate><guid>https://deisa-project.github.io/deisa-dask/blog/0.5.1/</guid><description>
&lt;p&gt;deisa-dask v0.5.1 is a patch release that introduces the following changes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;In CI, unit tests now run in parallel using pytest-xdist&lt;/li&gt;
&lt;li&gt;When sending data using the Bridge, the chunk position is now read from &lt;code&gt;arrays_metadata&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;codemeta.json&lt;/code&gt; is added to the repository root to be referenced in the NumPEx software catalogue&lt;/li&gt;
&lt;li&gt;&lt;code&gt;setup.py&lt;/code&gt; is replaced by &lt;code&gt;pyproject.toml&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Bugfix: data could go missing (garbage collected by the scheduler) in certain cases&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;It also includes contributions from 1 &lt;a
href="#contributors"&gt;new contributors&lt;/a&gt; and addresses long-standing community requests.&lt;/p&gt;
&lt;h2&gt;Bugfix&lt;span class="hx:absolute hx:-mt-20" id="bugfix"&gt;&lt;/span&gt;
&lt;a href="#bugfix" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;We&amp;rsquo;re happy to announce the release of deisa-dask 0.5.1, a maintenance release that fixes a critical issue affecting workflows that expose multiple arrays from a simulation callback.&lt;/p&gt;
&lt;h3&gt;The Problem&lt;span class="hx:absolute hx:-mt-20" id="the-problem"&gt;&lt;/span&gt;
&lt;a href="#the-problem" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;deisa-dask allows simulations to expose distributed data directly to Dask, enabling in situ analytics without writing intermediate files.
In some configurations, however, users could observe missing or incomplete data when multiple arrays were transferred during the same call to &lt;code&gt;Bridge.send()&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The root cause was related to the lifecycle management of Dask objects.
Under certain circumstances, arrays created by deisa-dask could be garbage collected by the Dask scheduler before the analytics workflow had a chance to consume them.
This issue was more likely to appear when multiple arrays were published concurrently and when the scheduler aggressively reclaimed unused objects.&lt;/p&gt;
&lt;h3&gt;The Fix&lt;span class="hx:absolute hx:-mt-20" id="the-fix"&gt;&lt;/span&gt;
&lt;a href="#the-fix" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;Version 0.5.1 addresses this issue by explicitly persisting generated Dask arrays using &lt;code&gt;client.persist()&lt;/code&gt;.
This ensures that arrays remain alive on the Dask cluster until they are consumed by the analysis workflow, preventing premature garbage collection and eliminating the possibility of missing data.&lt;/p&gt;
&lt;p&gt;In addition to the runtime fix, the release strengthens the test suite with:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Parametric tests covering different numbers of Dask workers.&lt;/li&gt;
&lt;li&gt;Improved handling of intermittent test failures.&lt;/li&gt;
&lt;li&gt;Additional exception handling in tests.&lt;/li&gt;
&lt;li&gt;More robust synchronization logic for worker-dependent scenarios.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Impact on Users&lt;span class="hx:absolute hx:-mt-20" id="impact-on-users"&gt;&lt;/span&gt;
&lt;a href="#impact-on-users" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;If your workflow transfers a single array per callback, you may never have encountered this issue.
However, users exposing multiple arrays simultaneously, such as coupled analytics pipelines, multi-field simulations, or machine-learning workflows, should upgrade to 0.5.1.&lt;/p&gt;
&lt;p&gt;The upgrade requires no API changes and is fully backward compatible with deisa-dask 0.5.0.&lt;/p&gt;
&lt;h2&gt;Additional Improvements&lt;span class="hx:absolute hx:-mt-20" id="additional-improvements"&gt;&lt;/span&gt;
&lt;a href="#additional-improvements" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This release also includes several infrastructure and maintenance updates:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Improved test execution through parallelized testing.&lt;/li&gt;
&lt;li&gt;Better handling of chunk position metadata.&lt;/li&gt;
&lt;li&gt;Migration from setup.py to a modern pyproject.toml build system.&lt;/li&gt;
&lt;li&gt;Addition of project metadata through codemeta.json.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Upgrading&lt;span class="hx:absolute hx:-mt-20" id="upgrading"&gt;&lt;/span&gt;
&lt;a href="#upgrading" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;h3&gt;Upgrade using pip&lt;span class="hx:absolute hx:-mt-20" id="upgrade-using-pip"&gt;&lt;/span&gt;
&lt;a href="#upgrade-using-pip" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fish" data-lang="fish"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nf"&gt;pip&lt;/span&gt; install &lt;span class="na"&gt;--upgrade&lt;/span&gt; deisa-dask&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;h3&gt;Upgrade using Spack&lt;span class="hx:absolute hx:-mt-20" id="upgrade-using-spack"&gt;&lt;/span&gt;
&lt;a href="#upgrade-using-spack" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;deisa-dask is packaged in the official &lt;a
href="https://github.com/spack/spack-packages"target="_blank" rel="noopener"&gt;spack-packages&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt; repository.
The latest release is found on the develop branch.&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fish" data-lang="fish"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nf"&gt;spack&lt;/span&gt; repo update &lt;span class="c"&gt;# update the repo that is tracking the develop branch
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nf"&gt;spack&lt;/span&gt; versions py-deisa-dask &lt;span class="c"&gt;# check that version 0.5.1 is found
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nf"&gt;spack&lt;/span&gt; install py-deisa-dask&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;h3&gt;Upgrade using Guix&lt;span class="hx:absolute hx:-mt-20" id="upgrade-using-guix"&gt;&lt;/span&gt;
&lt;a href="#upgrade-using-guix" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h3&gt;&lt;p&gt;deisa-dask is also packaged for Guix in the &lt;a
href="https://codeberg.org/guix-science/guix-science"target="_blank" rel="noopener"&gt;guix-science&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt; channel.
Your Guix setup must be configured to use the guix-science channel.&lt;/p&gt;
&lt;div class="hextra-code-block hx:relative hx:mt-6 hx:first:mt-0 hx:group/code"&gt;
&lt;div&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fish" data-lang="fish"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nf"&gt;guix&lt;/span&gt; pull # update guix and &lt;span class="nf"&gt;it&lt;/span&gt;&amp;#39;s channels&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&lt;div class="hextra-code-copy-btn-container hx:opacity-0 hx:transition hx:group-hover/code:opacity-100 hx:flex hx:gap-1 hx:absolute hx:m-[11px] hx:right-0 hx:top-0"&gt;
&lt;button
class="hextra-code-copy-btn hx:group/copybtn hx:cursor-pointer hx:transition-all hx:active:opacity-50 hx:bg-primary-700/5 hx:border hx:border-black/5 hx:text-gray-600 hx:hover:text-gray-900 hx:rounded-md hx:p-1.5 hx:dark:bg-primary-300/10 hx:dark:border-white/10 hx:dark:text-gray-400 hx:dark:hover:text-gray-50"
title="Copy code"
aria-label="Copy code"
data-copied-label="Copied!"
&gt;
&lt;div class="hextra-copy-icon hx:group-[.copied]/copybtn:hidden hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;div class="hextra-success-icon hx:hidden hx:group-[.copied]/copybtn:block hx:pointer-events-none hx:h-4 hx:w-4"&gt;&lt;/div&gt;
&lt;/button&gt;
&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;We recommend all users running deisa-dask 0.5.0 to upgrade to 0.5.1 to benefit from the improved reliability of data transfers and stronger test coverage.&lt;/p&gt;
&lt;h2&gt;Contributors&lt;span class="hx:absolute hx:-mt-20" id="contributors"&gt;&lt;/span&gt;
&lt;a href="#contributors" class="subheading-anchor" aria-label="Permalink for this section"&gt;&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This release was made possible by contributions from 1 new contributors:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a
href="https://github.com/abhinav-jay"target="_blank" rel="noopener"&gt;@abhinav-jay&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt; - &lt;code&gt;codemeta.json&lt;/code&gt; and this wonderful website&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Full Changelog&lt;/strong&gt;: &lt;a
href="https://github.com/deisa-project/deisa-dask/compare/v0.5.0...v0.5.1"target="_blank" rel="noopener"&gt;https://github.com/deisa-project/deisa-dask/compare/v0.5.0...v0.5.1&lt;svg class="hx:inline hx:rtl:rotate-270 hx:align-baseline" height="1em" aria-hidden="true" fill="none" stroke="currentColor" stroke-width="2" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"&gt;
&lt;path d="m9.1716 7.7574h7.0711m0 0v7.0711m0-7.0711-8.4853 8.4853" stroke-linecap="round" stroke-linejoin="round"/&gt;
&lt;/svg&gt;&lt;/a&gt;&lt;/p&gt;</description></item></channel></rss>