dask
dask.deisa
Deisa Objects
class Deisa(IDeisa)__init__
def __init__(feedback_queue_size: int = 1024, *args, **kwargs) -> NoneInitializes a class instance, configuring the client and setting up the necessary infrastructure for interactions. This includes setting up the necessary feedback queue length, performing handshake operations with the client, and initializing various metadata structures.
:param feedback_queue_size:The maximum size of the feedback queue. Defaults to 1024. -:type feedback_queue_size:int -:param args:Additional positional arguments passed to the initializer. -:type args:tuple -:param kwargs:Additional keyword arguments passed to the initializer. -:type kwargs:dict
register
def register(*callback_args: CallbackArgs,
exception_handler: IDeisa.
ExceptionHandler = __default_exception_handler,
when: Literal["AND", "OR"] = "AND",
precompute: bool = True) -> CallableRegisters a callback function with specific arguments, exception handling, and conditional execution criteria.
This function acts as a decorator that allows you to register a callback with parameters provided
through ``callback_args``. It also handles exceptions using the ``exception_handler`` and defines the
execution rules with ``when`` parameter.
Supports: Default window size is 1. ``@deisa.register("arr1")`` # default window size
``@deisa.register("arr1", "arr2")`` # two arrays, default window size
``@deisa.register(Window("arr1"))`` # default window size ``@deisa.register(Window("arr1", 2))`` #
window size 2 ``@deisa.register(Window("arr1", 2), Window("arr2", 5))`` # window size 2 for arr1 and 5
for arr2 ``@deisa.register(Window("arr1", 2), Window("arr2", 5), "arr3")`` # window size 2 for arr1 and
5 for arr2, default window size for arr3
Every callback is automatically analyzed for dask reduction operations (sum, mean, std, var, max, min,
prod) which are executed locally on each bridge before scatter to reduce network transfer. Precompute
analysis is always attempted (the default), and any callback that cannot be precomputed (no reductions,
or a reduction that depends on another reduction's output) raises at registration time. Use
``precompute=False`` to skip the analysis with a warning and fall back to the legacy full-chunk scatter
path.
Branch filing contract: registration merges branches in memory and files them with the handshake actor
immediately (required: a bridge on the wait_for_go=False path fetches lazily on its first send()).
``unregister_callback`` prunes memory only; the pruned set is re-filed at the per-cycle boundary by
``execute_callbacks()`` (which always flushes before unblocking the bridges). Callbacks registered
without a subsequent ``execute_callbacks()`` still reach the bridges via the immediate
registration-time filing.
:param callback_args:Variable-length arguments representing callback-specific parameters. -:param exception_handler:Optional exception handler to manage errors during callback execution. Defaults to__default_exception_handler. -:param when:Specifies the conditional logic for triggering the callback. Can be ‘AND’ or ‘OR’. Defaults to ‘AND’. -:param precompute:If False, skip precompute analysis with a warning and use the full-chunk scatter path. Defaults to True (analysis is required). -:return:A callable that wraps the provided callback with the configured parameters and logic. -:rtype:Callable
register_callback
def register_callback(callback: IDeisa.Callback,
*callback_args: CallbackArgs,
exception_handler: IDeisa.
ExceptionHandler = __default_exception_handler,
when: Literal["AND", "OR"] = "AND",
precompute: bool = True) -> CallableRegisters a callback function with specific arguments, exception handling, and conditional execution
criteria. This function allows you to register a callback with parameters provided through
callback_args. It also handles exceptions using the exception_handler and defines the execution
rules with when parameter. Supports: Default window size is 1. @deisa.register("arr1") #
default window size @deisa.register("arr1", "arr2") # two arrays, default window size
@deisa.register(Window("arr1")) # default window size @deisa.register(Window("arr1", 2)) #
window size 2 @deisa.register(Window("arr1", 2), Window("arr2", 5)) # window size 2 for arr1 and 5
for arr2 @deisa.register(Window("arr1", 2), Window("arr2", 5), "arr3") # window size 2 for arr1 and
5 for arr2, default window size for arr3
:param callback:Callback function to register. -:param callback_args:Variable-length arguments representing callback-specific parameters. -:param exception_handler:Optional exception handler to manage errors during callback execution. -:param when:Specifies the conditional logic for triggering the callback. Can be ‘AND’ or ‘OR’. -:param precompute:If False, skip precompute analysis with a warning and send the full data to workers. -:return:A callable that wraps the provided callback with the configured parameters and logic.
set
def set(key: str, value: Any, timestep: int) -> NoneSets a value in a queue for a given key, associating it with a specific timestep. This action is intended to store feedback or other time-specific data for the provided key.
:param key:The identifier for which the value is to be set. -:type key:str -:param value:The value to be stored, associated with the key and timestep. -:type value:Any -:param timestep:The timestamp that corresponds to when the value is set. -:type timestep:int -:return:None
execute_callbacks
def execute_callbacks() -> NoneExecutes a series of callbacks and waits for necessary processes to finish.
This method handles the execution of callbacks related to bridges and their completion while also
ensuring the orchestration of subsequent tasks. It is responsible for unblocking bridges and waiting
for dependencies to signal completion.
This is also the branch-filing boundary for unregistration: the pruned branch sets (pruned in memory by
``unregister_callback``) are re-filed to the handshake actor here, once per cycle, before the bridges
see them. - ``:param self:`` The instance of the class invoking this method. - ``:return:`` None
dask.__version__
dask.utils
get_mpi_comm_world
def get_mpi_comm_world(cart_coord_dims: int = 1) -> ICommunicatorComputes and returns an MPI Cartesian communicator based on the number of desired Cartesian coordinate dimensions.
This function uses the MPI library to calculate and create a Cartesian communicator from the global MPI
communicator (MPI.COMM_WORLD). The dimensions of the Cartesian coordinate grid are determined dynamically based
on the size of the Cartesian coordinate dimensions requested by the user and the size of the MPI communicator.
:param cart_coord_dims: Number of Cartesian coordinate dimensions used to compute the grid layout for the
Cartesian communicator. Default is 1. :type cart_coord_dims: int :return: A new Cartesian communicator
created from the MPI world communicator based on the computed dimensions. :rtype: mpi4py.MPI.Cartcomm
get_mpi_cart_coords
def get_mpi_cart_coords(*comms: Any) -> Optional[List[int]]The rank’s Cartesian coords from the first comm that carries mpi4py’s Cartcomm methods, else None.
Those methods exist only on the Cartcomm subclass – and MPI_Comm_split drops the topology, so the array’s
own sub-communicator and the parent are both tried.
make_precomputed_view
def make_precomputed_view(first: da.Array,
*,
t: int,
signatures: dict,
reapply,
registered_ndim: int,
pca_summary=None,
registered_shape=None) -> _PrecomputedDeisaArrayBuild the per-callback dispatch view WITHOUT touching Array.__new__.
Creates a genuine DeisaArray through :func:build_deisa_array (the same factory every other delivery path
uses), attaches the dispatch fields, then transplants the view class onto the instance.
DeisaArray/dask.array.Array instances are plain objects whose state lives in __dict__, so the layout is
identical across subclasses and the swap is safe (dask itself returns plain Array instances when a derived
subclass cannot be preserved).
signatures may be EMPTY here, but only when pca_summary is given: a callback whose only precomputable work
is a bridge-side PCA has no combined reduction to route, so it would otherwise trip the “non-empty signatures”
guard below for a view that is perfectly legitimate.
:param first:The carrier array the view is instantiated from. For a reduction callback this is the combined array of its first reduction; for a PCA-only callback it is the one-element placeholder fromDeisa._pca_carrier_array, whose own shape is 1 along every axis and therefore NOT the registered array’s shape.:param t:The timestep stamped onto the builtDeisaArray.:param signatures:{(op_name, reduction_axes): combined_array}, routed by_PrecomputedDeisaArray._dispatch.:param reapply:The subset ofsignatureswhose stored array is per-bridge partials to re-aggregate, not a final value.:param registered_ndim:ndimof the REGISTERED array, so a caller’s axis is normalized against the shape the callback was written against, not the reduced delivered shape.:param pca_summary:The merged in-situ PCA delivery (:class:~deisa.dask.branch.DeliveredPCA, or a bareDelayed/PCASummary), orNonefor a reduction-only callback. Stored as_deisa_pca_summarybecause that is the attribute :meth:~deisa.dask.mergeable_pca.MergeablePCA.fitlooks for.:param registered_shape:shapeof the REGISTERED array, orNonewhen it is unknown. Stored as_deisa_registered_shapebecause the PCA carrier’s own shape is a one-element placeholder, so the estimator needs the REGISTERED shape to know the real feature dimensiond; it is None only when no registered metadata was available at delivery time, in which case that check falls back to trusting the summary.
called_from_event_loop
def called_from_event_loop() -> boolWhether the current thread is running an asyncio event loop, i.e. we are inside a callback.
:return:True if an event loop is running in this thread, False otherwise.
run_coro_on_private_loop
def run_coro_on_private_loop(coro)Run coro on a private event loop hosted by a short-lived worker thread.
Needed when the calling thread already has a running event loop: blocking that thread on
loop.run_until_complete(coro) would deadlock, because a running loop cannot be re-entered. A dedicated thread
with its own loop is unaffected by whatever the caller is doing, so the coroutine completes and the caller blocks
only on the future’s result.
dask.handshake
Handshake Objects
class Handshake()all_bridges_ready
def all_bridges_ready(nb_bridge: int,
arrays_metadata: dict,
wait_for_go=True) -> NoneBridge must wait for analytics to be ready.
deisa_ready
def deisa_ready(feedback_queue_size: int = 1024,
wait_for_go=True,
*args,
**kwargs) -> NoneWhen analytics is ready, notify all Bridges
dask.mergeable_pca
Mergeable PCA primitives: exact per-block PCA summaries that merge without the source samples.
What this module provides
A local PCA summary of one data block – (n_samples, mean, components, singular_values) – and an associative
binary merge of two such summaries. A merge consumes only summaries: no source sample ever reaches a merge node.
That is what makes an in-situ PCA possible on deisa-dask, where the data stays on the MPI bridge and only a compact
summary crosses the process/network boundary.
Attribution: the algebra here is classical, NOT novel
Nothing in this module is a new algorithm, and both the paper and this docstring concede the prior art by name:
- The between-block mean-correction term is the Chan-Golub-LeVeque covariance merge (1979):
M_AB = M_A + M_B + (n_A * n_B / (n_A + n_B)) * delta * delta.Twithdelta = mean_A - mean_B. It is the standard scatter-matrix identity behind parallel BLAS. Here it enters as the extracorrectionrow of the compact matrix in :func:merge_pca, whose outer product reproduces thedelta * delta.Tterm. - The merge-plus-tree-reduction shape (merging rank-
rsummaries pairwise, re-truncating at each node) is Iwen & Ong, arXiv:1601.07010 (SIAM J. Matrix Anal. Appl. 37(4):1699-1718, 2016, DOI 10.1137/16M1058467) – their Lemma 1 states the merge and their hierarchical Algorithm 1 the reduction tree – and Vasudevan & Ramakrishna, arXiv:1710.02812 – merge-and-truncate. - The only thing claimed as ours is the placement: the local PCA runs on the MPI bridge, where the data already resides, and only the summary crosses to Dask.
Scalability: what saturates, and what a node costs
Rank saturation. A node that stacks summaries into a compact matrix of rows(compact) rows keeps
min(rows(compact), feature_dim) of them, so a merge yields rank min(r_a + r_b + 1, d) with d the feature
dimension. The summary rank SATURATES at min(rows(compact), feature_dim) and does not grow with tree
depth past that ceiling: the +1 each level adds is capped by d. With full local rank (rank=None) every
leaf already has rank min(n_block, d), so the first merge saturates at d and all deeper levels are flat. With
a truncated rank = R < d the rank does climb with depth until it reaches the same ceiling (measured, d = 50,
R = 5: 11, 23, 47, 50 for 2, 4, 8, 16 blocks), so a truncated summary can exceed R * d elements before
saturation.
Cost. A merge node SVDs a matrix of at most (2d + 1, d): O(d^3), independent of the total sample count N
and of the per-block sample count. A tree over B blocks has O(B) nodes, so the whole reduction is
O(B * d^3) no matter how large N is. That is the property which makes the summary a valid stand-in for the
samples; the summary size is not independent of d and of the local rank, only the per-node cost is.
Exactness. With full local rank the merged summary is not an approximation of the pooled PCA: it reproduces the singular values and the principal subspace of a batch SVD of the whole centered array, up to roundoff.
n-dimensional input: the axis policy
PCA needs two kinds of axis – SAMPLES (rows) and FEATURES (columns) – and an array of more than two dimensions does not say which is which. This module never guesses. Three constructor parameters say so explicitly:
axis_names: what each axis is called, positionally. A Dask array carries no axis names of its own, so the caller is the only place the names can come from; they then appear verbatim in every refusal message.feature_axes: which of those axes are features. Defaults to the axes the array’s CHUNKING leaves WHOLE, which are the axes complete on the rank that holds them (see the default rule below).sample_axes: which axes are samples. Defaults to every axis that is not a feature, so givingfeature_axesalone is enough.
Both feature_axes and sample_axes accept names or positions, and if BOTH are given they must partition the
axes exactly: an axis claimed by neither, by both, or twice, is refused rather than guessed. A 2-D array needs none of
this – its last axis is the feature axis – so the existing 2-D call sites are untouched.
The default feature axes come from the chunk layout, never from a name
A mergeable summary stacks by feature position, so the feature dimension has to be COMPLETE on the rank that holds it. Where an array has already been distributed, that fact is in the array itself: the distributed axes carry more than one Dask chunk and the axes each rank holds whole carry exactly one. So the default reads that structure instead of matching a name:
- The caller’s
feature_axeswins whenever it is given, verbatim. - Else, if the array splits SOME but not ALL of its axes, the axes it leaves WHOLE are the features and the axes it splits are the samples. A registration that distributes the sample axes and keeps the feature space whole on every rank therefore needs no configuration at all.
- Else – every axis whole, or every axis split, so the chunking says nothing – the LAST axis is the feature axis, which keeps single-rank and fully materialized slabs working under a defined, non-guessing rule. A 2-D array always takes this branch, exactly as it always has.
A numpy bridge chunk carries no chunk structure of its own (it is ONE rank’s slab), so the bridge leaf takes the
caller’s specification or, when the caller gave none, the same no-structure fallback of rule 3 – which is the rule
the estimator applies to a materialized array. A caller who wants a specific partition in situ states
feature_axes, which is shipped verbatim to the bridge.
Two layouts, measured, and they behave OPPOSITELY
The two branches of rule 2 are the two ways to read a distributed field. For a gyrokinetic distribution indexed
(species, tor1, tor2, tor3, vpar, mu) split across MPI ranks by the three spatial axes (each rank therefore owns
a contiguous spatial box with the COMPLETE velocity space), the split axes are the samples and the whole velocity
space is the feature set – rule 2, with no configuration:
- Layout A – PCA over velocity space, per spatial cell. The features are
(vpar, mu), sod = Nvpar * Nmuis fixed by the physics and INDEPENDENT of the rank count. Samples are the spatial cells of the rank’s own box. Measured on(tor1, tor2, tor3, vpar, mu) = (512, 128, 64, 128, 8), i.e.d = 1024: a full-rank leaf summary is 8.0 MiB against a 4096 MiB slab at 8 ranks (511x) and against a 512 MiB slab at 64 ranks (64x). It compresses at every rank count measured, so full local rank is viable and the merge is EXACT. - Layout B – PCA over the spatial box, per velocity cell. The features are
(tor1, tor2, tor3), sodis the LOCAL box size and GROWS AS RANKS DECREASE – the opposite of the usual scaling intuition. Measured on the same field at 8 ranks:d = 524288features,n_samples = 1024velocity cells, and the merged full-rank summary reaches rank 8199, i.e. 32800 MiB against a distributed slab of 32768 MiB: 1.00x, NO compression at all. (Sizing the leaf at the naiver = drather than the truemin(n_block, d) = 1024gives the 2097156 MiB figure some write-ups quote; either way the summary is not smaller than the data.)
Consequence for the API: the axis policy makes the difference explicit instead of hiding it, and a full-rank summary
that is not smaller than its input is REFUSED rather than produced – see :class:MergeablePCA for the guard and its
measured reason. The default is not a guess about the field’s meaning: it is the partition the chunk layout already
implies, and the guard still refuses the partition that cannot compress.
Repository constraints honoured here
ruffline length is 120 and it applies to docstring prose and toraiseliterals too, so long messages are wrapped across lines rather than left on one line.- Zero new runtime dependencies:
numpy.linalg.svdonly. scipy and scikit-learn are absent from this repo and are not needed – every SVD here is small by construction. - Python >= 3.10, with
from __future__ import annotations. - Minimal diff: this module is purely additive and refactors nothing else.
- All three functions are module level (no lambda, no closure) so they pickle, which is required for the future
bridge
branch_functarget and for use throughdask.delayed.
PCASummary Objects
@dataclass(frozen=True, eq=False)
class PCASummary()Exact mergeable PCA state of a set of samples.
The fields are named after what they hold. n_samples is a count; mean is the pooled feature-wise mean;
components holds the principal axes as ROWS (the right singular vectors, i.e. Vh of
numpy.linalg.svd, so that components.T @ components == I); singular_values holds the singular values
of the centered data, in descending order.
The mergeable form is A = singular_values[:, None] * components: merging stacks those products, so a
summary carries the scatter matrix of its samples without carrying a single sample.
Equality is deliberately not generated: the array fields would make == ambiguous (elementwise array), which
is a classic source of silent bugs in merge code. Compare fields, or use np.allclose.
:param n_samples:Number of samples represented, always >= 1.:param mean:Pooled mean,float64of shape(d,).:param components:Principal axes as rows,float64of shape(rank, d).:param singular_values:Singular values of the centered samples,float64of shape(rank,).
rank
@property
def rank() -> intRank of the summary, i.e. how many singular values/components it retains.
local_pca
def local_pca(block: np.ndarray, rank: int | None = None) -> PCASummarySummarize one data block: center it, take its SVD, and keep the leading rank directions.
rank=None keeps the full local rank min(n_block, d). Truncating to rank = R shrinks the summary (and
therefore what has to cross the bridge boundary) at the cost of discarding variance that the merge cannot
recover; see this module’s docstring for the rank law.
The block is never mutated: it is cast to float64 if needed and centering allocates a fresh array.
:param block:2-D(n_samples, n_features)block of one rank’s data.:param rank:Maximum retained rank, orNonefor the full local rankmin(n_samples, n_features).
merge_pca
def merge_pca(a: PCASummary, b: PCASummary) -> PCASummaryMerge two summaries of disjoint sample sets into the exact summary of their union.
This is the classical merge, stated once so the code and the citation cannot drift apart:
A = a.singular_values[:, None] * a.componentsand likewiseB, soA.T @ Ais the centered scatter matrix ofa’s samples andB.T @ Bthat ofb’s.correction = sqrt(n_a * n_b / (n_a + n_b)) * (a.mean - b.mean)is the between-block mean-correction term of the Chan-Golub-LeVeque (1979) scatter merge,(n_a * n_b / (n_a + n_b)) * delta * delta.Twithdelta = mean_a - mean_b. It is what makes the merge exact; dropping it silently biases every merge that sees blocks with different means.- Stacking
A,Bandcorrectionintocompactgivescompact.T @ compactequal to the pooled centered scatter matrix, so the SVD ofcompactyields the pooled singular values and principal axes.
Associativity holds up to roundoff, which is what justifies the balanced tree of :func:merge_tree.
:param a:Summary of the first (disjoint) sample set.:param b:Summary of the second (disjoint) sample set.
merge_tree
def merge_tree(summaries: Sequence[PCASummary]) -> PCASummaryReduce summaries with a balanced pairwise tree, the reduction shape of Iwen & Ong (arXiv:1601.07010, Algorithm 1).
Each level merges neighbours pairwise; an odd level carries its last summary up unchanged to the next level, so the tree stays balanced instead of degenerating into a chain. A single summary is returned as-is (base case) and an empty input is refused.
The tree is safe because :func:merge_pca is associative: any grouping of the same summaries gives the same
singular values up to roundoff, so the balanced shape is a cost decision, not a correctness one.
:param summaries:Sequence of at least one summary, over disjoint sample sets with one feature dimension.
local_pca_from_chunk
def local_pca_from_chunk(chunk: np.ndarray,
rank: int | None = None,
axis_names: Sequence[str] | None = None,
feature_axes: object = None,
sample_axes: object = None) -> PCASummarySummarize ONE bridge-local chunk under the estimator’s axis policy. The bridge branch’s callable.
Why this exists next to :func:local_pca: the bridge hands a numpy chunk, not a Dask array, so the axis policy
that turns a multi-dimensional field into (n_samples, n_features) has to be applied to numpy. Routing BOTH
sides through this one function is what makes “bridge-side == standalone” testable rather than merely asserted:
the bridge calls this, the estimator’s own leaf calls :func:local_pca on the already-flattened block, and both
bottom out in the identical SVD-and-mean code with no second implementation to drift. The axis CLASSIFICATION is
shared too – :func:_partition_axes, the same function :meth:MergeablePCA._as_2d calls – with one difference
that is a property of the input, not of the policy: a bridge chunk is one rank’s numpy slab and has no chunk
structure to read, so the default here is the caller’s feature_axes or the last-axis fallback, exactly as the
estimator resolves a materialized array.
A 2-D chunk takes the identical path :func:local_pca takes, because the policy leaves it untouched. A chunk
whose feature dimension is SPLIT across ranks is refused rather than summarized: every per-bridge summary must
span the same d for the merge to stack them, and a summary over half the features describes a different
problem, not a smaller one.
:param chunk:This bridge’s numpy chunk, of any dimensionality.:param rank:Maximum retained rank, orNonefor the full local rankmin(n_samples, n_features).:param axis_names:One name per axis of the registered array, orNoneto classify by position.:param feature_axes:Which axes are features, by name or position, orNonefor the default policy.:param sample_axes:Which axes are samples, by name or position, orNonefor every non-feature axis.
MergeablePCA Objects
class MergeablePCA()PCA of a Dask array computed from mergeable per-block summaries, never from the pooled samples.
The estimator is scikit-learn-STYLE – trailing-underscore fitted attributes, fit returning self, y
accepted and ignored – but it deliberately does not subclass sklearn.base.BaseEstimator: scikit-learn is not a
dependency of this repository and adding one to obtain a base class is not worth it. Everything scikit-learn’s
estimator protocol actually needs (validate params in __init__, do no work there, fit, transform) is here.
How it works
:meth:_fit_dask_delayed builds one dask.delayed(local_pca) task per input block and reduces those summaries
with a balanced tree of dask.delayed(merge_pca) tasks (the shape of Iwen & Ong, arXiv:1601.07010, Algorithm 1).
The reduction consumes ONLY summaries: no source sample ever reaches a merge node, so the summary is a valid
stand-in for the data it describes. With full local rank the result is not an approximation – it reproduces the
batch SVD of the whole centered array to roundoff, because the merge carries the Chan-Golub-LeVeque (1979)
between-block mean correction (see :func:merge_pca).
Two graph hooks, deliberately split
:meth:_fit_dask_delayed returns the constructed-but-unbuilt Dask graph and NEVER calls .compute();
:meth:_fit_dask is the thin .compute() wrapper on top of it. This split is not cosmetic: calling
compute() inside the builder is exactly what made the graph unvisualizable in the original design, because a
computed result is a value and values have no graph to draw. pca._fit_dask_delayed(X).visualize() therefore
works, which is the point of exposing it.
Internal vs public state
self._summary_ keeps the mergeable summary, whose rank is generally HIGHER than n_components because the
merge saturates at min(rows(compact), n_features) rather than at the requested component count. The public
attributes (components_, singular_values_, explained_variance_) describe the REQUESTED final PCA and are
truncated (and optionally whitened) from _summary_ afterwards. Truncation and whitening happen ONLY here at the
root, never mid-tree: both destroy mergeability and exactness, so the tree runs on full untruncated summaries.
Bridge-side code emits _summary_, not the truncated public form.
The accuracy/bandwidth trade-off of local_rank
local_rank=None keeps the full local rank min(n_block, n_features) and the merge is exact. local_rank=R
truncates each leaf to rank R before merging, which shrinks what has to cross a process or network boundary
from ~min(n_block, d) * d to ~R * d elements – the knob that makes the bridge path a bandwidth win for
wide, short blocks. The price is accuracy: the discarded leaf variance cannot be recovered by any merge, so the
retained subspace drifts from the batch one and the retained variance falls. Both losses are monotone in R and
vanish at R >= d. There is no universally safe R; :meth:fit refuses the one combination that cannot work
at all (a root rank below n_components) instead of silently returning fewer components.
n_components=None is the one request that a truncated merge cannot satisfy either, and it is refused for the
same reason: “keep everything” asks for min(n_samples, n_features) components, which a truncated merge does not
reach. The invariant n_components_ == components_.shape[0] holds for every configuration that fits, so a caller
sizing a buffer from n_components_ always gets the width it is promised.
n-dimensional input: the axis policy
An array of more than two dimensions does not say which axes are SAMPLES (rows) and which are FEATURES (columns), and PCA gives a different answer for each choice, so the choice is never guessed. Three constructor parameters state it explicitly:
axis_names: what each axis is called, positionally. A Dask array carries no axis names of its own (it has nodimsattribute), so the caller is the only place names can come from – and they then appear verbatim in every refusal message, so a message can name the offending axis rather than its position.feature_axes: which axes are features. Defaults to the axes the CHUNK LAYOUT leaves WHOLE, i.e. the axes complete on the rank that holds them, and to the last axis when the layout says nothing (every axis a single chunk, or every axis split). A 2-D array always takes the last axis, as it always has.sample_axes: which axes are samples. Defaults to every axis that is not a feature.
Both feature_axes and sample_axes accept names or positions, and when BOTH are given they must partition
the axes exactly. An axis in neither set, in both, or named twice is refused with a message naming it. A 2-D array
needs none of this – its last axis is the feature axis – so existing 2-D call sites are unaffected, and for 2-D
input axis_names is accepted and used only in messages.
The array is transposed so the feature axes come last, then reshaped to (n_samples, n_features). That reshape
is the ONLY flattening performed, it is value-preserving, and it is explicit in the reported shapes: for an array
whose split axes are the three spatial ones and whose whole axes are the two velocity ones, the features are
Nvpar * Nmu and the samples are the product of the split axes. One leaf is built per Dask block of the SAMPLE
axes, which is exactly one leaf per rank: a rank’s contiguous box is one leaf, complete in the feature space,
because the split axes define the leaves and the whole axes define the features.
On the BRIDGE path the leaf sees ONE rank’s numpy slab, which carries no chunk structure of its own, so there the
default is the caller’s specification, or, with none, the same last-axis fallback this estimator applies to a
materialized array – one policy function (:func:_partition_axes), one place it can be read. A caller who wants
a specific partition in situ states feature_axes, which is shipped to the bridge verbatim.
Why the structural default is the compressing one, and the other is refused
For a gyrokinetic distribution split across MPI ranks by the three spatial axes, each rank holds a contiguous
(tor1, tor2, tor3) box with the COMPLETE velocity space – so the SPLIT axes are the samples and the WHOLE
axes are the features, which is exactly what the default reads. Which axes are the features therefore decides the
whole regime, and the two choices behave oppositely. Measured on
(tor1, tor2, tor3, vpar, mu) = (512, 128, 64, 128, 8), a leaf summary being (rank, d) + d float64:
- Layout A, features
(vpar, mu)–d = 1024, fixed by the physics and INDEPENDENT of the rank count. A full-rank leaf summary is 8.0 MiB against a 4096 MiB slab at 8 ranks (511x) and a 512 MiB slab at 64 ranks (64x): it compresses at every rank count measured, so full local rank is viable and the merge is EXACT. - Layout B, features
(tor1, tor2, tor3)– the other partition of the same array, declared explicitly:dis the LOCAL box size and GROWS AS RANKS DECREASE, the opposite of the usual scaling intuition. At 8 ranksd = 524288withn_samples = 1024velocity cells, and the merged full-rank summary reaches rank 8199: 32800 MiB against a distributed slab of 32768 MiB, i.e. 1.00x. No compression. (Sizing a leaf at the naiver = drather than the truemin(n_block, d) = 1024yields the 2097156 MiB figure quoted elsewhere; either way the summary is not smaller than the data.)
A summary that is not smaller than its input is refused by :meth:_check_summary_smaller_than_input, which names
the measured reason and the remedy (local_rank=R, or the other axis partition). Under Layout B local_rank
is therefore effectively mandatory and is documented as such. Under Layout A the default local_rank=None is
the right answer.
Repository constraints honoured here
ruffline length is 120 and it applies to docstring prose and toraiseliterals too, so long messages are wrapped across lines.Zero new runtime dependencies:
daskandnumpyonly. scipy and scikit-learn are used in the TESTS as reference implementations, never imported bysrc/.Refuse, do not approximate: every rejection names the offending condition AND the remedy.
:param n_components:Number of final public components, orNoneto keep allmin(n_samples, n_features).:param whiten:IfTrue, scalecomponents_after truncation so projected columns have unit variance.:param copy:Accepted for scikit-learn API parity. It has NO effect:fitnever mutatesX, because every local PCA centers a freshfloat64copy of its own block.:param batch_size:Optional maximum rows per local-PCA leaf.Nonemeans one leaf per Dask row chunk. If set, each row chunk is split into leaves of at most this many rows before the local PCA. Merging is associative, so this only reshapes the tree and does not change the result above roundoff.:param local_rank:Maximum rank of each leaf summary, orNonefor the full local rankmin(n_block, n_features). Optional, andNone(full rank, exact) is the right default under Layout A. Under Layout B it is effectively MANDATORY: the feature dimension is the local spatial box, so a full-rank summary is not smaller than its input and :meth:_check_summary_smaller_than_inputrefuses it.:param axis_names:Optional name per axis, positionally, used in messages and in thefeature_axes/sample_axeslookup.Nonefor 2-D input, or when the axes are to be classified by position alone.:param feature_axes:Which axes are features, by name or position.Nonedefaults to the axes the chunk layout leaves WHOLE (the axes complete on the rank that holds them), else to the last axis.:param sample_axes:Which axes are samples, by name or position.Nonedefaults to every axis that is not a feature. Give it only when the features do NOT cover the rest, because giving both partitions exactly.
fit
def fit(X, y=None) -> MergeablePCAFit the estimator and return self.
Three paths, all ending in the SAME root state (_summary_ plus the materialized public attributes), so
“in-situ on the bridge” and “standalone on Dask” cannot drift apart:
- A delivered array (one carrying
_deisa_pca_summary, i.e. the per-bridge dispatch view deisa-dask hands a callback when the local PCA ran on the MPI bridge) takes :meth:_fit_delivered. The merge tree over the per-bridge summaries has already run; this only adopts the root summary. - A dask array is fitted through :meth:
_fit_dask_delayed/ :meth:_fit_dask. - A numpy array takes the in-memory path, the single-leaf base case of the same tree.
Input of any dimensionality is accepted under the axis policy described in this class’s docstring: the array is
transposed so the feature axes come last, then reshaped to (n_samples, n_features) before anything else
happens.
:param X:Array ofn_samples * n_featuresvalues, dask or numpy, 2-D or more. For more than two dimensions,axis_names/feature_axes/sample_axessay which axes are which.:param y:Accepted and ignored, for scikit-learn parity.
fit_transform
def fit_transform(X, y=None)Fit, then project X onto the components. Returns the projection, not the estimator.
:param X:2-D(n_samples, n_features)array, dask or numpy.:param y:Accepted and ignored, for scikit-learn parity.
transform
def transform(X)Project X onto the fitted components, honoring whiten.
Z = (X - mean_) @ components_.T, i.e. centered by the pooled mean first. With whiten=True the
components already carry the 1 / sqrt(explained_variance) scale, so the columns come out with unit sample
variance; this is scikit-learn’s formulation, expressed as one matrix product so Dask fuses it into the graph.
X goes through the SAME axis policy as fit, so a caller can project a 5-D field with the axis names
they fitted with. Only the feature dimension has to match the fitted one: the sample axes may be anything, as
usual for a projection.
:param X:Array matchingn_features_in_, dask or numpy, 2-D or more under the same axis policy as fit.
inverse_transform
def inverse_transform(Z)Map projected data back to the original feature space: X_hat = Z @ components_ + mean_.
Exact on the retained subspace only; the discarded directions cannot be recovered.
:param Z:2-D ``(n_samples, n_components_)` array, dask or numpy.
dask.branch
Branch-level local compute on the bridge.
A :class:BranchSpec describes one chunk-local sub-expression of the user’s callback that the bridge can execute on
its local numpy chunk and whose result the Deisa-side topic handler combines across bridges.
Branch of length-1 case: each BranchSpec corresponds to one detected reduction (arr.sum(),
arr.mean(axis=0), …). The branch is a single chunk callable followed by the reduction’s combine aggregator. For
multi-layer chains (length->=2 branches); the data structure below is designed to support that without further changes.
The structure mirrors the prior per-reduction branch metadata (kind / finalize / shape / dtype /
chunk_axis) so the bridge and Deisa-side combine code paths can be refactored to consume :class:BranchSpec
directly without changing semantics.
BranchSpec Objects
@dataclass
class BranchSpec()One chunk-local sub-expression the bridge can execute + combine.
Attributes
----------
output_key : str
Stable identifier for this branch (e.g. ``"f-mean"``). The bridge uses it to namespace its scatter key and
the Deisa topic handler uses it to route the per-bridge partials back to the same branch.
input_name : str
Registered array name the branch is rooted at (e.g. ``"a"``). The Deisa side groups branches per array using
this field (never by parsing ``output_key``) and files each group with ``set_task_branches`` under its own
array name via ``execute_callbacks`` -> ``_flush_branches_to_handshake``.
output_kind : str
One of ``"scalar"`` / ``"mean"`` / ``"moment"``. Drives the Deisa-side combine graph: ``scalar`` ->
``da.stack`` + dask sum, ``mean`` -> ``mean_agg`` over nested list of dicts, ``moment`` -> ``moment_agg`` (+
``np.sqrt`` for ``finalize == "sqrt"``).
branch_func : Callable
Python callable that, given a numpy chunk, returns the branch's per-bridge partial value (a scalar /
ndarray / dict). Pickled across the bridge process boundary. Currently a length-1 callable (``chunk_func``
from the prior branch); This may produce multi-callable composites.
chunk_axis : Optional[Tuple[int, ...]]
For reductions, the tuple of axes being reduced in the chunk (e.g. ``(0, 1)`` for full reduction on a 2-D
chunk). ``None`` for pointwise-only branches.
finalize : Optional[str]
``"sqrt"`` for std (apply ``np.sqrt`` after combining), else ``None``.
partial_shape : Tuple[int, ...]
Shape of the **per-bridge partial** (what the branch_func returns). For ``scalar`` reductions on a 2-D chunk
with ``keepdims=False`` this is ``()``; with ``keepdims=True`` it is ``(1, 1)``. For axis reductions the
partial keeps the un-reduced axes' full size (e.g. ``mean(axis=0)`` on ``(M, N)`` partial has shape ``(1,
N)``). The bridge records this on the topic event so the Deisa side knows what each bridge shipped.
partial_dtype : str
NumPy dtype string of the per-bridge partial.
merge_branches
def merge_branches(existing: List[BranchSpec],
new: List[BranchSpec]) -> List[BranchSpec]Merge two branch lists, deduping by output_key.
Identical signatures (same op, same reduction axes) share one bridge execution; same key with a DIFFERENT
reduction-axes signature (e.g. a window read vs. a true axis reduction over the same axis) always raises
:class:PrecomputeRuntimeError – the two deliver different shapes and sharing them silently delivers one callback
the other one’s result.
DeliveredPCA Objects
@dataclass(frozen=True)
class DeliveredPCA()A merged in-situ PCA summary plus the estimator configuration that should consume it.
Carried as an attribute (_deisa_pca_summary) on the per-callback dispatch view, which is what
:meth:MergeablePCA._fit_delivered looks for. A frozen dataclass pickles by value: no reconstruction callback, no
globals to resolve on a worker, and the Delayed inside it rebuilds its own graph when it does.
:param merged:Delayed(or any object with.compute()) yielding the mergedPCASummary.:param config:Root-only estimator configuration:n_components,whiten, and the axis policy.
compute
def compute() -> AnyMaterialize the merged summary. Equivalent to self.merged.compute().
Named explicitly so the estimator’s hasattr(delivered, "compute") branch treats this wrapper and a bare
Delayed identically, instead of a wrapper silently reading .merged as if it were the summary.
dask.task_branches
Extract reduction hints from a dask array’s task graph.
Given a dask array whose graph contains reductions (sum, mean, std, var, max, min, prod), walk the graph layers to find the aggregate layer and the matching chunk layer, then build a branch dict the bridge can execute on a local numpy chunk before scattering.
branch schema (the contract between the analytics side (precompute analyzer) and the bridge side (local chunk execution)):
.. code-block:: python
{ “output_key”: “fdistribu-sum”, # unique key for this reduction “op_name”: “sum”, # canonical op name “chunk_func_pickle”: …, # pickle of the chunk callable “chunk_kwargs”: {…}, # kwargs for the chunk callable “finalize”: “sqrt” | None, # post-step (sqrt for std) “array_name”: “a”, # registered array the reduction descends from “multi_source”: False, # True when the expression descends from >1 registered array }
This module is purely about reading the dask graph; it never executes user callbacks.
extract_reduction_hints
def extract_reduction_hints(
darr: da.Array,
array_name: str = "f",
output_key_seen: Optional[Dict] = None,
array_ndim: Optional[int] = None) -> List[Dict[str, Any]]Inspect darr’s task graph and return a branch dict per reduction.
array_ndim is the registered array’s ndim (when known, e.g. from its global_shape): it lets the
minimal-scalar-partial rule below recognize an axis tuple that covers every data axis. None keeps the
axis-None-only heuristic.
dask.constants
dask.precompute_analyzer
Compute-boundary precompute analyzer (no AST pattern matching, no callback execution).
PrecomputeError Objects
class PrecomputeError(Exception)Base class for precompute analysis errors.
UnsupportedReductionError Objects
class UnsupportedReductionError(PrecomputeError)A reduction is present but its input expression cannot be traced back to a dask array.
Covers opaque operands (da.sum(opaque_object)), wrapped reductions (a custom function around a dask call), and
untraceable parameters (da.sum(f * v2) where v2 is derived from a config object).
NoPrecomputableReductionError Objects
class NoPrecomputableReductionError(PrecomputeError)The callback contains no reductions we can precompute.
Example: callback only does da.fft.fft2(arr) - FFTs don’t reduce.
PrecomputeRuntimeError Objects
class PrecomputeRuntimeError(PrecomputeError)A runtime (post-registration) precompute invariant was violated.
Raised on the Deisa side (topic handler, callback dispatch view) and the bridge side (executing a branch on the local chunk) when a precompute contract breaks – e.g. a failed branch dropped a partial, a reduction’s partials do not cover the full chunk grid, or a callback calls a reduction that the analyzer did not record. Always prefer raising this over silently delivering a wrong number.
MaterializationError Objects
class MaterializationError(PrecomputeError)A full data materialization was detected in the callback.
Example: np.array(dask_array) - forces gathering the whole array, defeating the purpose.
RawFieldReadError Objects
class RawFieldReadError(PrecomputeError)The callback reads the registered array’s raw data, which precompute does not deliver.
The precompute path only ships per-bridge reduction partials – never the full chunk. Any callback that consumes
the raw data (binds it as a value, indexes it, plots it, or computes something that is not a recorded
chunk-local reduction) breaks the precompute contract: the data it needs is not on the network. Raised at
REGISTRATION time – never mid-iteration. Remedy: register the callback with precompute=False, which
switches it to the legacy full-chunk scatter path.
IncompatibleCallbackError Objects
class IncompatibleCallbackError(PrecomputeError)The callback pattern is not supported by the precompute system.
Example: dynamic dispatch (getattr), closures, exec/eval, etc.
NoComputeBoundaryError Objects
class NoComputeBoundaryError(PrecomputeError)The callback contains dask operations but no compute boundaries.
We can’t tell what the user wants computed. Without a .compute(), client.compute(...), or similar, the
analyzer has no way to know which dask arrays to extract hints for.
analyze_callback
def analyze_callback(
callback: Callable, registered_arrays: Dict[str, Any]
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]Analyze the callback’s source and return (reduction_hints, dask_arrays).
Strict: every condition that makes the callback unanalyzable raises its own :class:PrecomputeError
subclass directly (there is no leniency left in the precompute path to defer an error to). dask_arrays
carries the walker’s boundary expressions (e.g. (arr*arr).sum()) so
:func:deisa.dask.branch._analyze_branch can walk their graphs without a second AST pass.
This is the two-tuple half of :func:analyze_callback_full, kept as its own function because that pair is the
long-standing contract of every reduction-only call site. A callback that also requests a bridge-side PCA still
analyzes correctly through this wrapper – the PCA request is validated and its refusals raised – but its branch
metadata is only visible through the full call.
:param callback:The user callback to analyze. Never invoked.:param registered_arrays:{array_name: stub}for every array the callback is registered on.
analyze_callback_full
def analyze_callback_full(
callback: Callable, registered_arrays: Dict[str, Any]
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]], List[Dict[str, Any]]]The full analysis: (reduction_hints, dask_arrays, pca_requests).
The third element is the list of recorded MergeablePCA(...).fit(<registered array>) requests, one dict per
request with array / array_name / request / lineno. It is empty for every reduction-only callback,
so the two-tuple :func:analyze_callback is the exact same function minus this argument.
A PCA request has no dask graph: the local decomposition runs on the BRIDGE over the rank’s own numpy chunk, so
there is nothing for a graph walker to find. It is therefore returned separately rather than smuggled into
dask_arrays, where every entry is contractually a dask array.
:param callback:The user callback to analyze. Never invoked.:param registered_arrays:{array_name: stub}for every array the callback is registered on.
dask.bridge
Bridge Objects
class Bridge(IBridge)__init__
def __init__(comm: ICommunicator, arrays_metadata: Dict[str, Dict], *args,
**kwargs)Initializes an instance of the class, setting up communication, metadata validation, client connection (for id=0), workers initialization, and handshake configuration for the bridge.
:param comm:An ICommunicator facilitating communication between processes. Must provide Get_rank(), Get_size(), gather(), bcast(), barrier(), Split(color, key), and Free() — the same API as an MPI communicator.:param arrays_metadata:Dictionary containing metadata for arrays. eg:arrays_metadata = { ’temperature’: { ‘global_shape’: [20, 20], ‘chunk_shape’: [10, 10], ‘chunk_position’: [0, 0], } ‘pressure’: { ‘global_shape’: [20, 20], ‘chunk_shape’: [10, 10], ‘chunk_position’: [0, 0], } }
:type arrays_metadata: Dict[str, Dict]:param args:Additional positional arguments for the initialization.:param kwargs:Additional keyword arguments for the initialization. Can include configuration parameters like timeout used during client setup.
chunk_position may be omitted per array: it is then derived ONCE, here, from the communicator topology
(see _resolve_chunk_positions). Distributions that cannot be derived raise instead of being guessed.
__del__
def __del__()Clean up resources before destruction.
Never runs a blocking collective at interpreter shutdown
(sys.is_finalizing()): close() synchronizes all bridges with an
MPI barrier, and at shutdown the peer ranks may already be torn down,
so that barrier would hang forever – a hang is worse than an
exception.
Never runs it from an event-loop callback either. The collector runs
__del__ on whichever thread dropped the last reference, and for a
leaked bridge that is a distributed “IO loop” thread. close()
synchronizes with the world (MPI barrier) and with the scheduler
(Handshake.set_bridges_done -> Client.sync()), and Client.sync()
blocks the calling thread until the client’s loop runs the callback: from
inside that very loop the callback can never run, so the thread sits out the
whole sync() timeout (60 s measured) and, with an in-process cluster whose
scheduler, workers and client share that loop, freezes the whole cluster.
During normal (non-shutdown) garbage collection from any other thread the
full close() still runs.
close
def close(timestep: int) -> NoneClose the bridge: synchronize bridges, free sub-comms, shut down client.
:param timestep:The current timestep associated with the closure action.:type timestep:int:return:None
send
def send(array_name: str, chunk: np.ndarray, timestep: Union[int, np.ndarray],
*args, **kwargs)Distribute a data chunk to a Dask workers.
Scatters the chunk to the next worker (round-robin), then gathers all chunks metadata to bridge rank 0, which updates the Dask scheduler.
For single-bridge arrays (sub_comm_size == 1), skips the gather entirely and updates the scheduler directly.
:param array_name:The name of the data array being sent as a string.:param chunk:A numpy ndarray containing the data chunk to be sent to the workers.:param timestep:The current timestep associated to the sent data chunk.:param args:Additional positional arguments if required by the method implementation.:param kwargs:Additional keyword arguments for optional configurations. Supported kwargs: update_workers (bool), filter_workers (callable).:return:None
get
def get(key: str,
timestep: Optional[int] = None,
default: Any = None) -> Optional[Union[Deque, Any]]Retrieve an element associated with a specific key and optional timestep from a feedback queue. If a queue for the key does not exist, it initializes the queue for the specified key.
:param key:The unique identifier for the feedback queue.:type key:str:param timestep:An optional specific timestep to look for. If None, returns the entire deque.:type timestep:Optional[int]:param default:The default value to return if the specified timestep is not found.:type default:Any:return:The element associated with the specified timestep if found, the entire deque if no timestep is specified, or the default value if the timestep is not found.:rtype:Optional[Union[Deque, Any]]
dask.communicator
resolve_comm
def resolve_comm(comm,
cart_coord_dims=1,
use_mpi_if_available=True,
*args,
**kwargs) -> ICommunicatorhandle 3 cases to resolve comm:
- if comm is None: use_mpi_if_available or no MPI
- if comm is an MPI Comm: use it