delayed(f, options=Options())(args...; kwargs...) -> Thunk
delayed(f; options...)(args...; kwargs...) -> Thunk

Creates a Thunk object which can be executed later, which will call f with args and kwargs. options controls various properties of the resulting Thunk.

spawn(f, args...; kwargs...) -> EagerThunk

Spawns a task with f as the function and args as the arguments, returning an EagerThunk. Uses a scheduler running in the background to execute code.

Note that kwargs are passed to the Thunk constructor, and are documented in its docstring.

tochunk(x, proc::Processor, scope::AbstractScope; device=nothing, kwargs...) -> Chunk

Create a chunk from data x which resides on proc and which has scope scope.

device specifies a MemPool.StorageDevice (which is itself wrapped in a Chunk) which will be used to manage the reference contained in the Chunk generated by this function. If device is nothing (the default), the data will be inspected to determine if it's safe to serialize; if so, the default MemPool storage device will be used; if not, then a MemPool.CPURAMDevice will be used.

All other kwargs are passed directly to MemPool.poolset.

domain(x::T)

Returns metadata about x. This metadata will be in the domain field of a Chunk object when an object of type T is created as the result of evaluating a Thunk.

compute(ctx::Context, d::Thunk; options=nothing) -> Chunk

Compute a Thunk - creates the DAG, assigns ranks to nodes for tie breaking and runs the scheduler with the specified options. Returns a Chunk which references the result.

compute(ctx::Context, x::DArray; persist=true, options=nothing)

A DArray object may contain a thunk in it, in which case we first turn it into a Thunk and then compute it.

dependents(node::Thunk) -> Dict{Union{Thunk,Chunk}, Set{Thunk}}

Find the set of direct dependents for each task.

noffspring(dpents::Dict{Union{Thunk,Chunk}, Set{Thunk}}) -> Dict{Thunk, Int}

Recursively find the number of tasks dependent on each task in the DAG. Takes a Dict as returned by dependents.

order(node::Thunk, ndeps) -> Dict{Thunk,Int}

Given a root node of the DAG, calculates a total order for tie-breaking.

• Root node gets score 1,
• rest of the nodes are explored in DFS fashion but chunks of each node are explored in order of noffspring, i.e. total number of tasks depending on the result of the said node.

Args:

• node: root node
• ndeps: result of noffspring

Tree reduce

alignfirst(a) -> ArrayDomain

Make a subdomain a standalone domain.

Example

julia> alignfirst(ArrayDomain(11:25, 21:100))
ArrayDomain((1:15), (1:80))

view(c::DArray, d)

A view of a DArray chunk returns a DArray of Thunks.

execute!(proc::Processor, f, args...; kwargs...) -> Any

Executes the function f with arguments args and keyword arguments kwargs on processor proc. This function can be overloaded by Processor subtypes to allow executing function calls differently than normal Julia.

iscompatible(proc::Processor, opts, f, Targs...) -> Bool

Indicates whether proc can execute f over Targs given opts. Processor subtypes should overload this function to return true if and only if it is essentially guaranteed that f(::Targs...) is supported. Additionally, iscompatible_func and iscompatible_arg can be overriden to determine compatibility of f and Targs individually. The default implementation returns false.

default_enabled(proc::Processor) -> Bool

Returns whether processor proc is enabled by default. The default value is false, which is an opt-out of the processor from execution when not specifically requested by the user, and true implies opt-in, which causes the processor to always participate in execution when possible.

get_processors(proc::Processor) -> Set{<:Processor}

Returns the set of processors contained in proc, if any. Processor subtypes should overload this function if they can contain sub-processors. The default method will return a Set containing proc itself.

get_parent(proc::Processor) -> Processor

Returns the parent processor for proc. The ultimate parent processor is an OSProc. Processor subtypes should overload this to return their most direct parent.

move(from_proc::Processor, to_proc::Processor, x)

Moves and/or converts x such that it's available and suitable for usage on the to_proc processor. This function can be overloaded by Processor subtypes to transport arguments and convert them to an appropriate form before being used for exection. Subtypes of Processor wishing to implement efficient data movement should provide implementations where x::Chunk.

get_tls()

Gets all Dagger TLS variable as a NamedTuple.

set_tls!(tls)

Sets all Dagger TLS variables from the NamedTuple tls.

addprocs!(ctx::Context, xs)

Add new workers xs to ctx.

Workers will typically be assigned new tasks in the next scheduling iteration if scheduling is ongoing.

Workers can be either Processors or the underlying process IDs as Integers.

rmprocs!(ctx::Context, xs)

Remove the specified workers xs from ctx.

Workers will typically finish all their assigned tasks if scheduling is ongoing but will not be assigned new tasks after removal.

Workers can be either Processors or the underlying process IDs as Integers.

in_thunk()

Returns true if currently in a Thunk process, else false.

thunk_processor()

Get the current processor executing the current thunk.

Gets the scheduler handle for the currently-executing thunk.

Adds a new Thunk to the DAG.

Waits on a thunk to complete, and fetches its result.

Waits on a thunk to complete.

Executes an arbitrary function within the scheduler, returning the result.

Commands the scheduler to halt execution immediately.

Returns all Thunks IDs as a Dict, mapping a Thunk to its downstream dependents.

save(io::IO, val)

Save a value into the IO buffer. In the case of arrays and sparse matrices, this will save it in a memory-mappable way.

load(io::IO, t::Type, domain) will load the object given its domain

save(ctx, chunk::Union{Chunk, Thunk}, file_path::AbsractString)

Save a chunk to a file at file_path.

save(ctx, chunk, file_path)

Special case distmem writing - write to disk on the process with the chunk.

load(ctx::Context, file_path)

Load an Union{Chunk, Thunk} from a file.

load(ctx::Context, ::Type{Chunk}, fpath, io)

Load a Chunk object from a file, the file path is required for creating a FileReader object

@par [opts] f(args...; kwargs...) -> Thunk

Convenience macro to call Dagger.delayed on f with arguments args and keyword arguments kwargs. May also be called with a series of assignments like so:

x = @par begin
    a = f(1,2)
    b = g(a,3)
    h(a,b)
end

x will hold the Thunk representing h(a,b); additionally, a and b will be defined in the same local scope and will be equally accessible for later calls.

Options to the Thunk can be set as opts with namedtuple syntax, e.g. single=1. Multiple options may be provided, and will be applied to all generated thunks.

@spawn [opts] f(args...) -> Thunk

Convenience macro like Dagger.@par, but eagerly executed from the moment it's called (equivalent to spawn).

See the docs for @par for more information and usage examples.