Functions

General Functions

Dagger.delayedFunction
delayed(f; kwargs...)(args...)

Creates a Thunk object which can be executed later, which will call f with args. kwargs controls various properties of the resulting Thunk.

source
Dagger.spawnFunction
spawn(f, args...; kwargs...) -> EagerThunk

Spawns a task with f as the function and args as the arguments, returning an EagerThunk. Uses a scheduler running in the background to execute code.

Note that kwargs are passed to the Thunk constructor, and are documented in its docstring.

source
Dagger.tochunkFunction
tochunk(x, proc; persist=false, cache=false) -> Chunk

Create a chunk from sequential object x which resides on proc.

source
Dagger.domainFunction
domain(x::T)

Returns metadata about x. This metadata will be in the domain field of a Chunk object when an object of type T is created as the result of evaluating a Thunk.

source
Dagger.computeFunction
compute(ctx::Context, d::Thunk; options=nothing) -> Chunk

Compute a Thunk - creates the DAG, assigns ranks to nodes for tie breaking and runs the scheduler with the specified options. Returns a Chunk which references the result.

source
compute(ctx::Context, x::DArray; persist=true, options=nothing)

A DArray object may contain a thunk in it, in which case we first turn it into a Thunk and then compute it.

source
Dagger.dependentsFunction
dependents(node::Thunk) -> Dict{Union{Thunk,Chunk}, Set{Thunk}}

Find the set of direct dependents for each task.

source
Dagger.noffspringFunction
noffspring(dpents::Dict{Union{Thunk,Chunk}, Set{Thunk}}) -> Dict{Thunk, Int}

Recursively find the number of tasks dependent on each task in the DAG. Takes a Dict as returned by dependents.

source
Dagger.orderFunction
order(node::Thunk, ndeps) -> Dict{Thunk,Int}

Given a root node of the DAG, calculates a total order for tie-breaking.

  • Root node gets score 1,
  • rest of the nodes are explored in DFS fashion but chunks of each node are explored in order of noffspring, i.e. total number of tasks depending on the result of the said node.

Args:

source

Table Functions

Missing docstring.

Missing docstring for tabletype. Check Documenter's build log for details.

Missing docstring.

Missing docstring for tabletype!. Check Documenter's build log for details.

Missing docstring.

Missing docstring for trim. Check Documenter's build log for details.

Missing docstring.

Missing docstring for trim!. Check Documenter's build log for details.

Missing docstring.

Missing docstring for map. Check Documenter's build log for details.

Missing docstring.

Missing docstring for filter. Check Documenter's build log for details.

Missing docstring.

Missing docstring for reduce. Check Documenter's build log for details.

Missing docstring.

Missing docstring for groupby. Check Documenter's build log for details.

Missing docstring.

Missing docstring for leftjoin. Check Documenter's build log for details.

Missing docstring.

Missing docstring for innerjoin. Check Documenter's build log for details.

Missing docstring.

Missing docstring for keys. Check Documenter's build log for details.

Missing docstring.

Missing docstring for getindex. Check Documenter's build log for details.

Array Functions

Dagger.alignfirstFunction
alignfirst(a) -> ArrayDomain

Make a subdomain a standalone domain.

Example

julia> alignfirst(ArrayDomain(11:25, 21:100))
ArrayDomain((1:15), (1:80))
source
Base.viewFunction
view(c::DArray, d)

A view of a DArray chunk returns a DArray of Thunks.

source

Processor Functions

Dagger.execute!Function
execute!(proc::Processor, f, args...) -> Any

Executes the function f with arguments args on processor proc. This function can be overloaded by Processor subtypes to allow executing function calls differently than normal Julia.

source
Dagger.iscompatibleFunction
iscompatible(proc::Processor, opts, f, Targs...) -> Bool

Indicates whether proc can execute f over Targs given opts. Processor subtypes should overload this function to return true if and only if it is essentially guaranteed that f(::Targs...) is supported. Additionally, iscompatible_func and iscompatible_arg can be overriden to determine compatibility of f and Targs individually. The default implementation returns false.

source
Dagger.default_enabledFunction
default_enabled(proc::Processor) -> Bool

Returns whether processor proc is enabled by default (opt-out). Processor subtypes can override this function to make themselves opt-in (default returns false).

source
Dagger.get_processorsFunction
get_processors(proc::Processor) -> Vector{T} where T<:Processor

Returns the full list of processors contained in proc, if any. Processor subtypes should overload this function if they can contain sub-processors. The default method will return a Vector containing proc itself.

source
Dagger.get_parentFunction
get_parent(proc::Processor) -> Processor

Returns the parent processor for proc. The ultimate parent processor is an OSProc. Processor subtypes should overload this to return their most direct parent.

source
Dagger.moveFunction
move(from_proc::Processor, to_proc::Processor, x)

Moves and/or converts x such that it's available and suitable for usage on the to_proc processor. This function can be overloaded by Processor subtypes to transport arguments and convert them to an appropriate form before being used for exection. Subtypes of Processor wishing to implement efficient data movement should provide implementations where x::Chunk.

source
Dagger.capacityFunction
capacity(proc::Processor=OSProc()) -> Int

Returns the total processing capacity of proc.

source

Shard Functions

Dagger.@shard Dagger.shard

Context Functions

Dagger.addprocs!Function
addprocs!(ctx::Context, xs)

Add new workers xs to ctx.

Workers will typically be assigned new tasks in the next scheduling iteration if scheduling is ongoing.

Workers can be either Processors or the underlying process IDs as Integers.

source
Dagger.rmprocs!Function
rmprocs!(ctx::Context, xs)

Remove the specified workers xs from ctx.

Workers will typically finish all their assigned tasks if scheduling is ongoing but will not be assigned new tasks after removal.

Workers can be either Processors or the underlying process IDs as Integers.

source

Logging Functions

Dagger.get_logs!Function
get_logs!(::LocalEventLog, raw=false; only_local=false) -> Union{Vector{Timespan},Vector{Event}}

Get the logs from each process' local event log, clearing it in the process. Set raw to true to get potentially unmatched Events; the default is to return only matched events as Timespans. If only_local is set true, only process-local logs will be fetched; the default is to fetch logs from all processes.

source

Thunk Execution Environment Functions

These functions are used within the function called by a Thunk.

Dynamic Scheduler Control Functions

These functions query and control the scheduler remotely.

Base.fetchFunction

Waits on a thunk to complete, and fetches its result.

source

File IO Functions

Warning

These APIs are currently untested and may be removed or modified.

Dagger.saveFunction
save(io::IO, val)

Save a value into the IO buffer. In the case of arrays and sparse matrices, this will save it in a memory-mappable way.

load(io::IO, t::Type, domain) will load the object given its domain

source
save(ctx, chunk::Union{Chunk, Thunk}, file_path::AbsractString)

Save a chunk to a file at file_path.

source
save(ctx, chunk, file_path)

Special case distmem writing - write to disk on the process with the chunk.

source
Dagger.loadFunction
load(ctx::Context, file_path)

Load an Union{Chunk, Thunk} from a file.

source
load(ctx::Context, ::Type{Chunk}, fpath, io)

Load a Chunk object from a file, the file path is required for creating a FileReader object

source

Macros API

Dagger.@parMacro
@par [opts] f(args...) -> Thunk

Convenience macro to call Dagger.delayed on f with arguments args. May also be called with a series of assignments like so:

x = @par begin
    a = f(1,2)
    b = g(a,3)
    h(a,b)
end

x will hold the Thunk representing h(a,b); additionally, a and b will be defined in the same local scope and will be equally accessible for later calls.

Options to the Thunk can be set as opts with namedtuple syntax, e.g. single=1. Multiple options may be provided, and will be applied to all generated thunks.

source
Dagger.@spawnMacro
@spawn [opts] f(args...) -> Thunk

Convenience macro like Dagger.@par, but eagerly executed from the moment it's called (equivalent to spawn).

See the docs for @par for more information and usage examples.

source