Types

General Types

Dagger.ThunkType
Thunk

Wraps a callable object to be run with Dagger. A Thunk is typically created through a call to delayed or its macro equivalent @par.

Constructors

delayed(f; kwargs...)(args...)
@par [option=value]... f(args...)

Examples

julia> t = delayed(sin)(π)  # creates a Thunk to be computed later
Thunk(sin, (π,))

julia> collect(t)  # computes the result and returns it to the current process
1.2246467991473532e-16

Arguments

  • f: The function to be called upon execution of the Thunk.
  • args: The arguments to be passed to the Thunk.
  • kwargs: The properties describing unique behavior of this Thunk. Details

for each property are described in the next section.

  • option=value: The same as passing kwargs to delayed.

Public Properties

  • meta::Bool=false: If true, instead of fetching cached arguments from

Chunks and passing the raw arguments to f, instead pass the Chunk. Useful for doing manual fetching or manipulation of Chunk references. Non-Chunk arguments are still passed as-is.

  • processor::Processor=OSProc() - The processor associated with f. Useful if

f is a callable struct that exists on a given processor and should be transferred appropriately.

  • scope::Dagger.AbstractScope=AnyScope() - The scope associated with f.

Useful if f is a function or callable struct that may only be transferred to, and executed within, the specified scope.

Options

  • options: A Sch.ThunkOptions struct providing the options for the Thunk.

If omitted, options can also be specified by passing key-value pairs as kwargs.

source
Dagger.EagerThunkType
EagerThunk

Returned from spawn/@spawn calls. Represents a task that is in the scheduler, potentially ready to execute, executing, or finished executing. May be fetch'd or wait'd on at any time.

source
Dagger.ChunkType
Chunk

A reference to a piece of data located on a remote worker. Chunks are typically created with Dagger.tochunk(data), and the data can then be accessed from any worker with collect(::Chunk). Chunks are serialization-safe, and use distributed refcounting (provided by MemPool.DRef) to ensure that the data referenced by a Chunk won't be GC'd, as long as a reference exists on some worker.

Each Chunk is associated with a given Dagger.Processor, which is (in a sense) the processor that "owns" or contains the data. Calling collect(::Chunk) will perform data movement and conversions defined by that processor to safely serialize the data to the calling worker.

Constructors

See tochunk.

source

Array Types

Dagger.DArrayType
DArray{T,N,F}(domain, subdomains, chunks, concat)
DArray(T, domain, subdomains, chunks, [concat=cat])

An N-dimensional distributed array of element type T, with a concatenation function of type F.

Arguments

  • T: element type
  • domain::ArrayDomain{N}: the whole ArrayDomain of the array
  • subdomains::AbstractArray{ArrayDomain{N}, N}: a DomainBlocks of the same dimensions as the array
  • chunks::AbstractArray{Union{Chunk,Thunk}, N}: an array of chunks of dimension N
  • concat::F: a function of type F. concat(x, y; dims=d) takes two chunks x and y and concatenates them along dimension d. cat is used by default.
source
Dagger.BlocksType
Blocks(xs...)

Indicates the size of an array operation, specified as xs, whose length indicates the number of dimensions in the resulting array.

source

Processor Types

Dagger.ProcessorType
Processor

An abstract type representing a processing device and associated memory, where data can be stored and operated on. Subtypes should be immutable, and instances should compare equal if they represent the same logical processing device/memory. Subtype instances should be serializable between different nodes. Subtype instances may contain a pointer to a "parent" Processor to make it easy to transfer data to/from other types of Processor at runtime.

source
Dagger.OSProcType
OSProc <: Processor

Julia CPU (OS) process, identified by Distributed pid. The logical parent of all processors on a given node, but otherwise does not participate in computations.

source

Scope Types

Context Types

Dagger.ContextType
Context(xs::Vector{OSProc}) -> Context
Context(xs::Vector{Int}) -> Context

Create a Context, by default adding each available worker.

It is also possible to create a Context from a vector of OSProc, or equivalently the underlying process ids can also be passed directly as a Vector{Int}.

Special fields include:

  • 'log_sink': A log sink object to use, if any.
  • log_file::Union{String,Nothing}: Path to logfile. If specified, at

scheduler termination, logs will be collected, combined with input thunks, and written out in DOT format to this location.

  • profile::Bool: Whether or not to perform profiling with Profile stdlib.
source

Logging Types

Dagger.LocalEventLogType
LocalEventLog

Stores events in a process-local array. Accessing the logs is all-or-nothing; if multiple consumers call get_logs!, they will get different sets of logs.

source

Scheduling Types

Dagger.Sch.SchedulerOptionsType
SchedulerOptions

Stores DAG-global options to be passed to the Dagger.Sch scheduler.

Arguments

  • single::Int=0: Force all work onto worker with specified id. 0 disables

this option.

  • proclist=nothing: Force scheduler to use one or more processors that are

instances/subtypes of a contained type. Alternatively, a function can be supplied, and the function will be called with a processor as the sole argument and should return a Bool result to indicate whether or not to use the given processor. nothing enables all default processors.

  • allow_errors::Bool=true: Allow thunks to error without affecting

non-dependent thunks.

  • checkpoint=nothing: If not nothing, uses the provided function to save

the final result of the current scheduler invocation to persistent storage, for later retrieval by restore.

  • restore=nothing: If not nothing, uses the provided function to return the

(cached) final result of the current scheduler invocation, were it to execute. If this returns a Chunk, all thunks will be skipped, and the Chunk will be returned. If nothing is returned, restoring is skipped, and the scheduler will execute as usual. If this function throws an error, restoring will be skipped, and the error will be displayed.

  • round_robin::Bool=false: Whether to schedule in round-robin mode, which

spreads load instead of the default behavior of filling processors to capacity.

source
Dagger.Sch.ThunkOptionsType
ThunkOptions

Stores Thunk-local options to be passed to the Dagger.Sch scheduler.

Arguments

  • single::Int=0: Force thunk onto worker with specified id. 0 disables this

option.

  • proclist=nothing: Force thunk to use one or more processors that are

instances/subtypes of a contained type. Alternatively, a function can be supplied, and the function will be called with a processor as the sole argument and should return a Bool result to indicate whether or not to use the given processor. nothing enables all default processors.

  • procutil::Dict{Type,Any}=Dict{Type,Any}(): Indicates the maximum expected

processor utilization for this thunk. Each keypair maps a processor type to the utilization, where the value can be a real (approx. the number of processors of this type utilized), or MaxUtilization() (utilizes all processors of this type). By default, the scheduler assumes that this thunk only uses one processor.

  • allow_errors::Bool=true: Allow this thunk to error without affecting

non-dependent thunks.

  • checkpoint=nothing: If not nothing, uses the provided function to save

the result of the thunk to persistent storage, for later retrieval by restore.

  • restore=nothing: If not nothing, uses the provided function to return the

(cached) result of this thunk, were it to execute. If this returns a Chunk, this thunk will be skipped, and its result will be set to the Chunk. If nothing is returned, restoring is skipped, and the thunk will execute as usual. If this function throws an error, restoring will be skipped, and the error will be displayed.

source

File IO Types

Warning

These APIs are currently untested and may be removed or modified.