Dagger Types
Dagger.AnyScope
Dagger.ArrayDomain
Dagger.Blocks
Dagger.Chunk
Dagger.Context
Dagger.DArray
Dagger.DTask
Dagger.Deps
Dagger.Events.BytesAllocd
Dagger.Events.ProcessorSaturation
Dagger.Events.WorkerSaturation
Dagger.ExactScope
Dagger.In
Dagger.InOut
Dagger.NodeScope
Dagger.OSProc
Dagger.Options
Dagger.Out
Dagger.ProcessScope
Dagger.Processor
Dagger.SchedulerOptions
Dagger.Shard
Dagger.TaintScope
Dagger.ThreadProc
Dagger.Thunk
Dagger.UnionScope
Dagger.UnitDomain
Dagger.ProcessorTypeScope
Task Types
Dagger.Thunk
— TypeThunk
Wraps a callable object to be run with Dagger. A Thunk
is typically created through a call to delayed
or its macro equivalent @par
.
Constructors
delayed(f; kwargs...)(args...)
@par [option=value]... f(args...)
Examples
julia> t = delayed(sin)(π) # creates a Thunk to be computed later
Thunk(sin, (π,))
julia> collect(t) # computes the result and returns it to the current process
1.2246467991473532e-16
Arguments
fargs
: The function and arguments to be called upon execution of theThunk
.kwargs
: The properties describing unique behavior of thisThunk
. Details
for each property are described in the next section.
option=value
: The same as passingkwargs
todelayed
.
Options
options
: AnOptions
struct providing the options for theThunk
.
If omitted, options can also be specified by passing key-value pairs as kwargs
.
Dagger.DTask
— TypeDTask
Returned from Dagger.@spawn
/Dagger.spawn
calls. Represents a task that is in the scheduler, potentially ready to execute, executing, or finished executing. May be fetch
'd or wait
'd on at any time. See Dagger.@spawn
for more details.
Task Options Types
Dagger.Options
— TypeOptions
Stores per-task options to be passed to the scheduler.
Arguments
propagates::Vector{Symbol}
: The set of option names that will be propagated by this task to tasks that it spawns.processor::Processor
: The processor associated with this task's function. Generally ignored by the scheduler.compute_scope::AbstractScope
: The execution scope of the task, which determines where the task can be scheduled and executed.scope
is another name for this option.result_scope::AbstractScope
: The data scope of the task's result, which determines where the task's result can be accessed from.single::Int=0
: (Deprecated) Force task onto worker with specified id.0
disables this option.proclist=nothing
: (Deprecated) Force task to use one or more processors that are instances/subtypes of a contained type. Alternatively, a function can be supplied, and the function will be called with a processor as the sole argument and should return aBool
result to indicate whether or not to use the given processor.nothing
enables all default processors.get_result::Bool=false
: Whether the worker should store the result directly (true
) or as aChunk
(false
)meta::Bool=false
: Whentrue
, values are notmove
d, and are passed directly asChunk
, if they are not immediate valuessyncdeps::Set{Any}
: Contains any additional tasks to synchronize withtime_util::Dict{Type,Any}
: Indicates the maximum expected time utilization for this task. Each keypair maps a processor type to the utilization, where the value can be a real (approximately the number of nanoseconds taken), orMaxUtilization()
(utilizes all processors of this type). By default, the scheduler assumes that this task only uses one processor.alloc_util::Dict{Type,UInt64}
: Indicates the maximum expected memory utilization for this task. Each keypair maps a processor type to the utilization, where the value is an integer representing approximately the maximum number of bytes allocated at any one time.occupancy::Dict{Type,Real}
: Indicates the maximum expected processor occupancy for this task. Each keypair maps a processor type to the utilization, where the value can be a real between 0 and 1 (the occupancy ratio, where 1 is full occupancy). By default, the scheduler assumes that this task has full occupancy.checkpoint=nothing
: If notnothing
, uses the provided function to save the result of the task to persistent storage, for later retrieval byrestore
.restore=nothing
: If notnothing
, uses the provided function to return the (cached) result of this task, were it to execute. If this returns aChunk
, this task will be skipped, and its result will be set to theChunk
. Ifnothing
is returned, restoring is skipped, and the task will execute as usual. If this function throws an error, restoring will be skipped, and the error will be displayed.storage::Union{Chunk,Nothing}=nothing
: If notnothing
, references aMemPool.StorageDevice
which will be passed toMemPool.poolset
internally when constructingChunk
s (such as when constructing the return value). The device must supportMemPool.CPURAMResource
. Whennothing
, usesMemPool.GLOBAL_DEVICE[]
.storage_root_tag::Any=nothing
: If notnothing
, specifies the MemPool storage leaf tag to associate with the task's result. This tag can be used by MemPool's storage devices to manipulate their behavior, such as the file name used to store data on disk."storage_leaf_tag::Union{MemPool.Tag,Nothing}=nothing
: If notnothing
, specifies the MemPool storage leaf tag to associate with the task's result. This tag can be used by MemPool's storage devices to manipulate their behavior, such as the file name used to store data on disk."storage_retain::Union{Bool,Nothing}=nothing
: The value ofretain
to pass toMemPool.poolset
when constructing the resultChunk
.nothing
defaults tofalse
.name::Union{String,Nothing}=nothing
: If notnothing
, annotates the task with a name for logging purposes.stream_input_buffer_amount::Union{Int,Nothing}=nothing
: (Streaming only) Specifies the amount of slots to allocate for the input buffer of the task. Defaults to 1.stream_output_buffer_amount::Union{Int,Nothing}=nothing
: (Streaming only) Specifies the amount of slots to allocate for the output buffer of the task. Defaults to 1.stream_buffer_type::Union{Type,Nothing}=nothing
: (Streaming only) Specifies the type of buffer to use for the input and output buffers of the task. Defaults toDagger.ProcessRingBuffer
.stream_max_evals::Union{Int,Nothing}=nothing
: (Streaming only) Specifies the maximum number of times the task will be evaluated before returning a result. Defaults to infinite evaluations.
Dagger.SchedulerOptions
— TypeSchedulerOptions
Stores DAG-global options to be passed to the Dagger.Sch scheduler.
Arguments
single::Int=0
: (Deprecated) Force all work onto worker with specified id.0
disables this option.proclist=nothing
: (Deprecated) Force scheduler to use one or more processors that are instances/subtypes of a contained type. Alternatively, a function can be supplied, and the function will be called with a processor as the sole argument and should return aBool
result to indicate whether or not to use the given processor.nothing
enables all default processors.allow_errors::Bool=false
: Allow thunks to error without affecting non-dependent thunks.checkpoint=nothing
: If notnothing
, uses the provided function to save the final result of the current scheduler invocation to persistent storage, for later retrieval byrestore
.restore=nothing
: If notnothing
, uses the provided function to return the (cached) final result of the current scheduler invocation, were it to execute. If this returns aChunk
, all thunks will be skipped, and theChunk
will be returned. Ifnothing
is returned, restoring is skipped, and the scheduler will execute as usual. If this function throws an error, restoring will be skipped, and the error will be displayed.
Data Management Types
Dagger.Chunk
— TypeChunk
A reference to a piece of data located on a remote worker. Chunk
s are typically created with Dagger.tochunk(data)
, and the data can then be accessed from any worker with collect(::Chunk)
. Chunk
s are serialization-safe, and use distributed refcounting (provided by MemPool.DRef
) to ensure that the data referenced by a Chunk
won't be GC'd, as long as a reference exists on some worker.
Each Chunk
is associated with a given Dagger.Processor
, which is (in a sense) the processor that "owns" or contains the data. Calling collect(::Chunk)
will perform data movement and conversions defined by that processor to safely serialize the data to the calling worker.
Constructors
See tochunk
.
Dagger.Shard
— TypeMaps a value to one of multiple distributed "mirror" values automatically when used as a thunk argument. Construct using @shard
or shard
.
Data Dependencies Types
Dagger.In
— TypeSpecifies a read-only dependency.
Dagger.Out
— TypeSpecifies a write-only dependency.
Dagger.InOut
— TypeSpecifies a read-write dependency.
Dagger.Deps
— TypeSpecifies one or more dependencies.
Processor Types
Dagger.Processor
— TypeProcessor
An abstract type representing a processing device and associated memory, where data can be stored and operated on. Subtypes should be immutable, and instances should compare equal if they represent the same logical processing device/memory. Subtype instances should be serializable between different nodes. Subtype instances may contain a "parent" Processor
to make it easy to transfer data to/from other types of Processor
at runtime.
Dagger.OSProc
— TypeOSProc <: Processor
Julia CPU (OS) process, identified by Distributed pid. The logical parent of all processors on a given node, but otherwise does not participate in computations.
Dagger.ThreadProc
— TypeThreadProc <: Processor
Julia CPU (OS) thread, identified by Julia thread ID.
Scope Types
Dagger.AnyScope
— TypeWidest scope that contains all processors.
Dagger.NodeScope
— TypeScoped to the same physical node.
Dagger.ProcessScope
— TypeScoped to the same OS process.
Dagger.ProcessorTypeScope
— FunctionScoped to any processor with a given supertype.
Dagger.TaintScope
— TypeTaints a scope for later evaluation.
Dagger.UnionScope
— TypeUnion of two or more scopes.
Dagger.ExactScope
— TypeScoped to a specific processor.
Context Types
Dagger.Context
— TypeContext(xs::Vector{OSProc}) -> Context
Context(xs::Vector{Int}) -> Context
Create a Context, by default adding each available worker.
It is also possible to create a Context from a vector of OSProc
, or equivalently the underlying process ids can also be passed directly as a Vector{Int}
.
Special fields include:
- 'log_sink': A log sink object to use, if any.
profile::Bool
: Whether or not to perform profiling with Profile stdlib.
Array Types
Dagger.DArray
— TypeDArray{T,N,F}(domain, subdomains, chunks, concat)
DArray(T, domain, subdomains, chunks, [concat=cat])
An N-dimensional distributed array of element type T, with a concatenation function of type F.
Arguments
T
: element typedomain::ArrayDomain{N}
: the whole ArrayDomain of the arraysubdomains::AbstractArray{ArrayDomain{N}, N}
: aDomainBlocks
of the same dimensions as the arraychunks::AbstractArray{Union{Chunk,Thunk}, N}
: an array of chunks of dimension Nconcat::F
: a function of typeF
.concat(x, y; dims=d)
takes two chunksx
andy
and concatenates them along dimensiond
.cat
is used by default.
Dagger.Blocks
— TypeBlocks(xs...)
Indicates the size of an array operation, specified as xs
, whose length indicates the number of dimensions in the resulting array.
Dagger.ArrayDomain
— TypeArrayDomain{N}
An N
-dimensional domain over an array.
Dagger.UnitDomain
— TypeUnitDomain
Default domain – has no information about the value
Logging Event Types
Dagger.Events.BytesAllocd
— TypeBytesAllocd
Tracks memory allocated for Chunk
s.
Dagger.Events.ProcessorSaturation
— TypeProcessorSaturation
Tracks the compute saturation (running tasks) per-processor.
Dagger.Events.WorkerSaturation
— TypeWorkerSaturation
Tracks the compute saturation (running tasks).