Logging and Graphing

Dagger's scheduler keeps track of the important and potentially expensive actions it does, such as moving data between workers or executing thunks, and tracks how much time and memory allocations these operations consume. Saving this information somewhere accessible is disabled by default, but it's quite easy to turn it on:

ctx = Context()
log = Dagger.LocalEventLog()
ctx.log_sink = log

Now anytime ctx is used as the context for a scheduler, the scheduler will log events into log. A LocalEventLog logs information in-memory, and does so on each worker. The default log object is a NoOpLog, which doesn't store events at all. The FilterLog exists to allow writing events to a user-defined location (such as a database, file, or network socket).

Once sufficient data has been accumulated into a LocalEventLog, it can be gathered to a single host via Dagger.get_logs!(log). The result is a Vector of Dagger.Timespan objects, which describe some metadata about an operation that occured and the scheduler logged. These events may be introspected directly, or may also be rendered to a DOT-format string:

logs = Dagger.get_logs!(log)
str = Dagger.show_plan(logs)

Dagger.show_plan can also be called as Dagger.show_plan(io::IO, logs) to write the graph to a file or other IO object. The string generated by this function may be passed to an external tool like Graphviz for rendering. Note that this method doesn't display input arguments to the DAG (non-Thunks); you can call Dagger.show_plan(logs, thunk), where thunk is the output Thunk of the DAG, to render argument nodes.

with new ones from future DAGs.