Executors are responsible for executing steps within a job run. Once a run has launched and the process for the run (the run worker) has been allocated and started, the executor assumes responsibility for execution. Executors can range from single-process serial executors all the way to managing per-step computational resources with a sophisticated control plane.
Every job has an executor. The default executor is the multi_or_in_process_executor, which by default executes each step in its own process, but can be configured to execute each step within the same process.
from dagster import multiprocess_executor, job, graph
# Providing an executor using the job decorator@job(executor_def=multiprocess_executor)defthe_job():...@graphdefthe_graph():...# Providing an executor using graph_def.to_job(...)
other_job = the_graph.to_job(executor_def=multiprocess_executor)
A default executor can be specified for all jobs and assets provided to a repository using the default_executor_def argument of @repository. All jobs that don't specify an executor will use this default executor, but if a job explicitly specifies an executor, then the default provided to the repository will not be used.
from dagster import multiprocess_executor, define_asset_job, asset, repository
@assetdefthe_asset():pass
asset_job = define_asset_job("the_job", selection="*")@jobdefop_job():...# op_job and asset_job will both use the default_executor_def,# since neither define their own executor.@repository(default_executor_def=multiprocess_executor)defthe_repo():return[the_asset, asset_job, op_job]
The executor system is pluggable, and it is possible to write your own executor to target a different execution substrate. This is not well-documented, and the internal APIs continue to be in flux.