job.scheduler

Module Contents

Classes

Scheduler

Handle parallel execution of interdependent jobs.

Attributes

JobProviderCallback
logger
DEFAULT_JOB_MAX_DURATION

job.scheduler.JobProviderCallback

job.scheduler.logger

job.scheduler.DEFAULT_JOB_MAX_DURATION

class job.scheduler.Scheduler(job_provider: JobProviderCallback, collect: CollectCallback | None = None, queues: dict[str, int] | None = None, tokens: int = 1, job_timeout: int = DEFAULT_JOB_MAX_DURATION)

Handle parallel execution of interdependent jobs.

property is_finished: bool

Check if all jobs have been executed (internal).

Returns:

True if complete

safe_collect(job: e3.job.Job) → bool

Protect call to collect.

This ensures for job such as JobProcess that there are no calls to Run during collect. Main goal is to avoid leak of handles from collect to a process spawned by a Job. On Unixes consequences of such leak is more a security concern than an operational one. On Windows, this can lead easily to file locking issues and thus might cause crashes.

safe_job_provider(uid: str, data: Any, predecessors: FrozenSet[str], notify_end: collections.abc.Callable[[str], None]) → e3.job.Job

Protect call to job_provider.

See safe_collect commment above.

classmethod simple_provider(job_class: type[e3.job.Job]) → JobProviderCallback

Return a simple provider based on a given Job class.

Parameters:

job_class – a subclass of Job

init_state(dag: e3.collection.dag.DAG) → None

Reinitialize the scheduler state (internal function).

Parameters:

dag – the dag representing the list of job to execute

log_state() → None

Log the current state of the scheduler (internal).

run(dag: e3.collection.dag.DAG) → None

Launch the scheduler.

push(job: e3.job.Job) → None

Push a job into a queue.

enqueue() → None

Push into the queues job that are ready (internal).

launch() → None

Launch next jobs in the queues (internal).

wait() → None

Wait for the end of an active job.