job.scheduler

Job scheduler for parallel execution of interdependent jobs.

Module Contents

Classes

Scheduler

Handle parallel execution of interdependent jobs.

Attributes

JobProviderCallback

logger

DEFAULT_JOB_MAX_DURATION

job.scheduler.JobProviderCallback
job.scheduler.logger
job.scheduler.DEFAULT_JOB_MAX_DURATION
class job.scheduler.Scheduler(job_provider: JobProviderCallback, collect: CollectCallback | None = None, queues: dict[str, int] | None = None, tokens: int = 1, job_timeout: int = DEFAULT_JOB_MAX_DURATION)

Handle parallel execution of interdependent jobs.

property is_finished: bool

Check if all jobs have been executed (internal).

Returns:

True if complete

safe_collect(job: e3.job.Job) → bool

Protect call to collect.

This ensures for job such as JobProcess that there are no calls to Run during collect. Main goal is to avoid leak of handles from collect to a process spawned by a Job. On Unixes consequences of such leak is more a security concern than an operational one. On Windows, this can lead easily to file locking issues and thus might cause crashes.

Parameters:

job – job to collect

safe_job_provider(uid: str, data: Any, predecessors: frozenset[str], notify_end: collections.abc.Callable[[str], None]) → e3.job.Job

Protect call to job_provider.

See safe_collect commment above.

Parameters:

  • uid – unique job identifier

  • data – job data

  • predecessors – set of predecessor job IDs

  • notify_end – notification function

classmethod simple_provider(job_class: type[e3.job.Job]) → JobProviderCallback

Return a simple provider based on a given Job class.

Parameters:

job_class – a subclass of Job

init_state(dag: e3.collection.dag.DAG) → None

Reinitialize the scheduler state (internal function).

Parameters:

dag – the dag representing the list of job to execute

log_state() → None

Log the current state of the scheduler (internal).

run(dag: e3.collection.dag.DAG) → None

Launch the scheduler.

Parameters:

dag – DAG of jobs to execute

push(job: e3.job.Job) → None

Push a job into a queue.

Parameters:

job – job to push

enqueue() → None

Push into the queues job that are ready (internal).

launch() → None

Launch next jobs in the queues (internal).

wait() → None

Wait for the end of an active job.