pynenc.runner.multi_thread_runner

Module Contents

Classes

ProcessState
ProcessStatus
MultiThreadRunner	MultiThreadRunner spawns separate processes, each running a ThreadRunner. It scales processes based on pending invocations and terminates those that remain idle.

Functions

thread_runner_process_main

MultiThreadRunner manages multiple processes, each running a ThreadRunner.

API

class pynenc.runner.multi_thread_runner.ProcessState(*args, **kwds)

Bases: enum.Enum

ACTIVE

‘active’

IDLE

‘idle’

class pynenc.runner.multi_thread_runner.ProcessStatus

Bases: typing.NamedTuple

last_update: float

None

active_count: int

None

state: pynenc.runner.multi_thread_runner.ProcessState

None

is_idle(now: float, idle_timeout: float) → bool

Return True if the process is idle and has been idle longer than idle_timeout.

pynenc.runner.multi_thread_runner.thread_runner_process_main(app: pynenc.app.Pynenc, *, runner_cache: dict, shared_status: dict[str, pynenc.runner.multi_thread_runner.ProcessStatus], process_key: str) → None

MultiThreadRunner manages multiple processes, each running a ThreadRunner.

Unlike ThreadRunner, which operates within a single process, MultiThreadRunner spawns separate processes to distribute workload. The global context dictionary (via context.py) is critical here because each process must maintain its own ThreadRunner instance in its thread-local storage. This prevents conflicts between processes and ensures that task executions within a process use the correct runner. The context check is not typically needed in ThreadRunner alone, as it operates in a single-threaded or single-process environment where the instance-level runner suffices.

class pynenc.runner.multi_thread_runner.MultiThreadRunner(app: pynenc.app.Pynenc, runner_cache: Optional[dict] = None, extra_id: Optional[str] = None)

Bases: pynenc.runner.base_runner.BaseRunner

MultiThreadRunner spawns separate processes, each running a ThreadRunner. It scales processes based on pending invocations and terminates those that remain idle.

Initialization

WAITING_FOR_RESULTS_WARNING

‘waiting_for_results called on MultiThreadRunner from within a task. This should be handled by the Th…’

processes: dict[str, multiprocessing.Process]

None

manager: multiprocessing.Manager

None

shared_status: dict[str, pynenc.runner.multi_thread_runner.ProcessStatus]

None

max_processes: int

None

runner_cache: dict

None

conf() → pynenc.conf.config_runner.ConfigMultiThreadRunner

property cache: dict

Returns the shared cache for all processes.

static mem_compatible() → bool

Indicates if the runner is compatible with in-memory components.

Returns:

False, as each Thread runs in a separate process with independent memory.

property max_parallel_slots: int

The maximum number of parallel tasks that the runner can handle.

Returns:

An integer representing the maximum number of parallel tasks, based on config or CPU count.

_on_start() → None

_spawn_thread_runner_process() → None

_on_stop() → None

_safe_remove_shared_state(key: str) → None

Safely remove a process’s shared state, handling manager shutdown cases.

Parameters:

key (str) – The process key to remove from shared state

_on_stop_runner_loop() → None

Internal method called after receiving a signal to stop the runner loop.

_cleanup_dead_processes() → None

Remove processes that are no longer alive from our tracking dictionaries.

_scale_up_processes() → None

Spawns new processes based on enforce_max_processes setting and pending tasks.

_terminate_idle_processes() → None

Terminates processes that are idle longer than the configured timeout. A process is considered idle if its shared status indicates IDLE and the time since its last update exceeds idle_timeout_process_sec. Respects enforce_max_processes if enabled.

runner_loop_iteration() → None

_waiting_for_results(running_invocation: Optional[pynenc.invocation.dist_invocation.DistributedInvocation], result_invocations: list[pynenc.invocation.dist_invocation.DistributedInvocation], runner_args: Optional[dict[str, Any]] = None) → None

Handle waiting for results when called outside a process context.

This method warns if called directly on MultiThreadRunner, as result waiting should occur within a ThreadRunner process, which uses the context-set runner. The global context ensures each process handles its own results correctly.