pynenc.conf.config_runner

Module Contents

Classes

ConfigRunner

Specific configuration settings for any Runner in the system.

ConfigThreadRunner

Specific Configuration for the ThreadRunner

ConfigMultiThreadRunner

Specific Configuration for the MultiThreadRunner

API

class pynenc.conf.config_runner.ConfigRunner(config_values: Optional[dict[str, Any]] = None, config_filepath: Optional[str] = None)[source]

Bases: pynenc.conf.config_base.ConfigPynencBase

Specific configuration settings for any Runner in the system.

Variables:

  • invocation_wait_results_sleep_time_sec (ConfigField[float]) – Time in seconds that the runner waits before checking again for the results of a dependent invocation. This setting is used when an invocation is waiting for the results of another invocation, providing a delay between checks to avoid excessive polling.

  • runner_loop_sleep_time_sec (ConfigField[float]) – Time in seconds that the runner sleeps between iterations of its main loop. This delay is used to control the frequency at which the runner checks for new tasks to execute, and for any updates on the currently waiting tasks.

  • min_parallel_slots (ConfigField[int]) – Minimum number of parallel execution slots for tasks. This setting determines the minimum number of tasks that can run in parallel, regardless of the implementation (threads or processes). For instance, on a single-core machine, setting this to 2 would allow at least two tasks to run concurrently.

Initialization

invocation_wait_results_sleep_time_sec

‘ConfigField(…)’

runner_loop_sleep_time_sec

‘ConfigField(…)’

min_parallel_slots

‘ConfigField(…)’

class pynenc.conf.config_runner.ConfigThreadRunner(config_values: Optional[dict[str, Any]] = None, config_filepath: Optional[str] = None)[source]

Bases: pynenc.conf.config_runner.ConfigRunner

Specific Configuration for the ThreadRunner

Variables:

  • invocation_wait_results_sleep_time_sec (ConfigField[float]) – Time in seconds that the runner waits before checking again for the results of a dependent invocation. This setting is used when an invocation is waiting for the results of another invocation, providing a delay between checks to avoid excessive polling.

  • runner_loop_sleep_time_sec (ConfigField[float]) – Time in seconds that the runner sleeps between iterations of its main loop. This delay is used to control the frequency at which the runner checks for new tasks to execute, and for any updates on the currently waiting tasks.

  • min_threads (ConfigField[int]) – Minimum number of threads that the runner can handle. This setting determines the minimum number of threads that the runner can spawn to execute tasks. It is used to ensure that the runner has a minimum number of threads available to execute tasks.

  • max_threads (ConfigField[int]) – Maximum number of threads that the runner can handle. This setting determines the maximum number of threads that the runner can spawn to execute tasks. It is used to limit the number of threads created by the runner. default: 0 will set the max_threads to multiprocessing.cpu_count()

Initialization

invocation_wait_results_sleep_time_sec

‘ConfigField(…)’

runner_loop_sleep_time_sec

‘ConfigField(…)’

min_threads

‘ConfigField(…)’

max_threads

‘ConfigField(…)’

class pynenc.conf.config_runner.ConfigMultiThreadRunner(config_values: Optional[dict[str, Any]] = None, config_filepath: Optional[str] = None)[source]

Bases: pynenc.conf.config_runner.ConfigThreadRunner

Specific Configuration for the MultiThreadRunner

Variables:

  • max_threads (ConfigField[int]) – Override the default max_threads value from ConfigThreadRunner. In the ThreadRunner, the default value is multiprocessing.cpu_count(). But in the MultiThreadRunner, the number of ThreadRunner processes will depend by default on the number of CPU cores available. If the max_threads also depends on the CPU by default, may generate too much tasks in each ThreadRunner generating unnecessary overhead and memory consumption.

  • min_processes (ConfigField[int]) – Minimum number of processes that the runner can handle. This setting determines the minimum number of processes that the runner can spawn to execute ThreadRunners. It is used to ensure that the runner has a minimum number of ThreadRunner processes available to execute tasks.

  • max_processes (ConfigField[int]) – Maximum number of processes that the runner can handle. This setting determines the maximum number of processes that the runner can spawn to execute ThreadRunners. It is used to limit the number of processes created by the runner. default: 0 will set the max_processes to multiprocessing.cpu_count()

  • idle_timeout_process_sec (ConfigField[int]) – Time in seconds that the runner waits before killing an idle thread. This setting is used to control the behavior of the runner when there are no tasks to execute. If a thread is idle for longer than the specified time, the runner will kill the thread to free up resources.

  • enforce_max_processes (ConfigField[bool]) – Flag to enforce the maximum number of threads. If set to True, the runner will keep the number of threads spawned to the maximum number of threads specified in the configuration or cpu_count, regardless of the threads load. If set to False, the runner will scale the number of threads based on the thread load.

Initialization

max_threads

‘ConfigField(…)’

min_processes

‘ConfigField(…)’

max_processes

‘ConfigField(…)’

idle_timeout_process_sec

‘ConfigField(…)’

enforce_max_processes

‘ConfigField(…)’