`pynenc.builder`¶

Module Contents¶

Classes¶

PynencBuilder

A builder pattern implementation for creating and configuring Pynenc applications.

API¶

class pynenc.builder.PynencBuilder[source]¶

A builder pattern implementation for creating and configuring Pynenc applications.

This builder simplifies the configuration process by providing intuitive method chaining to set up various components of a Pynenc application, including Redis connections, runners, serializers, and performance tuning parameters.

Example:

# Create a Pynenc application with Redis and MultiThreadRunner
pynenc_app = (
    PynencBuilder()
    .serializer("pickle")
    .redis(url="redis://localhost:6379/14")
    .multi_thread_runner(min_threads=1, max_threads=4)
    .logging_level("info")
    .build()
)

Initialization

Initialize a new PynencBuilder with an empty configuration dictionary.

_VALID_LOG_LEVELS¶: [‘debug’, ‘info’, ‘warning’, ‘error’, ‘critical’]

_SERIALIZER_MAP¶: None

_MEMORY_ARG_CACHE¶: ‘MemArgCache’

_REDIS_ARG_CACHE¶: ‘RedisArgCache’

_DISABLED_ARG_CACHE¶: ‘DisabledArgCache’

_VALID_CONCURRENCY_MODES¶: None

redis(url: str, db: Optional[int] = None) → pynenc.builder.PynencBuilder[source]¶

Configure Redis components for the Pynenc application.

This sets up all Redis-related components (orchestrator, broker, state backend, and argument cache) to use Redis as their backend.

Parameters:

url (str) – The Redis URL to connect to.
db (Optional[int]) – The Redis database number to use. If provided, it will be appended to the URL.

Returns:

The builder instance for method chaining.

memory() → pynenc.builder.PynencBuilder[source]¶

Configure in-memory components for the Pynenc application.

This sets up all components (orchestrator, broker, state backend, and argument cache) to use in-memory backends. This is primarily for testing and development purposes.

Note: In-memory components are only compatible with certain runners.

Returns:: The builder instance for method chaining.

arg_cache(mode: Literal[redis, memory, disabled] = 'redis', min_size_to_cache: int = 1024, local_cache_size: int = 1024) → pynenc.builder.PynencBuilder[source]¶

Configure argument caching behavior.

Parameters:

mode – “redis”: Use Redis for argument caching (requires redis() to be called) “memory”: Use in-memory argument caching (for testing/development) “disabled”: Disable argument caching completely
min_size_to_cache (int) – Minimum string length (in characters) required to cache an argument. Arguments smaller than this size will be passed directly. Default is 1024 characters (roughly 1KB).
local_cache_size (int) – Maximum number of items to cache locally. Default is 1024.

Returns:

The builder instance for method chaining.

Raises:

ValueError – If “redis” mode is selected without prior redis() configuration

multi_thread_runner(min_threads: int = 1, max_threads: int = 1, enforce_max_processes: bool = False) → pynenc.builder.PynencBuilder[source]¶

Configure the MultiThreadRunner for concurrent task execution.

The MultiThreadRunner uses threads to execute tasks concurrently within the same process, providing efficient parallel execution with shared memory.

Parameters:

min_threads (int) – The minimum number of threads to keep in the thread pool.
max_threads (int) – The maximum number of threads allowed in the thread pool.
enforce_max_processes (bool) – If True, enforces the maximum number of processes that can run concurrently.

Returns:

The builder instance for method chaining.

persistent_process_runner(num_processes: int = 0) → pynenc.builder.PynencBuilder[source]¶

Configure the PersistentProcessRunner for concurrent task execution.

The PersistentProcessRunner maintains a pool of persistent processes for task execution, providing true parallel execution across multiple CPU cores, with isolated memory spaces.

Parameters:: num_processes (int) – The number of processes to create in the process pool. Default 0 will use the number of CPU cores.
Returns:: The builder instance for method chaining.

thread_runner(min_threads: int = 1, max_threads: int = 0) → pynenc.builder.PynencBuilder[source]¶

Configure the ThreadRunner for task execution.

The ThreadRunner uses a thread pool to execute tasks concurrently within the same process, providing efficient parallel execution with shared memory.

Parameters:

min_threads (int) – The minimum number of threads to keep in the thread pool.
max_threads (int) – The maximum number of threads allowed in the thread pool. Default 0 will use the number of CPU cores.

Returns:

The builder instance for method chaining.

process_runner() → pynenc.builder.PynencBuilder[source]¶

Configure the ProcessRunner for task execution.

The ProcessRunner creates a new process for each task execution, providing isolated execution context for each task.

Returns:: The builder instance for method chaining.

dummy_runner() → pynenc.builder.PynencBuilder[source]¶

Configure the DummyRunner for task execution.

The DummyRunner executes tasks in the main thread of the application. This is useful for testing and debugging purposes.

Returns:: The builder instance for method chaining.

dev_mode(force_sync_tasks: bool = True) → pynenc.builder.PynencBuilder[source]¶

Enable development mode for easier debugging.

In development mode, tasks can be forced to run synchronously, making debugging and testing easier.

Parameters:: force_sync_tasks (bool) – If True, forces all tasks to run synchronously in the same process.
Returns:: The builder instance for method chaining.

logging_level(level: str) → pynenc.builder.PynencBuilder[source]¶

Set the logging level for the application.

Parameters:: level (str) – The logging level to use (“debug”, “info”, “warning”, “error”, “critical”).
Returns:: The builder instance for method chaining.
Raises:: ValueError if an invalid logging level is provided

runner_tuning(runner_loop_sleep_time_sec: float = 0.01, invocation_wait_results_sleep_time_sec: float = 0.01, min_parallel_slots: int = 1) → pynenc.builder.PynencBuilder[source]¶

Configure runner performance tuning parameters.

These parameters control various aspects of the runner behavior, allowing for fine-tuning of performance characteristics.

Parameters:

runner_loop_sleep_time_sec (float) – Sleep time between runner loop iterations in seconds.
invocation_wait_results_sleep_time_sec (float) – Sleep time when waiting for invocation results in seconds.
min_parallel_slots (int) – Minimum number of parallel execution slots for tasks.

Returns:

The builder instance for method chaining.

task_control(cycle_control: bool = False, blocking_control: bool = False, queue_timeout_sec: float = 0.1) → pynenc.builder.PynencBuilder[source]¶

Configure task control parameters.

These parameters control various aspects of task dependency management and execution control.

Parameters:

cycle_control (bool) – Whether to enable cycle control for task dependencies.
blocking_control (bool) – Whether to enable blocking control for concurrent tasks.
queue_timeout_sec (float) – Timeout for queue operations in seconds.

Returns:

The builder instance for method chaining.

serializer(serializer: str) → pynenc.builder.PynencBuilder[source]¶

Configure the serializer for task arguments and results.

The serializer is responsible for converting Python objects to and from a serialized format for storage and transmission.

Parameters:: serializer (str) – The serializer to use. Can be a shortname (“json”, “pickle”) or the full class name (e.g., “JsonSerializer”, “PickleSerializer”).
Returns:: The builder instance for method chaining.
Raises:: ValueError if an invalid serializer name is provided

concurrency_control(running_concurrency: Optional[Union[str, pynenc.conf.config_task.ConcurrencyControlType]] = None, registration_concurrency: Optional[Union[str, pynenc.conf.config_task.ConcurrencyControlType]] = None) → pynenc.builder.PynencBuilder[source]¶

Configure concurrency control default behaviors for all tasks. A task can override these settings by specifying the concurrency control mode in the task decorator.

Concurrency control determines how tasks are scheduled and executed when multiple instances of the same task are invoked concurrently.

Parameters:

running_concurrency (Optional[Union[str, ConcurrencyControlType]]) – Controls the concurrency behavior of tasks at runtime. Can be a string (“DISABLED”, “TASK”, “ARGUMENTS”, “KEYS”) or a ConcurrencyControlType enum value.
registration_concurrency (Optional[Union[str, ConcurrencyControlType]]) – Controls the concurrency behavior of tasks at registration time. Can be a string (“DISABLED”, “TASK”, “ARGUMENTS”, “KEYS”) or a ConcurrencyControlType enum value.

Returns:

The builder instance for method chaining.

max_pending_seconds(seconds: float) → pynenc.builder.PynencBuilder[source]¶

Set the maximum time a task can remain in PENDING state.

Parameters:: seconds (float) – Maximum time in seconds a task can remain in PENDING state before it expires.
Returns:: The builder instance for method chaining.

argument_print_mode(mode: Union[str, pynenc.conf.config_pynenc.ArgumentPrintMode], truncate_length: int = 32) → pynenc.builder.PynencBuilder[source]¶

Configure how task arguments are printed in logs.

Parameters:

mode (Union[str, ArgumentPrintMode]) – The print mode to use. Can be a string (“HIDDEN”, “KEYS”, “FULL”, “TRUNCATED”) or an ArgumentPrintMode enum value.
truncate_length (int) – Maximum length for printed argument values when using TRUNCATED mode. Default is 32.

Returns:

The builder instance for method chaining.

hide_arguments() → pynenc.builder.PynencBuilder[source]¶

Configure logs to hide all task arguments.

Sets print_arguments=False, resulting in “

Returns:: The builder instance for method chaining.

show_argument_keys() → pynenc.builder.PynencBuilder[source]¶

Configure logs to show only argument names.

Results in “args(key1, key2)” in logs.

Returns:: The builder instance for method chaining.

show_full_arguments() → pynenc.builder.PynencBuilder[source]¶

Configure logs to show complete argument values without truncation.

Results in “args(key1=value1, key2=value2)” in logs.

Returns:: The builder instance for method chaining.

show_truncated_arguments(truncate_length: int = 32) → pynenc.builder.PynencBuilder[source]¶

Configure logs to show truncated argument values.

Results in “args(key1=trunc_value1, key2=trunc_value2)” in logs, with truncation based on the specified length.

Parameters:: truncate_length (int) – Maximum length for printed argument values. Must be greater than 0. Default is 32.
Returns:: The builder instance for method chaining.

custom_config(**kwargs: Any) → pynenc.builder.PynencBuilder[source]¶

Add arbitrary configuration values.

This method allows adding any custom configuration values that are not covered by the specialized methods.

Parameters:: kwargs (Any) – Custom configuration values to add to the configuration.
Returns:: The builder instance for method chaining.

_validate_memory_compatibility() → None[source]¶

Validate that the selected runner is compatible with memory components if they are being used.

Raises:: ValueError if memory components are used with an incompatible runner

build() → pynenc.Pynenc[source]¶

Build and return a configured Pynenc instance.

This method creates a new Pynenc instance using the configuration values that have been set through the builder methods.

Returns:: A configured Pynenc instance ready for use.
Raises:: ValueError if the configuration is invalid

pynenc.builder¶

Module Contents¶

Classes¶

API¶

`pynenc.builder`¶