Configuration System

Reference for all Pynenc configuration options, sources, and resolution order.

Configuration Sources

Configuration values resolve from these sources in priority order (highest first):

1. Direct assignment in the config instance

2. Environment variables (PYNENC__FIELD_NAME or PYNENC__CLASS__FIELD_NAME)

3. Configuration file specified by PYNENC__FILEPATH environment variable

4. Configuration file path passed to Pynenc(config_filepath=...) (YAML, TOML, or JSON)

5. pyproject.toml under [tool.pynenc]

6. Default values defined in the ConfigField

Environment Variables

Two naming conventions are supported:

# Default for all config classes
PYNENC__FIELD_NAME="value"

# Specific to a config class
PYNENC__CONFIGPYNENC__FIELD_NAME="value"

The specific form takes precedence over the default form.

Configuration Files

pyproject.toml

[tool.pynenc]
app_id = "my_application"
orchestrator_cls = "RedisOrchestrator"
broker_cls = "RedisBroker"
state_backend_cls = "RedisStateBackend"
runner_cls = "MultiThreadRunner"
serializer_cls = "JsonPickleSerializer"
queues = ["default", "payments", "reports"]
priority_rules = [
  { task_id = "billing.*", priority = 50.0 },
  { task_id = "reports.*", priority = -10.0 },
]

[tool.pynenc.orchestrator]
max_pending_seconds = 300

[tool.pynenc.runner]
min_threads = 2
max_threads = 8
queues = ["payments", "reports"]

[tool.pynenc.task]
running_concurrency = "task"

YAML

app_id: my_application
orchestrator_cls: RedisOrchestrator

orchestrator:
  max_pending_seconds: 300

runner:
  min_threads: 4
  max_threads: 16
  queues:
    - payments
    - reports

Load with:

from pynenc import Pynenc
app = Pynenc(config_filepath="/path/to/pynenc.yaml")

JSON

{
  "app_id": "my_application",
  "orchestrator_cls": "RedisOrchestrator"
}

ConfigPynenc Fields

Main application configuration (pynenc.conf.config_pynenc.ConfigPynenc).

Field	Type	Default	Description
app_id	str	"pynenc"	Application identifier
orchestrator_cls	str	"MemOrchestrator"	Orchestrator implementation class name
trigger_cls	str	"MemTrigger"	Trigger implementation class name
broker_cls	str	"MemBroker"	Broker implementation class name
state_backend_cls	str	"MemStateBackend"	State backend implementation class name
serializer_cls	str	"JsonPickleSerializer"	Serializer implementation class name
client_data_store_cls	str	"MemClientDataStore"	Client data store implementation class name
runner_cls	str	"DummyRunner"	Runner implementation class name
trigger_task_modules	set	set()	Modules imported at runner startup for trigger-backed tasks
dev_mode_force_sync_tasks	bool	False	Execute tasks synchronously in calling thread
logging_level	str	"info"	Logging level (debug, info, warning, error, critical)
print_arguments	bool	True	Print task arguments in log messages
truncate_arguments_length	int	32	Maximum printed argument length
argument_print_mode	str	"TRUNCATED"	Argument display mode: FULL, KEYS, TRUNCATED, HIDDEN
cached_status_time	float	0.1	Invocation status cache TTL (seconds)
compact_log_context	bool	True	Truncate IDs in log context for readability
log_use_colors	bool	True	Emit ANSI colour codes in text log output
log_stream	str	"stderr"	Log output stream: "stderr" or "stdout"
log_format	str	"text"	Log format: "text" (human-readable) or "json" (structured, one object per line)
atomic_service_interval_minutes	float	5.0	Cycle interval for atomic recovery services
atomic_service_spread_margin_minutes	float	1.0	Safety margin for time-slot allocation
atomic_service_check_interval_minutes	float	0.5	Runner check interval for atomic services
atomic_service_execution_retention_minutes	float	60.0	Retention window for atomic-service execution records (minutes)
atomic_service_execution_max_records	int	1000	Capacity cap for atomic-service execution records; oldest are dropped first
recover_pending_invocations_cron	str	"*/5 * * * *"	Cron expression for pending invocation recovery
max_pending_seconds	float	5.0	Maximum time an invocation can remain PENDING
recover_running_invocations_cron	str	"*/15 * * * *"	Cron expression for running invocation recovery
runner_considered_dead_after_minutes	float	10.0	Heartbeat timeout before runner is considered dead

ConfigTrigger Fields

Trigger configuration (pynenc.conf.config_trigger.ConfigTrigger).

Field	Type	Default	Description
scheduler_interval_seconds	int	60	Minimum interval for checking time-based triggers
enable_scheduler	bool	True	Enable or disable the time-based scheduler entirely
max_events_batch_size	int	100	Maximum number of events processed per event-loop iteration
event_retention_days	int	7	Maximum age, in days, of stored emitted events before auto-purge removes them
event_auto_purge_enabled	bool	True	When True, the trigger run loop applies the retention and capacity limits to events and trigger runs on every tick
event_max_records	int	0	Soft cap on the total number of stored events (0 disables capacity-based purging; only age-based purging applies)
trigger_run_max_records	int	0	Soft cap on the total number of stored trigger runs (0 disables capacity-based purging; only age-based purging applies)

trigger_task_modules belongs to the main app configuration because the runner uses it during startup. Normal task modules are loaded lazily when an invocation first reaches a runner, but trigger-backed tasks need to be known before any invocation exists: their conditions must already be registered when the app-level atomic service decides whether to create that first invocation.

Add every module that declares @app.task(triggers=...). Pynenc imports those modules when the runner starts and continues to lazy-load task modules that are not listed.

[tool.pynenc]
trigger_cls = "SQLiteTrigger"
trigger_task_modules = ["tasks"]
scheduler_interval_seconds = 60

ConfigBroker Fields

Broker configuration (pynenc.conf.config_broker.ConfigBroker).

Field	Type	Default	Description
queue_timeout_sec	float	0.1	Broker-specific blocking timeout for polling transports
queues	tuple	("default",)	Declared broker queues. default is added automatically when omitted
priority_rules	tuple	()	Task-id wildcard rules that override task priority
warn_on_queue_mismatch	bool	True	Log a warning when routing or consuming queues not declared in broker.queues
raise_on_queue_mismatch	bool	False	Raise ConfigError instead of allowing queue mismatches when strict mode is needed

Priority rules are structured values with task_id and priority. task_id uses shell-style wildcards such as billing.*. Matching rules override the task priority; the highest priority wins when several rules match. When no rule matches, the task’s concrete priority is used.

[tool.pynenc]
queues = ["default", "payments", "reports"]
priority_rules = [
  { task_id = "billing.*", priority = 50.0 },
  { task_id = "reports.*", priority = -10.0 },
]

Priority is applied within one queue. The broker receives one queue per dequeue request; if a runner consumes several queues, queue_selection_strategy decides which queue the runner asks next. round_robin is the default and advances after each successful dequeue, random shuffles queue attempts for each retrieval, and ordered always starts from the configured queue order. With ordered, later queues can starve if earlier queues keep receiving work.

Queue mismatches are flexible by default. A task can route to a queue that is not currently listed in broker.queues, and a runner can consume an explicitly named undeclared queue. Set raise_on_queue_mismatch = true for strict deployments.

ConfigTask Fields

Per-task configuration (pynenc.conf.config_task.ConfigTask). Configurable globally or per-task.

Field	Type	Default	Description
parallel_batch_size	int	100	Batch size for task.parallelize() routing
retry_for	tuple	(RetryError,)	Exception types that trigger a retry
max_retries	int	0	Maximum retry attempts (0 = no retries)
running_concurrency	str	"DISABLED"	Runtime concurrency control: DISABLED, TASK, ARGUMENTS, KEYS
registration_concurrency	str	"DISABLED"	Registration concurrency control: DISABLED, TASK, ARGUMENTS, KEYS
key_arguments	tuple	()	Arguments used for KEYS concurrency checks
on_diff_non_key_args_raise	bool	False	Raise error when non-key arguments differ in concurrency check
call_result_cache	bool	False	Cache results by call arguments
disable_cache_args	tuple	()	Arguments to exclude from cache key
is_workflow_task	bool	False	Internal marker used by @app.workflow to define workflow roots
reroute_on_concurrency_control	bool	False	Reroute blocked tasks instead of marking final
queue	str	"default"	Broker queue used to route invocations of this task
priority	float	0.0	Task-level priority; matching broker rules override this concrete value

Important

is_workflow_task is documented because it is the internal switch that makes a task define a workflow root or sub-workflow root. It enables workflow-defining invocations and allows root-only APIs such as task.wf.root.execute_task(...), task.wf.root.uuid(), task.wf.root.random(), and task.wf.root.utc_now().

Prefer @app.workflow instead of setting is_workflow_task through environment variables, YAML, TOML, or direct config values. External configuration can mark a task as a workflow at runtime, but it bypasses the clearer public API and the WorkflowTask type returned by the decorator. Use that escape hatch only for internal migration or controlled experiments.

Per-Task Configuration

Override settings for specific tasks using environment variables or config files:

Environment variables:

# Global task setting
PYNENC__CONFIGTASK__MAX_RETRIES="3"

# Task-specific (module#task separator)
PYNENC__CONFIGTASK__MYMODULE#MY_TASK__MAX_RETRIES="5"

Note

Use # (not __) to separate the module name from the task name in environment variables.

Configuration files:

task:
  max_retries: 3
  mymodule.my_task:
    max_retries: 5

Task decorator:

@app.task(
    max_retries=5,
    running_concurrency="task",
    queue="payments",
    priority=75.0,
)
def charge_card(payment_id: str) -> str:
    return payment_id

Task queue names are validated as portable queue names when configured. Queue alignment with broker.queues is checked by the broker when routing or consuming, according to warn_on_queue_mismatch and raise_on_queue_mismatch.

Task and broker-rule priorities use finite floats from -100.0 through 100.0. Higher values run first within the same queue, and equal priorities remain FIFO. Task priority is always concrete and defaults to 0.0. Matching broker priority_rules override the task value; the highest matching rule wins. Non-finite and out-of-range values are rejected before routing. Backends with a narrower native priority model may normalize this range and must document the resulting ordering precision.

ConfigRunner Fields

Base runner configuration (pynenc.conf.config_runner.ConfigRunner).

Field	Type	Default	Description
invocation_wait_results_sleep_time_sec	float	0.1	Sleep time between result polling checks
runner_loop_sleep_time_sec	float	0.1	Sleep time between runner loop iterations
min_parallel_slots	int	1	Minimum parallel execution slots
queues	tuple	()	Queues consumed by this runner. Empty means current broker queues
queue_selection_strategy	str	"round_robin"	Queue selection across multiple consumed queues: round_robin, random, or ordered

Runner queue selection uses the same configuration and environment-variable override mechanisms as other runner fields. Leave queues empty to consume the current broker queues, or set explicit queue names for a dedicated worker.

The broker dequeues only one queue at a time. When queues resolves to multiple queues, queue_selection_strategy controls the order in which the runner asks for them. Prefer round_robin for general workers. Use ordered only when the first configured queue should dominate; it can starve later queues if the first queues are never empty.

For a runner-only environment override, use the class-qualified config name:

PYNENC__CONFIGRUNNER__QUEUES=payments,reports pynenc runner start

Runners may also consume queues that are no longer declared in broker config. The broker applies the configured mismatch warning or error policy when those queues are consumed, and Pynmon marks them as not configured; this keeps old queues drainable after deployments.

ThreadRunner Configuration

Field	Type	Default	Description
min_threads	int	1	Minimum thread pool size
max_threads	int	0	Maximum thread pool size (0 = CPU count)

MultiThreadRunner Configuration

Field	Type	Default	Description
min_threads	int	1	Threads per child process
max_threads	int	1	Threads per child process
min_processes	int	1	Minimum worker processes
max_processes	int	0	Maximum worker processes (0 = CPU count)
idle_timeout_process_sec	int	4	Seconds idle before a process is terminated
enforce_max_processes	bool	True	Always maintain max_processes regardless of load

PersistentProcessRunner Configuration

Field	Type	Default	Description
num_processes	int	CPU count	Number of persistent worker processes

Plugin Configuration

Backend plugins add their own configuration sections. For example:

Redis Plugin

[tool.pynenc.redis]
redis_host = "localhost"
redis_port = 6379
redis_db = 0

MongoDB Plugin

[tool.pynenc.mongodb]
connection_string = "mongodb://localhost:27017"
database_name = "pynenc"

Hierarchical Resolution

Pynenc supports hierarchical configuration classes with inheritance. The most specific (child) configuration takes precedence:

[tool.pynenc]
test_field = "default"

[tool.pynenc.child]
test_field = "child_override"

See Architecture for how configuration fits within the architecture. See PynencBuilder Reference for programmatic configuration with PynencBuilder.