pynenc.util.import_tools

… _import_tools:

Module Contents

Classes

TaskModules
ImportedTask

Functions

get_object_filepath	Retrieve the file path of the module where a given object was defined.
get_pynenc_instance_filepath	Get the file path of the Pynenc and Task instances.
get_frame_after	Get the first frame in the call stack after the last occurrence of a frame ending with a specified path.
get_module_from_path	Convert a relative file path to a module path format.
get_task_modules	Determine the module names for the application and task based on their file paths.
get_base_path	Extract the base path from an absolute path given a relative path segment.
import_task	Import a task based on the application and task module paths and the task name.

API

pynenc.util.import_tools.get_object_filepath(obj: object) → str | None

Retrieve the file path of the module where a given object was defined.

Parameters:

obj (object) – The object whose module file path is to be found.

Returns:

The file path of the module if found, otherwise None.

Note

This function is currently not in use in the main Pynenc workflow.

pynenc.util.import_tools.get_pynenc_instance_filepath() → str

Get the file path of the Pynenc and Task instances.

This method retrieves the file path by inspecting the call stack and finding the frame after the last occurrence of a frame that includes ‘pynenc’ or ‘app.py’.

Returns:

The file path of the Pynenc or Task instance.

Raises:

RuntimeError – If the instance file path cannot be found.

Note

This function is currently not in use in the main Pynenc workflow.

pynenc.util.import_tools.get_frame_after(ends_with: list[str]) → inspect.FrameInfo | None

Get the first frame in the call stack after the last occurrence of a frame ending with a specified path.

Parameters:

ends_with (list[str]) – The paths to search for in the call stack.

Returns:

The frame information after the last occurrence of the specified path, or None if not found.

Note

This function is currently not in use in the main Pynenc workflow.

pynenc.util.import_tools.get_module_from_path(relative_path: str) → str

Convert a relative file path to a module path format.

Parameters:

relative_path (str) – The relative file path to be converted.

Returns:

The converted module path in dot-separated format.

Note

This function is currently not in use in the main Pynenc workflow.

class pynenc.util.import_tools.TaskModules

Bases: typing.NamedTuple

app_module: str

None

task_module: str

None

pynenc.util.import_tools.get_task_modules(app_filepath: str, task_filepath: str) → pynenc.util.import_tools.TaskModules

Determine the module names for the application and task based on their file paths.

This function calculates the common path between the application and task file paths and then derives the relative module names for both.

Parameters:

• app_filepath (str) – The file path of the application.

• task_filepath (str) – The file path of the task.

Returns:

A named tuple containing the module names for the application and task.

Note

This function is currently not in use in the main Pynenc workflow.

pynenc.util.import_tools.get_base_path(absolute_path: str, relative_path: str) → str

Extract the base path from an absolute path given a relative path segment.

Parameters:

• absolute_path (str) – The absolute path from which the base path is to be extracted.

• relative_path (str) – The relative path segment used for extraction.

Returns:

The base path extracted from the absolute path.

Note

This function is currently not in use in the main Pynenc workflow.

class pynenc.util.import_tools.ImportedTask

Bases: typing.NamedTuple

callable: Callable

None

instance_filepath: str

None

pynenc.util.import_tools.import_task(app_filepath: str, app_module: str, task_module: str, task_name: str) → pynenc.util.import_tools.ImportedTask

Import a task based on the application and task module paths and the task name.

This function handles the dynamic import of a task from its module path, considering the relative paths of the application and task modules.

Parameters:

• app_filepath (str) – The file path of the application.

• app_module (str) – The module path of the application.

• task_module (str) – The module path of the task.

• task_name (str) – The name of the task to be imported.

Returns:

A named tuple containing the callable task and its instance file path.

Raises:

ImportError – If the task cannot be imported from the specified file path.

Note

This function is currently not in use in the main Pynenc workflow.

import_tools.py

Overview

The import_tools.py module was developed to address a specific challenge in task execution within the Pynenc framework. This challenge arises when a task is defined in a Python module that is executed as a standalone script. Such a module, when executed directly (via python script.py or python -m module), has its __name__ attribute set to "__main__". This scenario presents a limitation in standard task creation practices, where func.__module__ is used. The module name being "__main__" disrupts task instantiation and serialization, as the task identifier __main__.task_name in the initiator script does not match the distributed environment, where __main__ refers to the worker itself.

Approach

import_tools.py employs an alternative strategy to address this limitation. It utilizes the relative path from the file where the Pynenc application is instantiated to the file where the task is defined. This method ensures accurate location and retrieval of the task file, independent of local environment differences (filepath may differe from local environment to Docker images, however relative path between app and task should be the same).

Current Status

Currently, import_tools.py is not in use due to an unforeseen side effect of its approach. Importing tasks using relative paths leads to the creation of parallel task instances in the system, separate from the standard instances. This results in loss of context information and challenges in maintaining task consistency and reliability. To preserve simplicity and effectiveness in task management, and to avoid such complications, the current implementation does not support importing tasks from modules that can run as scripts.

Note

This documentation outlines the design decisions and evolution of the import_tools.py module within the Pynenc framework.