Task Helpers#
Dependencies#
Generic dependencies for an electron
- class covalent._workflow.deps.Deps(apply_fn=None, apply_args=[], apply_kwargs={}, *, retval_keyword='')[source]#
Generic dependency class used in specifying any kind of dependency for an electron.
- apply_fn#
function to be executed in the backend environment
- apply_args#
list of arguments to be applied in the backend environment
- apply_kwargs#
dictionary of keyword arguments to be applied in the backend environment
Methods:
apply()Encapsulates the exact function and args/kwargs to be executed in the backend environment.
- apply()[source]#
Encapsulates the exact function and args/kwargs to be executed in the backend environment.
- Parameters
None –
- Return type
Tuple[TransportableObject,TransportableObject,TransportableObject,str]- Returns
A tuple of transportable objects containing the function and optional args/kwargs
Examples#
Add a Bash dependency to an electron
Add a callable dependency to an electron
Add a Pip dependency to an electron
Bash Dependencies#
Shell commands to run before an electron execution.
- class covalent._workflow.depsbash.DepsBash(commands=[])[source]#
Shell commands to run before an electron
Deps class to encapsulate Bash dependencies for an electron.
The specified commands will be executed as subprocesses in the same environment as the electron.
- commands#
A list of bash commands to execute before the electron runs.
Methods:
from_dict(object_dict)Rehydrate a dictionary representation
to_dict()Return a JSON-serializable dictionary representation of self
Examples#
Add a Bash dependency to an electron
Call Dependencies#
Functions, shell commands, PyPI packages, and other dependencies to be called in an electron’s execution environment.
- class covalent._workflow.depscall.DepsCall(func=None, args=[], kwargs={}, *, retval_keyword='', override_reserved_retval_keys=False)[source]#
Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electron’s execution environment
Deps class to encapsulate python functions to be called in the same execution environment as the electron.
- func#
A callable
- args#
args list
- kwargs#
kwargs dict
- retval_keyword#
An optional string referencing the return value of func.
If retval_keyword is specified, the return value of func will be passed during workflow execution as an argument to the electron corresponding to the parameter of the same name.
Notes
Electron parameters to be injected during execution must have default parameter values.
It is the user’s responsibility to ensure that retval_keyword is actually a parameter of the electron. Unexpected behavior may occur otherwise.
Methods:
from_dict(object_dict)Rehydrate a dictionary representation
to_dict()Return a JSON-serializable dictionary representation of self
Examples#
Add a callable dependency to an electron
Pip Dependencies#
PyPI packages to be installed before executing an electron
- class covalent._workflow.depspip.DepsPip(packages=[], reqs_path='')[source]#
PyPI packages to be installed before executing an electron
A specification of Pip packages to be installed
- packages#
A list of PyPI packages to install
- reqs_path#
Path to requirements.txt (overrides packages)
These packages are installed in an electron’s execution environment just before the electron is run.
Methods:
from_dict(object_dict)Rehydrate a dictionary representation
to_dict()Return a JSON-serializable dictionary representation of self
Examples#
Add a Pip dependency to an electron
File Transfer#
File Transfer from (source) and to (destination) local or remote files prior/post electron execution. Instances are are provided to files keyword argument in an electron decorator.
- class covalent._file_transfer.file.File(filepath=None, is_remote=False, is_dir=False, include_folder=False)[source]#
File class to store components of provided URI including scheme (s3://, file://, ect.) determine if the file is remote, and acts a facade to facilitate filesystem operations.
- filepath#
File path corresponding to the file.
- is_remote#
Flag determining if file is remote (override). Default is resolved automatically from file scheme.
- is_dir#
Flag determining if file is a directory (override). Default is determined if file uri contains trailing slash.
- include_folder#
Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.
- class covalent._file_transfer.folder.Folder(filepath=None, is_remote=False, is_dir=True, include_folder=False)[source]#
Folder class to store components of provided URI including scheme (s3://, file://, ect.), determine if the file is remote, and act as facade to facilitate filesystem operations. Folder is a child of the File class which sets is_dir flag to True.
- include_folder#
Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.
- class covalent._file_transfer.file_transfer.FileTransfer(from_file=None, to_file=None, order=<Order.BEFORE: 'before'>, strategy=None)[source]#
FileTransfer object class that takes two File objects or filepaths (from, to) and a File Transfer Strategy to perform remote or local file transfer operations.
- from_file#
Filepath or File object corresponding to the source file.
- to_file#
Filepath or File object corresponding to the destination file.
- order#
Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution.
- strategy#
Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.
- covalent._file_transfer.file_transfer.TransferFromRemote(from_filepath, to_filepath=None, strategy=None, order=<Order.BEFORE: 'before'>)[source]#
Factory for creating a FileTransfer instance where from_filepath is implicitly created as a remote File Object, and the order (Order.BEFORE) is set so that this file transfer will occur prior to electron execution.
- Parameters
from_filepath (
str) – File path corresponding to remote file (source).to_filepath (
Optional[str]) – File path corresponding to local file (destination)strategy (
Optional[FileTransferStrategy]) – Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.order (
Order) – Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is BEFORE
- Return type
- Returns
FileTransfer instance with implicit Order.BEFORE enum set and from (source) file marked as remote
- covalent._file_transfer.file_transfer.TransferToRemote(to_filepath, from_filepath=None, strategy=None, order=<Order.AFTER: 'after'>)[source]#
Factory for creating a FileTransfer instance where to_filepath is implicitly created as a remote File Object, and the order (Order.AFTER) is set so that this file transfer will occur post electron execution.
- Parameters
to_filepath (
str) – File path corresponding to remote file (destination)from_filepath (
Optional[str]) – File path corresponding to local file (source).strategy (
Optional[FileTransferStrategy]) – Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.order (
Order) – Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is AFTER
- Return type
- Returns
FileTransfer instance with implicit Order.AFTER enum set and to (destination) file marked as remote
Examples#
Transfer files to and from a remote host
File Transfer Strategies#
A set of classes that support various protocols. All FileTransferStrategy classes share an interface to perform copy, download, and upload operations on two File objects (a source and a destination).
Examples#
Transfer files to and from a remote host