Covalent#
Main Covalent public functionality.
Classes:
|
Shell commands to run before an electron |
|
Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electron’s execution environment |
|
PyPI packages to be installed before executing an electron |
|
A generalization of an Electron to languages other than Python. |
|
A function is converted to a transportable object by serializing it using cloudpickle and then whenever executing it, the transportable object is deserialized. |
alias of |
Functions:
|
Cancel a running dispatch. |
|
Wrapping the dispatching functionality to allow input passing and server address specification. |
|
Wrapping the synchronous dispatching functionality to allow input passing and server address specification. |
|
Electron decorator to be called upon a function. |
|
Return a configuration setting. |
|
Get the results of a dispatch from a file. |
|
Lattice decorator to be called upon a function. |
Reload the configuration from the TOML file. |
|
|
Update the configuration. |
|
Synchronization call. |
|
Instructs Covalent that an electron should wait for some other tasks to complete before it is dispatched. |
- class covalent.DepsBash(commands=[])#
Bases:
covalent._workflow.deps.Deps
Shell commands to run before an electron
Deps class to encapsulate Bash dependencies for an electron.
The specified commands will be executed as subprocesses in the same environment as the electron.
- commands#
A list of bash commands to execute before the electron runs.
Methods:
from_dict
(object_dict)Rehydrate a dictionary representation
to_dict
()Return a JSON-serializable dictionary representation of self
- class covalent.DepsCall(func=None, args=[], kwargs={}, *, retval_keyword='', override_reserved_retval_keys=False)#
Bases:
covalent._workflow.deps.Deps
Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electron’s execution environment
Deps class to encapsulate python functions to be called in the same execution environment as the electron.
- func#
A callable
- args#
args list
- kwargs#
kwargs dict
- retval_keyword#
An optional string referencing the return value of func.
If retval_keyword is specified, the return value of func will be passed during workflow execution as an argument to the electron corresponding to the parameter of the same name.
Notes
Electron parameters to be injected during execution must have default parameter values.
It is the user’s responsibility to ensure that retval_keyword is actually a parameter of the electron. Unexpected behavior may occur otherwise.
Methods:
from_dict
(object_dict)Rehydrate a dictionary representation
to_dict
()Return a JSON-serializable dictionary representation of self
- class covalent.DepsPip(packages=[], reqs_path='')#
Bases:
covalent._workflow.deps.Deps
PyPI packages to be installed before executing an electron
A specification of Pip packages to be installed
- packages#
A list of PyPI packages to install
- reqs_path#
Path to requirements.txt (overrides packages)
These packages are installed in an electron’s execution environment just before the electron is run.
Methods:
from_dict
(object_dict)Rehydrate a dictionary representation
to_dict
()Return a JSON-serializable dictionary representation of self
- class covalent.Lepton(language='python', *, library_name='', function_name='', argtypes=[], command='', named_outputs=[], display_name='', executor='dask', files=[], deps_bash=[], deps_pip=None, call_before=[], call_after=[])#
Bases:
covalent._workflow.electron.Electron
A generalization of an Electron to languages other than Python.
Leptons inherit from Electrons, overloading the function attribute with a wrapper function. Users specify the foreign function’s signature as well as its location by providing a library and entrypoint. When one of the executors invokes the task’s function, the foreign function is called by way of the wrapper function defined here. If compilation scripts are required, these must be separately copied to the backend.
- language#
Language in which the task specification is written.
- library_name#
Name of the library or module which specifies the function.
- function_name#
Name of the foreign function.
- argtypes#
List of tuples specifying data types and input/output properties.
- executor#
Alternative executor object to be used for lepton execution. If not passed, the dask
- executor is used by default
- files#
An optional list of FileTransfer objects which copy files to/from remote or local filesystems.
Attributes:
Methods:
Return a lepton wrapper function.
- INPUT = 0#
- INPUT_OUTPUT = 2#
- OUTPUT = 1#
- class covalent.TransportableObject(obj)#
Bases:
object
A function is converted to a transportable object by serializing it using cloudpickle and then whenever executing it, the transportable object is deserialized. The object will also contain additional info like the python version used to serialize it.
- _object#
The serialized object.
- python_version#
The python version used on the client’s machine.
Methods:
deserialize
(data)Deserialize the transportable object.
deserialize_dict
(collection)Recursively deserializes a dict of TransportableObjects.
deserialize_from_json
(json_string)Reconstruct a transportable object from JSON
deserialize_list
(collection)Recursively deserializes a list of TransportableObjects.
from_dict
(object_dict)Rehydrate a dictionary representation
Get the deserialized transportable object.
Get the serialized transportable object.
make_transportable
(obj)- rtype
Serialize the transportable object.
Serialize the transportable object to JSON.
to_dict
()Return a JSON-serializable dictionary representation of self
Attributes:
- static deserialize(data)[source]#
Deserialize the transportable object.
- Parameters
data (
bytes
) – Cloudpickled function.- Returns
The deserialized transportable object.
- Return type
object
- static deserialize_dict(collection)[source]#
Recursively deserializes a dict of TransportableObjects. More precisely, collection is a dict, each of whose entries is assumed to be either a TransportableObject, a list, or dict`
- Return type
dict
- static deserialize_from_json(json_string)[source]#
Reconstruct a transportable object from JSON
- Parameters
json_string (
str
) – A JSON string representation of a TransportableObject- Return type
str
- Returns
A TransportableObject instance
- static deserialize_list(collection)[source]#
Recursively deserializes a list of TransportableObjects. More precisely, collection is a list, each of whose entries is assumed to be either a TransportableObject, a list, or dict`
- Return type
list
- static from_dict(object_dict)[source]#
Rehydrate a dictionary representation
- Parameters
object_dict – a dictionary representation returned by to_dict
- Return type
- Returns
A TransportableObject represented by object_dict
- get_deserialized()[source]#
Get the deserialized transportable object.
- Parameters
None –
- Returns
The deserialized object/callable function.
- Return type
function
- get_serialized()[source]#
Get the serialized transportable object.
- Parameters
None –
- Returns
The serialized transportable object.
- Return type
object
- property json#
- serialize()[source]#
Serialize the transportable object.
- Parameters
None –
- Returns
The serialized object alongwith the python version.
- Return type
pickled_object
- covalent.cancel(dispatch_id, dispatcher='localhost:48008')#
Cancel a running dispatch.
- Parameters
dispatch_id (
str
) – The dispatch id of the dispatch to be cancelled.dispatcher (
str
) – The address to the dispatcher in the form of hostname:port, e.g. “localhost:8080”.
- Return type
str
- Returns
None
- covalent.dispatch(orig_lattice, dispatcher_addr=None)#
Wrapping the dispatching functionality to allow input passing and server address specification.
Afterwards, send the lattice to the dispatcher server and return the assigned dispatch id.
- Parameters
orig_lattice (
Lattice
) – The lattice/workflow to send to the dispatcher server.dispatcher_addr (
Optional
[str
]) – The address of the dispatcher server. If None then then defaults to the address set in Covalent’s config.
- Return type
Callable
- Returns
Wrapper function which takes the inputs of the workflow as arguments
- covalent.dispatch_sync(lattice, dispatcher_addr=None)#
Wrapping the synchronous dispatching functionality to allow input passing and server address specification.
Afterwards, sends the lattice to the dispatcher server and return the result of the executed workflow.
- Parameters
orig_lattice – The lattice/workflow to send to the dispatcher server.
dispatcher_addr (
Optional
[str
]) – The address of the dispatcher server. If None then then defaults to the address set in Covalent’s config.
- Return type
Callable
- Returns
Wrapper function which takes the inputs of the workflow as arguments
- covalent.electron(_func=None, *, backend=None, executor=None, files=[], deps_bash=None, deps_pip=None, call_before=[], call_after=[])#
Electron decorator to be called upon a function. Returns the wrapper function with the same functionality as _func.
- Parameters
_func (
Optional
[Callable
]) – function to be decorated- Keyword Arguments
backend – DEPRECATED: Same as executor.
executor – Alternative executor object to be used by the electron execution. If not passed, the dask executor is used by default.
deps_bash – An optional DepsBash object specifying a list of shell commands to run before _func
deps_pip – An optional DepsPip object specifying a list of PyPI packages to install before running _func
call_before – An optional list of DepsCall objects specifying python functions to invoke before the electron
call_after – An optional list of DepsCall objects specifying python functions to invoke after the electron
files – An optional list of FileTransfer objects which copy files to/from remote or local filesystems.
- Returns
Electron object inside which the decorated function exists.
- Return type
- covalent.get_config(entries=None)#
Return a configuration setting.
Invocation with no arguments returns the full configuration description; with a list of arguments returns a dictionary of configuration settings; with a string key name returns the corresponding value for a single setting.
- Parameters
entries (
Union
[str
,List
,None
]) – A string or list of strings specifying key names.- Returns
- A dictionary or string describing the corresponding configuration
settings.
- Return type
config
- covalent.get_result(dispatch_id, wait=False, dispatcher_addr=None)#
Get the results of a dispatch from a file.
- Parameters
dispatch_id (
str
) – The dispatch id of the result.wait (
bool
) – Controls how long the method waits for the server to return a result. If False, the method will not wait and will return the current status of the workflow. If True, the method will wait for the result to finish and keep retrying for sys.maxsize.dispatcher_addr (
Optional
[str
]) – Dispatcher server address, if None then defaults to the address set in Covalent’s config.
- Return type
- Returns
The result from the file.
- covalent.lattice(_func=None, *, backend=None, executor=None, workflow_executor=None, deps_bash=None, deps_pip=None, call_before=[], call_after=[])#
Lattice decorator to be called upon a function. Returns a new Lattice <covalent._workflow.lattice.Lattice> object.
- Parameters
_func (
Optional
[Callable
]) – function to be decorated- Keyword Arguments
backend – DEPRECATED: Same as executor.
executor – Alternative executor object to be used in the execution of each node. If not passed, the local executor is used by default.
workflow_executor – Executor for postprocessing the workflow. Defaults to the built-in dask executor or the local executor depending on whether Covalent is started with the –no-cluster option.
deps_bash – An optional DepsBash object specifying a list of shell commands to run before _func
deps_pip – An optional DepsPip object specifying a list of PyPI packages to install before running _func
call_before – An optional list of DepsCall objects specifying python functions to invoke before the electron
call_after – An optional list of DepsCall objects specifying python functions to invoke after the electron
- Returns
Lattice object inside which the decorated function exists.
- Return type
- covalent.reload_config()#
Reload the configuration from the TOML file.
- Parameters
None –
- Return type
None
- Returns
None
- covalent.set_config(new_config, new_value=None)#
Update the configuration.
Users may pass a dictionary of new settings, or a string key with a value to set a single configuration setting.
- Parameters
new_config (
Union
[Dict
,str
]) – The new configuration dictionary, or a string key name.new_value (
Optional
[Any
]) – A new configuration value, if the first argument is a string.
- Return type
None
- Returns
None
- covalent.status#
alias of
covalent.RESULT_STATUS
Attributes:CANCELLED
COMPLETED
FAILED
NEW_OBJECT
PENDING_POSTPROCESSING
POSTPROCESSING
POSTPROCESSING_FAILED
RUNNING
- covalent.sync(dispatch_id=None)#
Synchronization call. Returns when one or more dispatches have completed.
- Parameters
dispatch_id (
Union
[str
,List
[str
],None
]) – One or more dispatch IDs to wait for before returning.- Return type
None
- Returns
None
- covalent.wait(child, parents)#
Instructs Covalent that an electron should wait for some other tasks to complete before it is dispatched.
- Parameters
child – the dependent electron
parents – Electron(s) which must complete before waiting_electron starts
- Returns
waiting_electron
Useful when execution of an electron relies on a side-effect from another one.