Covalent

Main Covalent public functionality.

Classes:

DepsBash([commands])	Shell commands to run before an electron
DepsCall([func, args, kwargs, …])	Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electron’s execution environment
DepsPip([packages, reqs_path])	PyPI packages to be installed before executing an electron
Lepton([language, library_name, …])	A generalization of an Electron to languages other than Python.
TransportableObject(obj)	A function is converted to a transportable object by serializing it using cloudpickle and then whenever executing it, the transportable object is deserialized.
status	alias of covalent.RESULT_STATUS

Functions:

cancel(dispatch_id[, task_ids, dispatcher_addr])	Cancel a running dispatch.
dispatch(orig_lattice[, dispatcher_addr, …])	Wrapping the dispatching functionality to allow input passing and server address specification.
dispatch_sync(lattice[, dispatcher_addr])	Wrapping the synchronous dispatching functionality to allow input passing and server address specification.
electron([_func, backend, executor, files, …])	Electron decorator to be called upon a function.
get_config([entries])	Return a configuration setting.
get_result(dispatch_id[, wait, …])	Get the results of a dispatch from the Covalent server.
lattice([_func, backend, executor, …])	Lattice decorator to be called upon a function.
redispatch(dispatch_id[, dispatcher_addr, …])	Wrapping the dispatching functionality to allow input passing and server address specification.
reload_config()	Reload the configuration from the TOML file.
set_config(new_config[, new_value])	Update the configuration.
stop_triggers(dispatch_ids[, …])	Stop observing on all triggers of all given dispatch ids registered on the Triggers server.
sync([dispatch_id])	Synchronization call.
wait(child, parents)	Instructs Covalent that an electron should wait for some other tasks to complete before it is dispatched.

class covalent.DepsBash(commands=[])

Bases: covalent._workflow.deps.Deps

Shell commands to run before an electron

Deps class to encapsulate Bash dependencies for an electron.

The specified commands will be executed as subprocesses in the same environment as the electron.

commands

A list of bash commands to execute before the electron runs.

Methods:

from_dict(object_dict)	Rehydrate a dictionary representation
to_dict()	Return a JSON-serializable dictionary representation of self

from_dict(object_dict)

Rehydrate a dictionary representation

Parameters

object_dict – a dictionary representation returned by to_dict

Return type

DepsBash

Returns

self

Instance attributes will be overwritten.

to_dict()

Return a JSON-serializable dictionary representation of self

Return type

dict

class covalent.DepsCall(func=None, args=[], kwargs={}, *, retval_keyword='', override_reserved_retval_keys=False)

Bases: covalent._workflow.deps.Deps

Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electron’s execution environment

Deps class to encapsulate python functions to be called in the same execution environment as the electron.

func

A callable

args

args list

kwargs

kwargs dict

retval_keyword

An optional string referencing the return value of func.

If retval_keyword is specified, the return value of func will be passed during workflow execution as an argument to the electron corresponding to the parameter of the same name.

Notes

Electron parameters to be injected during execution must have default parameter values.

It is the user’s responsibility to ensure that retval_keyword is actually a parameter of the electron. Unexpected behavior may occur otherwise.

Methods:

from_dict(object_dict)	Rehydrate a dictionary representation
to_dict()	Return a JSON-serializable dictionary representation of self

from_dict(object_dict)

Rehydrate a dictionary representation

Parameters

object_dict – a dictionary representation returned by to_dict

Return type

DepsCall

Returns

self

Instance attributes will be overwritten.

to_dict()

Return a JSON-serializable dictionary representation of self

Return type

dict

class covalent.DepsPip(packages=[], reqs_path='')

Bases: covalent._workflow.deps.Deps

PyPI packages to be installed before executing an electron

A specification of Pip packages to be installed

packages

A list of PyPI packages to install

reqs_path

Path to requirements.txt (overrides packages)

These packages are installed in an electron’s execution environment just before the electron is run.

Methods:

from_dict(object_dict)	Rehydrate a dictionary representation
to_dict()	Return a JSON-serializable dictionary representation of self

from_dict(object_dict)

Rehydrate a dictionary representation

Parameters

object_dict – a dictionary representation returned by to_dict

Return type

DepsPip

Returns

self

Instance attributes will be overwritten.

to_dict()

Return a JSON-serializable dictionary representation of self

Return type

dict

class covalent.Lepton(language='python', *, library_name='', function_name='', argtypes=[], command='', named_outputs=[], display_name='', executor='dask', files=[], deps_bash=[], deps_pip=None, call_before=[], call_after=[])

Bases: covalent._workflow.electron.Electron

A generalization of an Electron to languages other than Python.

Leptons inherit from Electrons, overloading the function attribute with a wrapper function. Users specify the foreign function’s signature as well as its location by providing a library and entrypoint. When one of the executors invokes the task’s function, the foreign function is called by way of the wrapper function defined here. If compilation scripts are required, these must be separately copied to the backend.

language

Language in which the task specification is written.

library_name

Name of the library or module which specifies the function.

function_name

Name of the foreign function.

argtypes

List of tuples specifying data types and input/output properties.

executor

Alternative executor object to be used for lepton execution. If not passed, the dask

executor is used by default

files

An optional list of FileTransfer objects which copy files to/from remote or local filesystems.

Attributes:

INPUT
INPUT_OUTPUT
OUTPUT

Methods:

wrap_task()

Return a lepton wrapper function.

INPUT = 0

INPUT_OUTPUT = 2

OUTPUT = 1

wrap_task()

Return a lepton wrapper function.

Return type

Callable

class covalent.TransportableObject(obj)

Bases: object

A function is converted to a transportable object by serializing it using cloudpickle and then whenever executing it, the transportable object is deserialized. The object will also contain additional info like the python version used to serialize it.

Attributes:

attrs
object_string
python_version

Methods:

deserialize(serialized, *[, header_only, …])	Deserialize the transportable object from the archived transportable object.
deserialize_dict(collection)	Recursively deserializes a dict of TransportableObjects.
deserialize_from_json(json_string)	Reconstruct a transportable object from JSON
deserialize_list(collection)	Recursively deserializes a list of TransportableObjects.
from_dict(object_dict)	Rehydrate a dictionary representation.
get_deserialized()	Get the deserialized transportable object.
get_serialized()	Get the serialized transportable object.
make_transportable(obj)	Make an object transportable.
serialize()	Serialize the transportable object to the archived transportable object.
serialize_to_json()	Serialize the transportable object to JSON.
to_dict()	Return a JSON-serializable dictionary representation of self.

property attrs

static deserialize(serialized, *, header_only=False, string_only=False)

Deserialize the transportable object from the archived transportable object.

Parameters

data – Serialized transportable object

Return type

TransportableObject

Returns

The deserialized transportable object.

static deserialize_dict(collection)

Recursively deserializes a dict of TransportableObjects. More precisely, collection is a dict, each of whose entries is assumed to be either a TransportableObject, a list, or dict`

Parameters

collection (dict) – A dictionary of TransportableObjects.

Return type

dict

Returns

A dictionary of deserialized objects.

static deserialize_from_json(json_string)

Reconstruct a transportable object from JSON

Parameters

json_string (str) – A JSON string representation of a TransportableObject.

Return type

str

Returns

A TransportableObject instance.

static deserialize_list(collection)

Recursively deserializes a list of TransportableObjects. More precisely, collection is a list, each of whose entries is assumed to be either a TransportableObject, a list, or dict`

Parameters

collection (list) – A list of TransportableObjects.

Return type

list

Returns

A list of deserialized objects.

static from_dict(object_dict)

Rehydrate a dictionary representation.

Parameters

object_dict – a dictionary representation returned by to_dict.

Return type

TransportableObject

Returns

A TransportableObject represented by object_dict.

get_deserialized()

Get the deserialized transportable object.

Note that this method is different from the deserialize method which deserializes from the archived transportable object.

Returns

The deserialized object/callable function.

Return type

function

get_serialized()

Get the serialized transportable object.

Note that this is different from the serialize method which serializes the archived transportable object.

Returns

The serialized transportable object.

Return type

object

static make_transportable(obj)

Make an object transportable.

Parameters

obj (Any) – The object to make transportable.

Return type

TransportableObject

Returns

Transportable object.

property object_string

property python_version

serialize()

Serialize the transportable object to the archived transportable object.

Return type

bytes

Returns

The serialized object along with the python version.

serialize_to_json()

Serialize the transportable object to JSON.

Return type

str

Returns

A JSON string representation of the transportable object.

to_dict()

Return a JSON-serializable dictionary representation of self.

Returns

A JSON-serializable dictionary representation of self.

Return type

dict

covalent.cancel(dispatch_id, task_ids=None, dispatcher_addr=None)

Cancel a running dispatch.

Parameters

• dispatch_id (str) – The dispatch id of the dispatch to be cancelled.

• task_ids (Optional[List[int]]) – Optional, list of task ids to cancel within the workflow

• dispatcher_addr (Optional[str]) – Dispatcher server address, if None then defaults to the address set in Covalent’s config.

Return type

str

Returns

Cancellation response

covalent.dispatch(orig_lattice, dispatcher_addr=None, disable_run=False)

Wrapping the dispatching functionality to allow input passing and server address specification.

Afterwards, send the lattice to the dispatcher server and return the assigned dispatch id.

Parameters

• orig_lattice (Lattice) – The lattice/workflow to send to the dispatcher server.

• dispatcher_addr (Optional[str]) – The address of the dispatcher server. If None then defaults to the address set in Covalent’s config.

• disable_run (bool) – Whether to disable running the workflow and rather just save it on Covalent’s server for later execution

Return type

Callable

Returns

Wrapper function which takes the inputs of the workflow as arguments

covalent.dispatch_sync(lattice, dispatcher_addr=None)

Wrapping the synchronous dispatching functionality to allow input passing and server address specification.

Afterwards, sends the lattice to the dispatcher server and return the result of the executed workflow.

Parameters

• orig_lattice – The lattice/workflow to send to the dispatcher server.

• dispatcher_addr (Optional[str]) – The address of the dispatcher server. If None then defaults to the address set in Covalent’s config.

Return type

Callable

Returns

Wrapper function which takes the inputs of the workflow as arguments.

covalent.electron(_func=None, *, backend=None, executor=None, files=[], deps_bash=None, deps_pip=None, call_before=[], call_after=[])

Electron decorator to be called upon a function. Returns the wrapper function with the same functionality as _func.

Parameters

_func (Optional[Callable]) – function to be decorated

Keyword Arguments

• backend – DEPRECATED: Same as executor.

• executor – Alternative executor object to be used by the electron execution. If not passed, the dask executor is used by default.

• deps_bash – An optional DepsBash object specifying a list of shell commands to run before _func

• deps_pip – An optional DepsPip object specifying a list of PyPI packages to install before running _func

• call_before – An optional list of DepsCall objects specifying python functions to invoke before the electron

• call_after – An optional list of DepsCall objects specifying python functions to invoke after the electron

• files – An optional list of FileTransfer objects which copy files to/from remote or local filesystems.

Returns

Electron object inside which the decorated function exists.

Return type

Electron

covalent.get_config(entries=None)

Return a configuration setting.

Invocation with no arguments returns the full configuration description; with a list of arguments returns a dictionary of configuration settings; with a string key name returns the corresponding value for a single setting.

Parameters

entries (Union[str, List, None]) – A string or list of strings specifying key names.

Returns

A dictionary or string describing the corresponding configuration

settings.

Return type

config

covalent.get_result(dispatch_id, wait=False, dispatcher_addr=None, status_only=False)

Get the results of a dispatch from the Covalent server.

Parameters

• dispatch_id (str) – The dispatch id of the result.

• wait (bool) – Controls how long the method waits for the server to return a result. If False, the method will not wait and will return the current status of the workflow. If True, the method will wait for the result to finish and keep retrying for sys.maxsize.

• dispatcher_addr (Optional[str]) – Dispatcher server address, if None then defaults to the address set in Covalent’s config.

• status_only (bool) – If true, only returns result status, not the full result object, default is False.

Return type

Result

Returns

The Result object from the Covalent server

covalent.lattice(_func=None, *, backend=None, executor=None, workflow_executor=None, deps_bash=None, deps_pip=None, call_before=[], call_after=[], triggers=None)

Lattice decorator to be called upon a function. Returns a new Lattice <covalent._workflow.lattice.Lattice> object.

Parameters

_func (Optional[Callable]) – function to be decorated

Keyword Arguments

• backend – DEPRECATED: Same as executor.

• executor – Alternative executor object to be used in the execution of each node. If not passed, the local executor is used by default.

• workflow_executor – Executor for postprocessing the workflow. Defaults to the built-in dask executor or the local executor depending on whether Covalent is started with the –no-cluster option.

• deps_bash – An optional DepsBash object specifying a list of shell commands to run before _func

• deps_pip – An optional DepsPip object specifying a list of PyPI packages to install before running _func

• call_before – An optional list of DepsCall objects specifying python functions to invoke before the electron

• call_after – An optional list of DepsCall objects specifying python functions to invoke after the electron

• triggers – Any triggers that need to be attached to this lattice, default is None

Returns

Lattice object inside which the decorated function exists.

Return type

Lattice

covalent.redispatch(dispatch_id, dispatcher_addr=None, replace_electrons=None, reuse_previous_results=False, is_pending=False)

Wrapping the dispatching functionality to allow input passing and server address specification.

Parameters

• dispatch_id (str) – The dispatch id of the workflow to re-dispatch.

• dispatcher_addr (Optional[str]) – The address of the dispatcher server. If None then then defaults to the address set in Covalent’s config.

• replace_electrons (Optional[Dict[str, Callable]]) – A dictionary of electron names and the new electron to replace them with.

• reuse_previous_results (bool) – Boolean value whether to reuse the results from the previous dispatch.

Return type

Callable

Returns

Wrapper function which takes the inputs of the workflow as arguments.

covalent.reload_config()

Reload the configuration from the TOML file.

Parameters

None –

Return type

None

Returns

None

covalent.set_config(new_config, new_value=None)

Update the configuration.

Users may pass a dictionary of new settings, or a string key with a value to set a single configuration setting.

Parameters

• new_config (Union[Dict, str]) – The new configuration dictionary, or a string key name.

• new_value (Optional[Any]) – A new configuration value, if the first argument is a string.

Return type

None

Returns

None

covalent.status

alias of covalent.RESULT_STATUS Attributes:

CANCELLED
COMPLETED
DISPATCHING_SUBLATTICE
FAILED
NEW_OBJECT
PENDING_POSTPROCESSING
PENDING_REUSE
POSTPROCESSING
POSTPROCESSING_FAILED
RUNNING
STARTING

covalent.stop_triggers(dispatch_ids, triggers_server_addr=None)

Stop observing on all triggers of all given dispatch ids registered on the Triggers server. :type dispatch_ids: Union[str, List[str]] :param dispatch_ids: Dispatch ID(s) for whose triggers are to be stopped :type triggers_server_addr: Optional[str] :param triggers_server_addr: Address of the Triggers server; configured dispatcher’s address is used as default

Return type

None

Returns

None

covalent.sync(dispatch_id=None)

Synchronization call. Returns when one or more dispatches have completed.

Parameters

dispatch_id (Union[str, List[str], None]) – One or more dispatch IDs to wait for before returning.

Return type

None

Returns

None

covalent.wait(child, parents)

Instructs Covalent that an electron should wait for some other tasks to complete before it is dispatched.

Parameters

• child – the dependent electron

• parents – Electron(s) which must complete before waiting_electron starts

Returns

waiting_electron

Useful when execution of an electron relies on a side-effect from another one.