Covalent

Main Covalent public functionality.

Classes:

DepsBash([commands])	Shell commands to run before an electron
DepsCall([func, args, kwargs, …])	Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electron’s execution environment
DepsPip([packages, reqs_path])	PyPI packages to be installed before executing an electron
Lepton([language, library_name, …])	A generalization of an Electron to languages other than Python.
TransportableObject(obj)	A function is converted to a transportable object by serializing it using cloudpickle and then whenever executing it, the transportable object is deserialized.
status	alias of covalent.RESULT_STATUS

Functions:

cancel(dispatch_id[, dispatcher])	Cancel a running dispatch.
dispatch(orig_lattice[, dispatcher_addr])	Wrapping the dispatching functionality to allow input passing and server address specification.
dispatch_sync(lattice[, dispatcher_addr])	Wrapping the synchronous dispatching functionality to allow input passing and server address specification.
electron([_func, backend, executor, files, …])	Electron decorator to be called upon a function.
get_config([entries])	Return a configuration setting.
get_result(dispatch_id[, wait, dispatcher_addr])	Get the results of a dispatch from a file.
lattice([_func, backend, executor, …])	Lattice decorator to be called upon a function.
reload_config()	Reload the configuration from the TOML file.
set_config(new_config[, new_value])	Update the configuration.
sync([dispatch_id])	Synchronization call.
wait(child, parents)	Instructs Covalent that an electron should wait for some other tasks to complete before it is dispatched.

class covalent.DepsBash(commands=[])

Bases: covalent._workflow.deps.Deps

Shell commands to run before an electron

Deps class to encapsulate Bash dependencies for an electron.

The specified commands will be executed as subprocesses in the same environment as the electron.

commands

A list of bash commands to execute before the electron runs.

Methods:

from_dict(object_dict)	Rehydrate a dictionary representation
to_dict()	Return a JSON-serializable dictionary representation of self

from_dict(object_dict)

Rehydrate a dictionary representation

Parameters

object_dict – a dictionary representation returned by to_dict

Return type

DepsBash

Returns

self

Instance attributes will be overwritten.

to_dict()

Return a JSON-serializable dictionary representation of self

Return type

dict

class covalent.DepsCall(func=None, args=[], kwargs={}, *, retval_keyword='', override_reserved_retval_keys=False)

Bases: covalent._workflow.deps.Deps

Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electron’s execution environment

Deps class to encapsulate python functions to be called in the same execution environment as the electron.

func

A callable

args

args list

kwargs

kwargs dict

retval_keyword

An optional string referencing the return value of func.

If retval_keyword is specified, the return value of func will be passed during workflow execution as an argument to the electron corresponding to the parameter of the same name.

Notes

Electron parameters to be injected during execution must have default parameter values.

It is the user’s responsibility to ensure that retval_keyword is actually a parameter of the electron. Unexpected behavior may occur otherwise.

Methods:

from_dict(object_dict)	Rehydrate a dictionary representation
to_dict()	Return a JSON-serializable dictionary representation of self

from_dict(object_dict)

Rehydrate a dictionary representation

Parameters

object_dict – a dictionary representation returned by to_dict

Return type

DepsCall

Returns

self

Instance attributes will be overwritten.

to_dict()

Return a JSON-serializable dictionary representation of self

Return type

dict

class covalent.DepsPip(packages=[], reqs_path='')

Bases: covalent._workflow.deps.Deps

PyPI packages to be installed before executing an electron

A specification of Pip packages to be installed

packages

A list of PyPI packages to install

reqs_path

Path to requirements.txt (overrides packages)

These packages are installed in an electron’s execution environment just before the electron is run.

Methods:

from_dict(object_dict)	Rehydrate a dictionary representation
to_dict()	Return a JSON-serializable dictionary representation of self

from_dict(object_dict)

Rehydrate a dictionary representation

Parameters

object_dict – a dictionary representation returned by to_dict

Return type

DepsPip

Returns

self

Instance attributes will be overwritten.

to_dict()

Return a JSON-serializable dictionary representation of self

Return type

dict

class covalent.Lepton(language='python', *, library_name='', function_name='', argtypes=[], command='', named_outputs=[], display_name='', executor='dask', files=[], deps_bash=[], deps_pip=None, call_before=[], call_after=[])

Bases: covalent._workflow.electron.Electron

A generalization of an Electron to languages other than Python.

Leptons inherit from Electrons, overloading the function attribute with a wrapper function. Users specify the foreign function’s signature as well as its location by providing a library and entrypoint. When one of the executors invokes the task’s function, the foreign function is called by way of the wrapper function defined here. If compilation scripts are required, these must be separately copied to the backend.

language

Language in which the task specification is written.

library_name

Name of the library or module which specifies the function.

function_name

Name of the foreign function.

argtypes

List of tuples specifying data types and input/output properties.

executor

Alternative executor object to be used for lepton execution. If not passed, the dask

executor is used by default

files

An optional list of FileTransfer objects which copy files to/from remote or local filesystems.

Attributes:

INPUT
INPUT_OUTPUT
OUTPUT

Methods:

wrap_task()

Return a lepton wrapper function.

INPUT = 0

INPUT_OUTPUT = 2

OUTPUT = 1

wrap_task()

Return a lepton wrapper function.

Return type

Callable

class covalent.TransportableObject(obj)

Bases: object

A function is converted to a transportable object by serializing it using cloudpickle and then whenever executing it, the transportable object is deserialized. The object will also contain additional info like the python version used to serialize it.

_object

The serialized object.

python_version

The python version used on the client’s machine.

Methods:

deserialize(data)	Deserialize the transportable object.
deserialize_dict(collection)	Recursively deserializes a dict of TransportableObjects.
deserialize_from_json(json_string)	Reconstruct a transportable object from JSON
deserialize_list(collection)	Recursively deserializes a list of TransportableObjects.
from_dict(object_dict)	Rehydrate a dictionary representation
get_deserialized()	Get the deserialized transportable object.
get_serialized()	Get the serialized transportable object.
make_transportable(obj)	rtype TransportableObject
serialize()	Serialize the transportable object.
serialize_to_json()	Serialize the transportable object to JSON.
to_dict()	Return a JSON-serializable dictionary representation of self

Attributes:

json

static deserialize(data)

Deserialize the transportable object.

Parameters

data (bytes) – Cloudpickled function.

Returns

The deserialized transportable object.

Return type

object

static deserialize_dict(collection)

Recursively deserializes a dict of TransportableObjects. More precisely, collection is a dict, each of whose entries is assumed to be either a TransportableObject, a list, or dict`

Return type

dict

static deserialize_from_json(json_string)

Reconstruct a transportable object from JSON

Parameters

json_string (str) – A JSON string representation of a TransportableObject

Return type

str

Returns

A TransportableObject instance

static deserialize_list(collection)

Recursively deserializes a list of TransportableObjects. More precisely, collection is a list, each of whose entries is assumed to be either a TransportableObject, a list, or dict`

Return type

list

static from_dict(object_dict)

Rehydrate a dictionary representation

Parameters

object_dict – a dictionary representation returned by to_dict

Return type

TransportableObject

Returns

A TransportableObject represented by object_dict

get_deserialized()

Get the deserialized transportable object.

Parameters

None –

Returns

The deserialized object/callable function.

Return type

function

get_serialized()

Get the serialized transportable object.

Parameters

None –

Returns

The serialized transportable object.

Return type

object

property json

static make_transportable(obj)

Return type

TransportableObject

serialize()

Serialize the transportable object.

Parameters

None –

Returns

The serialized object alongwith the python version.

Return type

pickled_object

serialize_to_json()

Serialize the transportable object to JSON.

Parameters

None –

Return type

str

Returns

A JSON string representation of the transportable object

to_dict()

Return a JSON-serializable dictionary representation of self

Return type

dict

covalent.cancel(dispatch_id, dispatcher='localhost:48008')

Cancel a running dispatch.

Parameters

• dispatch_id (str) – The dispatch id of the dispatch to be cancelled.

• dispatcher (str) – The address to the dispatcher in the form of hostname:port, e.g. “localhost:8080”.

Return type

str

Returns

None

covalent.dispatch(orig_lattice, dispatcher_addr=None)

Wrapping the dispatching functionality to allow input passing and server address specification.

Afterwards, send the lattice to the dispatcher server and return the assigned dispatch id.

Parameters

• orig_lattice (Lattice) – The lattice/workflow to send to the dispatcher server.

• dispatcher_addr (Optional[str]) – The address of the dispatcher server. If None then then defaults to the address set in Covalent’s config.

Return type

Callable

Returns

Wrapper function which takes the inputs of the workflow as arguments

covalent.dispatch_sync(lattice, dispatcher_addr=None)

Wrapping the synchronous dispatching functionality to allow input passing and server address specification.

Afterwards, sends the lattice to the dispatcher server and return the result of the executed workflow.

Parameters

• orig_lattice – The lattice/workflow to send to the dispatcher server.

• dispatcher_addr (Optional[str]) – The address of the dispatcher server. If None then then defaults to the address set in Covalent’s config.

Return type

Callable

Returns

Wrapper function which takes the inputs of the workflow as arguments

covalent.electron(_func=None, *, backend=None, executor=None, files=[], deps_bash=None, deps_pip=None, call_before=[], call_after=[])

Electron decorator to be called upon a function. Returns the wrapper function with the same functionality as _func.

Parameters

_func (Optional[Callable]) – function to be decorated

Keyword Arguments

• backend – DEPRECATED: Same as executor.

• executor – Alternative executor object to be used by the electron execution. If not passed, the dask executor is used by default.

• deps_bash – An optional DepsBash object specifying a list of shell commands to run before _func

• deps_pip – An optional DepsPip object specifying a list of PyPI packages to install before running _func

• call_before – An optional list of DepsCall objects specifying python functions to invoke before the electron

• call_after – An optional list of DepsCall objects specifying python functions to invoke after the electron

• files – An optional list of FileTransfer objects which copy files to/from remote or local filesystems.

Returns

Electron object inside which the decorated function exists.

Return type

Electron

covalent.get_config(entries=None)

Return a configuration setting.

Invocation with no arguments returns the full configuration description; with a list of arguments returns a dictionary of configuration settings; with a string key name returns the corresponding value for a single setting.

Parameters

entries (Union[str, List, None]) – A string or list of strings specifying key names.

Returns

A dictionary or string describing the corresponding configuration

settings.

Return type

config

covalent.get_result(dispatch_id, wait=False, dispatcher_addr=None)

Get the results of a dispatch from a file.

Parameters

• dispatch_id (str) – The dispatch id of the result.

• wait (bool) – Controls how long the method waits for the server to return a result. If False, the method will not wait and will return the current status of the workflow. If True, the method will wait for the result to finish and keep retrying for sys.maxsize.

• dispatcher_addr (Optional[str]) – Dispatcher server address, if None then defaults to the address set in Covalent’s config.

Return type

Result

Returns

The result from the file.

covalent.lattice(_func=None, *, backend=None, executor=None, workflow_executor=None, deps_bash=None, deps_pip=None, call_before=[], call_after=[])

Lattice decorator to be called upon a function. Returns a new Lattice <covalent._workflow.lattice.Lattice> object.

Parameters

_func (Optional[Callable]) – function to be decorated

Keyword Arguments

• backend – DEPRECATED: Same as executor.

• executor – Alternative executor object to be used in the execution of each node. If not passed, the local executor is used by default.

• workflow_executor – Executor for postprocessing the workflow. Defaults to the built-in dask executor or the local executor depending on whether Covalent is started with the –no-cluster option.

• deps_bash – An optional DepsBash object specifying a list of shell commands to run before _func

• deps_pip – An optional DepsPip object specifying a list of PyPI packages to install before running _func

• call_before – An optional list of DepsCall objects specifying python functions to invoke before the electron

• call_after – An optional list of DepsCall objects specifying python functions to invoke after the electron

Returns

Lattice object inside which the decorated function exists.

Return type

Lattice

covalent.reload_config()

Reload the configuration from the TOML file.

Parameters

None –

Return type

None

Returns

None

covalent.set_config(new_config, new_value=None)

Update the configuration.

Users may pass a dictionary of new settings, or a string key with a value to set a single configuration setting.

Parameters

• new_config (Union[Dict, str]) – The new configuration dictionary, or a string key name.

• new_value (Optional[Any]) – A new configuration value, if the first argument is a string.

Return type

None

Returns

None

covalent.status

alias of covalent.RESULT_STATUS Attributes:

CANCELLED
COMPLETED
FAILED
NEW_OBJECT
PENDING_POSTPROCESSING
POSTPROCESSING
POSTPROCESSING_FAILED
RUNNING

covalent.sync(dispatch_id=None)

Synchronization call. Returns when one or more dispatches have completed.

Parameters

dispatch_id (Union[str, List[str], None]) – One or more dispatch IDs to wait for before returning.

Return type

None

Returns

None

covalent.wait(child, parents)

Instructs Covalent that an electron should wait for some other tasks to complete before it is dispatched.

Parameters

• child – the dependent electron

• parents – Electron(s) which must complete before waiting_electron starts

Returns

waiting_electron

Useful when execution of an electron relies on a side-effect from another one.