File and Data Transfer#

Basic File transfer interface#

File Transfer from (source) and to (destination) local or remote files prior/post electron execution. Instances are are provided to files keyword argument in an electron decorator.

Classes:

File([filepath, is_remote, is_dir, …])

File class to store components of provided URI including scheme (s3://, file://, ect.) determine if the file is remote, and acts a facade to facilitate filesystem operations.

FileTransfer([from_file, to_file, order, …])

FileTransfer object class that takes two File objects or filepaths (from, to) and a File Transfer Strategy to perform remote or local file transfer operations.

Folder([filepath, is_remote, is_dir, …])

Folder class to store components of provided URI including scheme (s3://, file://, ect.), determine if the file is remote, and act as facade to facilitate filesystem operations.

Order(value)

An enumeration.

Functions:

TransferFromRemote(from_filepath[, …])

Factory for creating a FileTransfer instance where from_filepath is implicitly created as a remote File Object, and the order (Order.BEFORE) is set so that this file transfer will occur prior to electron execution.

TransferToRemote(to_filepath[, …])

Factory for creating a FileTransfer instance where to_filepath is implicitly created as a remote File Object, and the order (Order.AFTER) is set so that this file transfer will occur post electron execution.

class covalent.fs.File(filepath=None, is_remote=False, is_dir=False, include_folder=False)#

Bases: object

File class to store components of provided URI including scheme (s3://, file://, ect.) determine if the file is remote, and acts a facade to facilitate filesystem operations.

filepath#

File path corresponding to the file.

is_remote#

Flag determining if file is remote (override). Default is resolved automatically from file scheme.

is_dir#

Flag determining if file is a directory (override). Default is determined if file uri contains trailing slash.

include_folder#

Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.

Attributes:

filepath

rtype

str

is_dir

is_remote

is_temp_file

mapped_strategy_type

rtype

FileTransferStrategyTypes

Methods:

get_path_obj(path)

get_temp_filepath()

get_uri(scheme, path)

rtype

str

resolve_scheme(path)

rtype

FileSchemes

touch()

property filepath: str#
Return type

str

static get_path_obj(path)#
get_temp_filepath()#
static get_uri(scheme, path)#
Return type

str

property is_dir#
property is_remote#
property is_temp_file#
property mapped_strategy_type: covalent._file_transfer.enums.FileTransferStrategyTypes#
Return type

FileTransferStrategyTypes

static resolve_scheme(path)#
Return type

FileSchemes

touch()#
class covalent.fs.FileTransfer(from_file=None, to_file=None, order=<Order.BEFORE: 'before'>, strategy=None)#

Bases: object

FileTransfer object class that takes two File objects or filepaths (from, to) and a File Transfer Strategy to perform remote or local file transfer operations.

from_file#

Filepath or File object corresponding to the source file.

to_file#

Filepath or File object corresponding to the destination file.

order#

Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution.

strategy#

Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.

Methods:

cp()

cp()#
class covalent.fs.Folder(filepath=None, is_remote=False, is_dir=True, include_folder=False)#

Bases: covalent._file_transfer.file.File

Folder class to store components of provided URI including scheme (s3://, file://, ect.), determine if the file is remote, and act as facade to facilitate filesystem operations. Folder is a child of the File class which sets is_dir flag to True.

include_folder#

Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.

class covalent.fs.Order(value)#

Bases: str, enum.Enum

An enumeration.

Attributes:

AFTER = 'after'#
BEFORE = 'before'#
covalent.fs.TransferFromRemote(from_filepath, to_filepath=None, strategy=None, order=<Order.BEFORE: 'before'>)#

Factory for creating a FileTransfer instance where from_filepath is implicitly created as a remote File Object, and the order (Order.BEFORE) is set so that this file transfer will occur prior to electron execution.

Parameters
  • from_filepath (str) – File path corresponding to remote file (source).

  • to_filepath (Optional[str]) – File path corresponding to local file (destination)

  • strategy (Optional[FileTransferStrategy]) – Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.

  • order (Order) – Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is BEFORE

Return type

FileTransfer

Returns

FileTransfer instance with implicit Order.BEFORE enum set and from (source) file marked as remote

covalent.fs.TransferToRemote(to_filepath, from_filepath=None, strategy=None, order=<Order.AFTER: 'after'>)#

Factory for creating a FileTransfer instance where to_filepath is implicitly created as a remote File Object, and the order (Order.AFTER) is set so that this file transfer will occur post electron execution.

Parameters
  • to_filepath (str) – File path corresponding to remote file (destination)

  • from_filepath (Optional[str]) – File path corresponding to local file (source).

  • strategy (Optional[FileTransferStrategy]) – Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.

  • order (Order) – Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is AFTER

Return type

FileTransfer

Returns

FileTransfer instance with implicit Order.AFTER enum set and to (destination) file marked as remote

File transfer strategies#

A set of classes with a shared interface to perform copy, download, and upload operations given two (source & destination) File objects that support various protocols.

Classes:

HTTP()

Implements Base FileTransferStrategy class to use HTTP to download files from public URLs.

Rsync([user, host, private_key_path])

Implements Base FileTransferStrategy class to use rsync to move files to and from remote or local filesystems.

S3([credentials, profile, region_name])

Implements Base FileTransferStrategy class to upload/download files from S3 Bucket.

class covalent.fs_strategies.HTTP#

Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy

Implements Base FileTransferStrategy class to use HTTP to download files from public URLs.

Methods:

cp(from_file[, to_file])

rtype

File

download(from_file[, to_file])

rtype

File

upload(from_file[, to_file])

rtype

File

cp(from_file, to_file=<covalent._file_transfer.file.File object>)#
Return type

File

download(from_file, to_file=<covalent._file_transfer.file.File object>)#
Return type

File

upload(from_file, to_file=<covalent._file_transfer.file.File object>)#
Return type

File

class covalent.fs_strategies.Rsync(user='', host='', private_key_path=None)#

Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy

Implements Base FileTransferStrategy class to use rsync to move files to and from remote or local filesystems. Rsync via ssh is used if one of the provided files is marked as remote.

user#

(optional) Determine user to specify for remote host if using rsync with ssh

host#

(optional) Determine what host to connect to if using rsync with ssh

private_key_path#

(optional) Filepath for ssh private key to use if using rsync with ssh

Methods:

cp(from_file[, to_file])

rtype

None

download(from_file[, to_file])

rtype

File

get_rsync_cmd(from_file, to_file[, …])

rtype

str

get_rsync_ssh_cmd(local_file, remote_file[, …])

rtype

str

return_subprocess_callable(cmd)

rtype

None

upload(from_file, to_file)

rtype

None

cp(from_file, to_file=<covalent._file_transfer.file.File object>)#
Return type

None

download(from_file, to_file=<covalent._file_transfer.file.File object>)#
Return type

File

get_rsync_cmd(from_file, to_file, transfer_from_remote=False)#
Return type

str

get_rsync_ssh_cmd(local_file, remote_file, transfer_from_remote=False)#
Return type

str

return_subprocess_callable(cmd)#
Return type

None

upload(from_file, to_file)#
Return type

None

class covalent.fs_strategies.S3(credentials='/home/docs/.aws/credentials', profile=None, region_name=None)#

Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy

Implements Base FileTransferStrategy class to upload/download files from S3 Bucket.

Methods:

cp(from_file[, to_file])

rtype

File

download(from_file[, to_file])

rtype

File

upload(from_file[, to_file])

rtype

File

cp(from_file, to_file=<covalent._file_transfer.file.File object>)#
Return type

File

download(from_file, to_file=<covalent._file_transfer.file.File object>)#
Return type

File

upload(from_file, to_file=<covalent._file_transfer.file.File object>)#
Return type

File