File and Data Transfer#

Basic File transfer interface#

File Transfer from (source) and to (destination) local or remote files prior/post electron execution. Instances are are provided to files keyword argument in an electron decorator.

Classes:

`File`([filepath, is_remote, is_dir, …])	File class to store components of provided URI including scheme (s3://, file://, ect.) determine if the file is remote, and acts a facade to facilitate filesystem operations.
`FileTransfer`([from_file, to_file, order, …])	FileTransfer object class that takes two File objects or filepaths (from, to) and a File Transfer Strategy to perform remote or local file transfer operations.
`Folder`([filepath, is_remote, is_dir, …])	Folder class to store components of provided URI including scheme (s3://, file://, ect.), determine if the file is remote, and act as facade to facilitate filesystem operations.
`Order`(value)	An enumeration.

Functions:

`TransferFromRemote`(from_filepath[, …])	Factory for creating a FileTransfer instance where from_filepath is implicitly created as a remote File Object, and the order (Order.BEFORE) is set so that this file transfer will occur prior to electron execution.
`TransferToRemote`(to_filepath[, …])	Factory for creating a FileTransfer instance where to_filepath is implicitly created as a remote File Object, and the order (Order.AFTER) is set so that this file transfer will occur post electron execution.

class covalent.fs.File(filepath=None, is_remote=False, is_dir=False, include_folder=False)#

Bases: object

File class to store components of provided URI including scheme (s3://, file://, ect.) determine if the file is remote, and acts a facade to facilitate filesystem operations.

filepath#: File path corresponding to the file.

is_remote#: Flag determining if file is remote (override). Default is resolved automatically from file scheme.

is_dir#: Flag determining if file is a directory (override). Default is determined if file uri contains trailing slash.

include_folder#: Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.

Attributes:

`filepath`	rtype `str`
`is_dir`
`is_remote`
`is_temp_file`
`mapped_strategy_type`	rtype `FileTransferStrategyTypes`

Methods:

`get_path_obj`(path)
`get_temp_filepath`()
`get_uri`(scheme, path)	rtype `str`
`resolve_scheme`(path)	rtype `FileSchemes`
`touch`()

property filepath: str#

Return type: str

static get_path_obj(path)#

get_temp_filepath()#

static get_uri(scheme, path)#

Return type: str

property is_dir#

property is_remote#

property is_temp_file#

property mapped_strategy_type: covalent._file_transfer.enums.FileTransferStrategyTypes#

Return type: FileTransferStrategyTypes

static resolve_scheme(path)#

Return type: FileSchemes

touch()#

class covalent.fs.FileTransfer(from_file=None, to_file=None, order=<Order.BEFORE: 'before'>, strategy=None)#

Bases: object

FileTransfer object class that takes two File objects or filepaths (from, to) and a File Transfer Strategy to perform remote or local file transfer operations.

from_file#: Filepath or File object corresponding to the source file.

to_file#: Filepath or File object corresponding to the destination file.

order#: Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution.

strategy#: Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.

Methods:

cp()

cp()#

class covalent.fs.Folder(filepath=None, is_remote=False, is_dir=True, include_folder=False)#

Bases: covalent._file_transfer.file.File

Folder class to store components of provided URI including scheme (s3://, file://, ect.), determine if the file is remote, and act as facade to facilitate filesystem operations. Folder is a child of the File class which sets is_dir flag to True.

include_folder#: Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.

class covalent.fs.Order(value)#

Bases: str, enum.Enum

An enumeration.

Attributes:

`AFTER`
`BEFORE`

AFTER = 'after'#

BEFORE = 'before'#

covalent.fs.TransferFromRemote(from_filepath, to_filepath=None, strategy=None, order=<Order.BEFORE: 'before'>)#

Factory for creating a FileTransfer instance where from_filepath is implicitly created as a remote File Object, and the order (Order.BEFORE) is set so that this file transfer will occur prior to electron execution.

Parameters

from_filepath (str) – File path corresponding to remote file (source).
to_filepath (Optional[str]) – File path corresponding to local file (destination)
strategy (Optional[FileTransferStrategy]) – Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.
order (Order) – Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is BEFORE

Return type

FileTransfer

Returns

FileTransfer instance with implicit Order.BEFORE enum set and from (source) file marked as remote

covalent.fs.TransferToRemote(to_filepath, from_filepath=None, strategy=None, order=<Order.AFTER: 'after'>)#

Factory for creating a FileTransfer instance where to_filepath is implicitly created as a remote File Object, and the order (Order.AFTER) is set so that this file transfer will occur post electron execution.

Parameters

to_filepath (str) – File path corresponding to remote file (destination)
from_filepath (Optional[str]) – File path corresponding to local file (source).
strategy (Optional[FileTransferStrategy]) – Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.
order (Order) – Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is AFTER

Return type

FileTransfer

Returns

FileTransfer instance with implicit Order.AFTER enum set and to (destination) file marked as remote

File transfer strategies#

A set of classes with a shared interface to perform copy, download, and upload operations given two (source & destination) File objects that support various protocols.

Classes:

`HTTP`()	Implements Base FileTransferStrategy class to use HTTP to download files from public URLs.
`Rsync`([user, host, private_key_path])	Implements Base FileTransferStrategy class to use rsync to move files to and from remote or local filesystems.
`S3`([credentials, profile, region_name])	Implements Base FileTransferStrategy class to upload/download files from S3 Bucket.

class covalent.fs_strategies.HTTP#

Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy

Implements Base FileTransferStrategy class to use HTTP to download files from public URLs.

Methods:

cp(from_file[, to_file])

rtype: File

download(from_file[, to_file])

rtype: File

upload(from_file[, to_file])

rtype: File

cp(from_file, to_file=<covalent._file_transfer.file.File object>)#

Return type: File

download(from_file, to_file=<covalent._file_transfer.file.File object>)#

Return type: File

upload(from_file, to_file=<covalent._file_transfer.file.File object>)#

Return type: File

class covalent.fs_strategies.Rsync(user='', host='', private_key_path=None)#

Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy

Implements Base FileTransferStrategy class to use rsync to move files to and from remote or local filesystems. Rsync via ssh is used if one of the provided files is marked as remote.

user#: (optional) Determine user to specify for remote host if using rsync with ssh

host#: (optional) Determine what host to connect to if using rsync with ssh

private_key_path#: (optional) Filepath for ssh private key to use if using rsync with ssh

Methods:

`cp`(from_file[, to_file])	rtype `None`
`download`(from_file[, to_file])	rtype `File`
`get_rsync_cmd`(from_file, to_file[, …])	rtype `str`
`get_rsync_ssh_cmd`(local_file, remote_file[, …])	rtype `str`
`return_subprocess_callable`(cmd)	rtype `None`
`upload`(from_file, to_file)	rtype `None`

cp(from_file, to_file=<covalent._file_transfer.file.File object>)#

Return type: None

download(from_file, to_file=<covalent._file_transfer.file.File object>)#

Return type: File

get_rsync_cmd(from_file, to_file, transfer_from_remote=False)#

Return type: str

get_rsync_ssh_cmd(local_file, remote_file, transfer_from_remote=False)#

Return type: str

return_subprocess_callable(cmd)#

Return type: None

upload(from_file, to_file)#

Return type: None

class covalent.fs_strategies.S3(credentials=None, profile=None, region_name=None)#

Bases: covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy

Implements Base FileTransferStrategy class to upload/download files from S3 Bucket.

Methods:

cp(from_file[, to_file])

rtype: File

download(from_file[, to_file])

Download files or the contents of folders from S3 bucket.

upload(from_file[, to_file])

Upload files or folders to S3 bucket.

cp(from_file, to_file=<covalent._file_transfer.file.File object>)#

Return type: File

download(from_file, to_file=<covalent._file_transfer.file.File object>)#

Download files or the contents of folders from S3 bucket.

Return type: File

upload(from_file, to_file=<covalent._file_transfer.file.File object>)#

Upload files or folders to S3 bucket.

Return type: File