File and Data Transfer#
Basic File transfer interface#
File Transfer from (source) and to (destination) local or remote files prior/post electron execution. Instances are are provided to files keyword argument in an electron decorator.
Classes:
|
File class to store components of provided URI including scheme (s3://, file://, ect.) determine if the file is remote, and acts a facade to facilitate filesystem operations. |
|
FileTransfer object class that takes two File objects or filepaths (from, to) and a File Transfer Strategy to perform remote or local file transfer operations. |
|
Folder class to store components of provided URI including scheme (s3://, file://, ect.), determine if the file is remote, and act as facade to facilitate filesystem operations. |
|
An enumeration. |
Functions:
|
Factory for creating a FileTransfer instance where from_filepath is implicitly created as a remote File Object, and the order (Order.BEFORE) is set so that this file transfer will occur prior to electron execution. |
|
Factory for creating a FileTransfer instance where to_filepath is implicitly created as a remote File Object, and the order (Order.AFTER) is set so that this file transfer will occur post electron execution. |
- class covalent.fs.File(filepath=None, is_remote=False, is_dir=False, include_folder=False)#
Bases:
object
File class to store components of provided URI including scheme (s3://, file://, ect.) determine if the file is remote, and acts a facade to facilitate filesystem operations.
- filepath#
File path corresponding to the file.
- is_remote#
Flag determining if file is remote (override). Default is resolved automatically from file scheme.
- is_dir#
Flag determining if file is a directory (override). Default is determined if file uri contains trailing slash.
- include_folder#
Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.
Attributes:
- rtype
str
- rtype
FileTransferStrategyTypes
Methods:
get_path_obj
(path)get_uri
(scheme, path)- rtype
str
resolve_scheme
(path)- rtype
FileSchemes
touch
()- property filepath: str#
- Return type
str
- static get_path_obj(path)#
- get_temp_filepath()#
- static get_uri(scheme, path)#
- Return type
str
- property is_dir#
- property is_remote#
- property is_temp_file#
- property mapped_strategy_type: covalent._file_transfer.enums.FileTransferStrategyTypes#
- Return type
FileTransferStrategyTypes
- static resolve_scheme(path)#
- Return type
FileSchemes
- touch()#
- class covalent.fs.FileTransfer(from_file=None, to_file=None, order=<Order.BEFORE: 'before'>, strategy=None)#
Bases:
object
FileTransfer object class that takes two File objects or filepaths (from, to) and a File Transfer Strategy to perform remote or local file transfer operations.
- from_file#
Filepath or File object corresponding to the source file.
- to_file#
Filepath or File object corresponding to the destination file.
- order#
Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution.
- strategy#
Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.
Methods:
cp
()- cp()#
- class covalent.fs.Folder(filepath=None, is_remote=False, is_dir=True, include_folder=False)#
Bases:
covalent._file_transfer.file.File
Folder class to store components of provided URI including scheme (s3://, file://, ect.), determine if the file is remote, and act as facade to facilitate filesystem operations. Folder is a child of the File class which sets is_dir flag to True.
- include_folder#
Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.
- class covalent.fs.Order(value)#
Bases:
str
,enum.Enum
An enumeration.
Attributes:
- AFTER = 'after'#
- BEFORE = 'before'#
- covalent.fs.TransferFromRemote(from_filepath, to_filepath=None, strategy=None, order=<Order.BEFORE: 'before'>)#
Factory for creating a FileTransfer instance where from_filepath is implicitly created as a remote File Object, and the order (Order.BEFORE) is set so that this file transfer will occur prior to electron execution.
- Parameters
from_filepath (
str
) – File path corresponding to remote file (source).to_filepath (
Optional
[str
]) – File path corresponding to local file (destination)strategy (
Optional
[FileTransferStrategy
]) – Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.order (
Order
) – Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is BEFORE
- Return type
- Returns
FileTransfer instance with implicit Order.BEFORE enum set and from (source) file marked as remote
- covalent.fs.TransferToRemote(to_filepath, from_filepath=None, strategy=None, order=<Order.AFTER: 'after'>)#
Factory for creating a FileTransfer instance where to_filepath is implicitly created as a remote File Object, and the order (Order.AFTER) is set so that this file transfer will occur post electron execution.
- Parameters
to_filepath (
str
) – File path corresponding to remote file (destination)from_filepath (
Optional
[str
]) – File path corresponding to local file (source).strategy (
Optional
[FileTransferStrategy
]) – Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.order (
Order
) – Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is AFTER
- Return type
- Returns
FileTransfer instance with implicit Order.AFTER enum set and to (destination) file marked as remote
File transfer strategies#
A set of classes with a shared interface to perform copy, download, and upload operations given two (source & destination) File objects that support various protocols.
Classes:
|
Implements Base FileTransferStrategy class to use HTTP to download files from public URLs. |
|
Implements Base FileTransferStrategy class to use rsync to move files to and from remote or local filesystems. |
|
Implements Base FileTransferStrategy class to upload/download files from S3 Bucket. |
- class covalent.fs_strategies.HTTP#
Bases:
covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy
Implements Base FileTransferStrategy class to use HTTP to download files from public URLs.
Methods:
- class covalent.fs_strategies.Rsync(user='', host='', private_key_path=None)#
Bases:
covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy
Implements Base FileTransferStrategy class to use rsync to move files to and from remote or local filesystems. Rsync via ssh is used if one of the provided files is marked as remote.
- user#
(optional) Determine user to specify for remote host if using rsync with ssh
- host#
(optional) Determine what host to connect to if using rsync with ssh
- private_key_path#
(optional) Filepath for ssh private key to use if using rsync with ssh
Methods:
cp
(from_file[, to_file])- rtype
None
download
(from_file[, to_file])- rtype
get_rsync_cmd
(from_file, to_file[, …])- rtype
str
get_rsync_ssh_cmd
(local_file, remote_file[, …])- rtype
str
- rtype
None
upload
(from_file, to_file)- rtype
None
- cp(from_file, to_file=<covalent._file_transfer.file.File object>)#
- Return type
None
- get_rsync_cmd(from_file, to_file, transfer_from_remote=False)#
- Return type
str
- get_rsync_ssh_cmd(local_file, remote_file, transfer_from_remote=False)#
- Return type
str
- return_subprocess_callable(cmd)#
- Return type
None
- upload(from_file, to_file)#
- Return type
None
- class covalent.fs_strategies.S3(credentials=None, profile=None, region_name=None)#
Bases:
covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy
Implements Base FileTransferStrategy class to upload/download files from S3 Bucket.
Methods:
cp
(from_file[, to_file])- rtype
download
(from_file[, to_file])Download files or the contents of folders from S3 bucket.
upload
(from_file[, to_file])Upload files or folders to S3 bucket.
- download(from_file, to_file=<covalent._file_transfer.file.File object>)#
Download files or the contents of folders from S3 bucket.
- Return type