fluke.storage

class fluke.storage.AmazonS3Dir(*args, **kwargs)[source]

This class represents a virtual directory which resides within an Amazon S3 bucket.

Parameters

auth (AWSAuth) – An AWSAuth instance used for authenticating with AWS.
bucket (str) – The name of the bucket in which the directory resides.
path (str | None) – The path pointing to the directory. If None, then the whole bucket is considered. Defaults to None.
cache (bool) – Indicates whether it is allowed for any fetched data to be cached for faster subsequent access. Defaults to False.
create_if_missing (bool) – If set to True, then the directory to which the provided path points will be automatically created in case it does not already exist, instead of an exception being thrown. Defaults to False.

Raises

BucketNotFoundError – The specified bucket does not exist.
InvalidPathError – The provided path does not exist.
InvalidDirectoryError – The provided path does not point to a directory.

Note

The provided path must not begin with a separator.

Wrong: /path/to/dir
Right: path/to/dir

close() → None: Closes all open connections.

count(recursively: bool = False) → int

Returns the total number of files within within the directory.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting number may vary depending on the value of parameter recursively.

get_bucket_name() → str[source]: Returns the name of the bucket in which the directory resides.

get_contents(recursively: bool = False, show_abs_path: bool = False) → list[str]

Returns a list containing the paths that correspond to the directory’s contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

The resulting list may vary depending on the value of parameter recursively.

get_file(path: str) → AmazonS3File[source]

Returns the file residing in the specified path as a AmazonS3File instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the file in question.

Raises

InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path does not point to a file within the directory.

Note

The provided path, if absolute, must not begin with a separator.

Wrong: /path/to/file.txt
Right: path/to/file.txt

get_metadata(file_path: str) → dict[str, str]

Returns a dictionary containing any metadata associated with the file in question.

Parameters: file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
Raises: InvalidFileError – The provided path does not point to a file within the directory.

get_name() → Optional[str]: Returns the name of the directory, or None if it’s the root directory.

get_path() → str: Returns the path to the directory.

get_size(recursively: bool = False) → int

Returns the total sum of the sizes of all files within the directory, in bytes.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting size may vary depending on the value of parameter recursively.

get_subdir(path: str) → AmazonS3Dir[source]

Returns the directory residing in the specified path as an AmazonS3Dir instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the subdirectory in question.

Raises

InvalidPathError – The provided path does not exist.

Note

The provided path, if absolute, must not begin with a separator.

Wrong: /path/to/dir
Right: path/to/dir

get_uri() → str[source]: Returns the directory’s URI.

is_cacheable() → bool: Returns True if directory has been defined as cacheable, else returns False.

is_file(path: str) → bool

Returns True if the provided path points to a file, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

load_metadata(recursively: bool = False) → None

Loads any metadata associated with the files within the directory, which can then be accessed via the instance’s get_metadata method.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.

Note

The number of the loaded metadata may vary depending on the value of parameter recursively.
Any metadata set via the set_metadata method will be overridden after invoking this method.

ls(recursively: bool = False, show_abs_path: bool = False) → None

Lists the contents of the directory.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

Empty directories will not be considered if parameter recursively is set to True.

open() → None: Opens all necessary connections.

path_exists(path: str) → bool

Returns True if the provided path exists within the directory, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

purge() → None: If cacheable, then purges the directory’s cache, else does nothing.

set_metadata(file_path: str, metadata: dict[str, str]) → None

Associates the provided metadata with the file located in the given path.

Parameters

file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.
InvalidFileError – The provided path does not point to a file within the directory.

transfer_to(dst: _Directory, recursively: bool = False, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, filter: Optional[Callable[[str], bool]] = None, suppress_output: bool = False) → bool

Copies all files within this directory into the destination directory. Returns True if all files were successfully transferred, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
overwrite (bool) – Indicates whether to overwrite any files if they already exist. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting files. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
filter (Callable[[str], bool] | None) – A filter function to be used in order to exclude certain files from being transferred. This function is considered to receive a single parameter, namely the file’s absolute path within the source directory, and to return a boolean value, based on which it is determined whether the file is to be filtered out during the transfer (True) or not (False). Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.

traverse(recursively: bool = False, show_abs_path: bool = False) → Iterator[str]

Returns an iterator capable of traversing the directory by going through the paths of its contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Indicates whether it should be displayed the absolute or the relative path of the contents. Defaults to False.

Note

The resulting iterator may vary depending on the value of parameter recursively.

class fluke.storage.AmazonS3File(*args, **kwargs)[source]

This class represents an object which resides within an Amazon S3 bucket.

Parameters

auth (AWSAuth) – An AWSAuth instance used for authenticating with AWS.
bucket (str) – The name of the bucket in which the file resides.
path (str) – The path pointing to the file.
load_metadata (bool) – Indicates whether any existing metadata should be loaded during the the file’s instantiation. Defaults to False.
cache (bool) – Indicates whether it is allowed for any fetched data to be cached for faster subsequent access. Defaults to False.

Raises

BucketNotFoundError – The specified bucket does not exist.
InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path points to a directory.

Note

The provided path must not begin with a separator.

Wrong: /path/to/file.txt
Right: path/to/file.txt

cat(encoding: str = 'utf-8') → None

Prints the contents of a file as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

close() → None: Closes all open connections.

get_bucket_name() → str[source]: Returns the name of the bucket in which the directory resides.

get_metadata() → dict[str, str]: Returns a dictionary containing any metadata associated with the file.

get_name() → str: Returns the file’s name.

get_path() → str: Returns the file’s absolute path.

get_size() → int: Returns the file’s size in bytes.

get_uri() → str[source]: Returns the object’s URI.

is_cacheable() → bool: Returns True if file has been defined as cacheable, else returns False.

load_metadata() → None

Loads any metadata associated with the file, which can then be accessed via the instance’s get_metadata method.

Note: Any metadata set via the set_metadata method will be overridden after invoking this method.

open() → None: Opens all necessary connections.

purge() → None: If cacheable, then purges the file’s cache, else does nothing.

read() → bytes: Reads and returns the file’s contents in bytes.

read_chunks(chunk_size: int = 8192) → Iterator[bytes]

Returns an iterator capable of going through the file’s contents as distinct chunks of bytes.

Parameters: chunk_size (int) – The file chunk size in bytes. Defaults to 8192.

read_lines(chunk_size: Optional[int] = None, encoding: str = 'utf-8') → Iterator[str]

Returns an iterator capable of going through the file line-by-line.

Parameters

chunk_size (int | None) – If not None, then the file is read in distinct chunks, whose size are equal to this parameter value. Defaults to None.
encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.

Note

See Standard Encodings for all available encodings.

read_range(start: Optional[int], end: Optional[int]) → bytes

Returns the file contents that correspond to the provided byte range.

Parameters

start (int | None) – The point in file from which to begin reading bytes. If None, then begin reading from the start of the file.
end (int | None) – The point in file at which to stop reading bytes. If None, then stop reading at the end of the file.

read_text(encoding: str = 'utf-8') → str

Reads and returns the file’s contents as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

set_metadata(metadata: dict[str, str]) → None

Associates the provided metadata with this file.

Parameters

metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.

transfer_to(dst: _Directory, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, suppress_output: bool = False) → bool

Copies the file into the provided directory. Returns True if everything went as expected, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
overwrite (bool) – Indicates whether to overwrite the file if it already exists. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting file. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.

class fluke.storage.AzureBlobDir(*args, **kwargs)[source]

This class represents a virtual directory which resides within an Azure blob container.

Parameters

auth (AzureAuth) – An AzureAuth instance used for authenticating with Microsoft Azure.
container (str) – The name of the container in which the directory resides.
path (str | None) – The path pointing to the directory. If None, then the whole container is considered. Defaults to None.
cache (bool) – Indicates whether it is allowed for any fetched data to be cached for faster subsequent access. Defaults to False.
create_if_missing (bool) – If set to True, then the directory to which the provided path points will be automatically created in case it does not already exist, instead of an exception being thrown. Defaults to False.

Raises

ContainerNotFoundError – The specified container does not exist.
InvalidPathError – The provided path does not exist.
InvalidDirectoryError – The provided path does not point to a directory.

Note

The provided path must not begin with a separator.

Wrong: /path/to/dir
Right: path/to/dir

close() → None: Closes all open connections.

count(recursively: bool = False) → int

Returns the total number of files within within the directory.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting number may vary depending on the value of parameter recursively.

get_container_name() → str[source]: Returns the name of the bucket in which the directory resides.

get_contents(recursively: bool = False, show_abs_path: bool = False) → list[str]

Returns a list containing the paths that correspond to the directory’s contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

The resulting list may vary depending on the value of parameter recursively.

get_file(path: str) → AzureBlobFile[source]

Returns the file residing in the specified path as a AzureBlobFile instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the file in question.

Raises

InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path does not point to a file within the directory.

Note

The provided path, if absolute, must not begin with a separator.

Wrong: /path/to/file.txt
Right: path/to/file.txt

get_metadata(file_path: str) → dict[str, str]

Returns a dictionary containing any metadata associated with the file in question.

Parameters: file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
Raises: InvalidFileError – The provided path does not point to a file within the directory.

get_name() → Optional[str]: Returns the name of the directory, or None if it’s the root directory.

get_path() → str: Returns the path to the directory.

get_size(recursively: bool = False) → int

Returns the total sum of the sizes of all files within the directory, in bytes.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting size may vary depending on the value of parameter recursively.

get_subdir(path: str) → AzureBlobDir[source]

Returns the directory residing in the specified path as an AzureBlobDir instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the subdirectory in question.

Raises

InvalidPathError – The provided path does not exist.

Note

The provided path, if absolute, must not begin with a separator.

Wrong: /path/to/dir
Right: path/to/dir

get_uri() → str[source]: Returns the directory’s URI.

is_cacheable() → bool: Returns True if directory has been defined as cacheable, else returns False.

is_file(path: str) → bool

Returns True if the provided path points to a file, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

load_metadata(recursively: bool = False) → None

Loads any metadata associated with the files within the directory, which can then be accessed via the instance’s get_metadata method.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.

Note

The number of the loaded metadata may vary depending on the value of parameter recursively.
Any metadata set via the set_metadata method will be overridden after invoking this method.

ls(recursively: bool = False, show_abs_path: bool = False) → None

Lists the contents of the directory.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

Empty directories will not be considered if parameter recursively is set to True.

open() → None: Opens all necessary connections.

path_exists(path: str) → bool

Returns True if the provided path exists within the directory, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

purge() → None: If cacheable, then purges the directory’s cache, else does nothing.

set_metadata(file_path: str, metadata: dict[str, str]) → None

Associates the provided metadata with the file located in the given path.

Parameters

file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.
InvalidFileError – The provided path does not point to a file within the directory.

transfer_to(dst: _Directory, recursively: bool = False, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, filter: Optional[Callable[[str], bool]] = None, suppress_output: bool = False) → bool

Copies all files within this directory into the destination directory. Returns True if all files were successfully transferred, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
overwrite (bool) – Indicates whether to overwrite any files if they already exist. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting files. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
filter (Callable[[str], bool] | None) – A filter function to be used in order to exclude certain files from being transferred. This function is considered to receive a single parameter, namely the file’s absolute path within the source directory, and to return a boolean value, based on which it is determined whether the file is to be filtered out during the transfer (True) or not (False). Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.

traverse(recursively: bool = False, show_abs_path: bool = False) → Iterator[str]

Returns an iterator capable of traversing the directory by going through the paths of its contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Indicates whether it should be displayed the absolute or the relative path of the contents. Defaults to False.

Note

The resulting iterator may vary depending on the value of parameter recursively.

class fluke.storage.AzureBlobFile(*args, **kwargs)[source]

This class represents a blob which resides within an Azure blob container.

Parameters

auth (AzureAuth) – An AzureAuth instance used for authenticating with Microsoft Azure.
container (str) – The name of the container in which the blob resides.
path (str) – The path pointing to the file.
load_metadata (bool) – Indicates whether any existing metadata should be loaded during the the file’s instantiation. Defaults to False.
cache (bool) – Indicates whether it is allowed for any fetched data to be cached for faster subsequent access. Defaults to False.

Raises

ContainerNotFoundError – The specified container does not exist.
InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path points to a directory.

Note

The provided path must not begin with a separator.

Wrong: /path/to/file.txt
Right: path/to/file.txt

cat(encoding: str = 'utf-8') → None

Prints the contents of a file as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

close() → None: Closes all open connections.

get_container_name() → str[source]: Returns the name of the bucket in which the directory resides.

get_metadata() → dict[str, str]: Returns a dictionary containing any metadata associated with the file.

get_name() → str: Returns the file’s name.

get_path() → str: Returns the file’s absolute path.

get_size() → int: Returns the file’s size in bytes.

get_uri() → str[source]: Returns the object’s URI.

is_cacheable() → bool: Returns True if file has been defined as cacheable, else returns False.

load_metadata() → None

Loads any metadata associated with the file, which can then be accessed via the instance’s get_metadata method.

Note: Any metadata set via the set_metadata method will be overridden after invoking this method.

open() → None: Opens all necessary connections.

purge() → None: If cacheable, then purges the file’s cache, else does nothing.

read() → bytes: Reads and returns the file’s contents in bytes.

read_chunks(chunk_size: int = 8192) → Iterator[bytes]

Returns an iterator capable of going through the file’s contents as distinct chunks of bytes.

Parameters: chunk_size (int) – The file chunk size in bytes. Defaults to 8192.

read_lines(chunk_size: Optional[int] = None, encoding: str = 'utf-8') → Iterator[str]

Returns an iterator capable of going through the file line-by-line.

Parameters

chunk_size (int | None) – If not None, then the file is read in distinct chunks, whose size are equal to this parameter value. Defaults to None.
encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.

Note

See Standard Encodings for all available encodings.

read_range(start: Optional[int], end: Optional[int]) → bytes

Returns the file contents that correspond to the provided byte range.

Parameters

start (int | None) – The point in file from which to begin reading bytes. If None, then begin reading from the start of the file.
end (int | None) – The point in file at which to stop reading bytes. If None, then stop reading at the end of the file.

read_text(encoding: str = 'utf-8') → str

Reads and returns the file’s contents as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

set_metadata(metadata: dict[str, str]) → None

Associates the provided metadata with this file.

Parameters

metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.

transfer_to(dst: _Directory, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, suppress_output: bool = False) → bool

Copies the file into the provided directory. Returns True if everything went as expected, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
overwrite (bool) – Indicates whether to overwrite the file if it already exists. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting file. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.

class fluke.storage.GCPStorageDir(*args, **kwargs)[source]

This class represents a virtual directory which resides within a Google Cloud Storage bucket.

Parameters

auth (GCPAuth) – A GCPAuth instance used for authenticating with GCP.
bucket (str) – The name of the bucket in which the directory resides.
path (str | None) – The path pointing to the directory. If None, then the whole bucket is considered. Defaults to None.
cache (bool) – Indicates whether it is allowed for any fetched data to be cached for faster subsequent access. Defaults to False.
create_if_missing (bool) – If set to True, then the directory to which the provided path points will be automatically created in case it does not already exist, instead of an exception being thrown. Defaults to False.

Raises

BucketNotFoundError – The specified bucket does not exist.
InvalidPathError – The provided path does not exist.
InvalidDirectoryError – The provided path does not point to a directory.

Note

The provided path must not begin with a separator.

Wrong: /path/to/dir
Right: path/to/dir

close() → None: Closes all open connections.

count(recursively: bool = False) → int

Returns the total number of files within within the directory.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting number may vary depending on the value of parameter recursively.

get_bucket_name() → str[source]: Returns the name of the bucket in which the directory resides.

get_contents(recursively: bool = False, show_abs_path: bool = False) → list[str]

Returns a list containing the paths that correspond to the directory’s contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

The resulting list may vary depending on the value of parameter recursively.

get_file(path: str) → GCPStorageFile[source]

Returns the file residing in the specified path as a GCPStorageFile instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the file in question.

Raises

InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path does not point to a file within the directory.

Note

The provided path, if absolute, must not begin with a separator.

Wrong: /path/to/file.txt
Right: path/to/file.txt

get_metadata(file_path: str) → dict[str, str]

Returns a dictionary containing any metadata associated with the file in question.

Parameters: file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
Raises: InvalidFileError – The provided path does not point to a file within the directory.

get_name() → Optional[str]: Returns the name of the directory, or None if it’s the root directory.

get_path() → str: Returns the path to the directory.

get_size(recursively: bool = False) → int

Returns the total sum of the sizes of all files within the directory, in bytes.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting size may vary depending on the value of parameter recursively.

get_subdir(path: str) → GCPStorageDir[source]

Returns the directory residing in the specified path as an GCPStorageDir instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the subdirectory in question.

Raises

InvalidPathError – The provided path does not exist.

Note

The provided path, if absolute, must not begin with a separator.

Wrong: /path/to/dir
Right: path/to/dir

get_uri() → str[source]: Returns the directory’s URI.

is_cacheable() → bool: Returns True if directory has been defined as cacheable, else returns False.

is_file(path: str) → bool

Returns True if the provided path points to a file, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

load_metadata(recursively: bool = False) → None

Loads any metadata associated with the files within the directory, which can then be accessed via the instance’s get_metadata method.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.

Note

The number of the loaded metadata may vary depending on the value of parameter recursively.
Any metadata set via the set_metadata method will be overridden after invoking this method.

ls(recursively: bool = False, show_abs_path: bool = False) → None

Lists the contents of the directory.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

Empty directories will not be considered if parameter recursively is set to True.

open() → None: Opens all necessary connections.

path_exists(path: str) → bool

Returns True if the provided path exists within the directory, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

purge() → None: If cacheable, then purges the directory’s cache, else does nothing.

set_metadata(file_path: str, metadata: dict[str, str]) → None

Associates the provided metadata with the file located in the given path.

Parameters

file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.
InvalidFileError – The provided path does not point to a file within the directory.

transfer_to(dst: _Directory, recursively: bool = False, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, filter: Optional[Callable[[str], bool]] = None, suppress_output: bool = False) → bool

Copies all files within this directory into the destination directory. Returns True if all files were successfully transferred, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
overwrite (bool) – Indicates whether to overwrite any files if they already exist. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting files. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
filter (Callable[[str], bool] | None) – A filter function to be used in order to exclude certain files from being transferred. This function is considered to receive a single parameter, namely the file’s absolute path within the source directory, and to return a boolean value, based on which it is determined whether the file is to be filtered out during the transfer (True) or not (False). Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.

traverse(recursively: bool = False, show_abs_path: bool = False) → Iterator[str]

Returns an iterator capable of traversing the directory by going through the paths of its contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Indicates whether it should be displayed the absolute or the relative path of the contents. Defaults to False.

Note

The resulting iterator may vary depending on the value of parameter recursively.

class fluke.storage.GCPStorageFile(*args, **kwargs)[source]

This class represents an object which resides within a Google Cloud Storage bucket.

Parameters

auth (GCPAuth) – A GCPAuth instance used for authenticating with GCP.
bucket (str) – The name of the bucket in which the file resides.
path (str) – The path pointing to the file.
load_metadata (bool) – Indicates whether any existing metadata should be loaded during the the file’s instantiation. Defaults to False.
cache (bool) – Indicates whether it is allowed for any fetched data to be cached for faster subsequent access. Defaults to False.

Raises

BucketNotFoundError – The specified bucket does not exist.
InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path points to a directory.

Note

The provided path must not begin with a separator.

Wrong: /path/to/file.txt
Right: path/to/file.txt

cat(encoding: str = 'utf-8') → None

Prints the contents of a file as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

close() → None: Closes all open connections.

get_bucket_name() → str[source]: Returns the name of the bucket in which the directory resides.

get_metadata() → dict[str, str]: Returns a dictionary containing any metadata associated with the file.

get_name() → str: Returns the file’s name.

get_path() → str: Returns the file’s absolute path.

get_size() → int: Returns the file’s size in bytes.

get_uri() → str[source]: Returns the object’s URI.

is_cacheable() → bool: Returns True if file has been defined as cacheable, else returns False.

load_metadata() → None

Loads any metadata associated with the file, which can then be accessed via the instance’s get_metadata method.

Note: Any metadata set via the set_metadata method will be overridden after invoking this method.

open() → None: Opens all necessary connections.

purge() → None: If cacheable, then purges the file’s cache, else does nothing.

read() → bytes: Reads and returns the file’s contents in bytes.

read_chunks(chunk_size: int = 8192) → Iterator[bytes]

Returns an iterator capable of going through the file’s contents as distinct chunks of bytes.

Parameters: chunk_size (int) – The file chunk size in bytes. Defaults to 8192.

read_lines(chunk_size: Optional[int] = None, encoding: str = 'utf-8') → Iterator[str]

Returns an iterator capable of going through the file line-by-line.

Parameters

chunk_size (int | None) – If not None, then the file is read in distinct chunks, whose size are equal to this parameter value. Defaults to None.
encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.

Note

See Standard Encodings for all available encodings.

read_range(start: Optional[int], end: Optional[int]) → bytes

Returns the file contents that correspond to the provided byte range.

Parameters

start (int | None) – The point in file from which to begin reading bytes. If None, then begin reading from the start of the file.
end (int | None) – The point in file at which to stop reading bytes. If None, then stop reading at the end of the file.

read_text(encoding: str = 'utf-8') → str

Reads and returns the file’s contents as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

set_metadata(metadata: dict[str, str]) → None

Associates the provided metadata with this file.

Parameters

metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.

transfer_to(dst: _Directory, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, suppress_output: bool = False) → bool

Copies the file into the provided directory. Returns True if everything went as expected, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
overwrite (bool) – Indicates whether to overwrite the file if it already exists. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting file. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.

class fluke.storage.LocalDir(*args, **kwargs)[source]

This class represents a directory which resides within the local file system.

Parameters

path (str) – The path pointing to the directory.
create_if_missing (bool) – If set to True, then the directory to which the provided path points will be automatically created in case it does not already exist, instead of an exception being thrown. Defaults to False.

Raises

InvalidPathError – The provided path does not exist.
InvalidDirectoryError – The provided path does not point to a directory.

count(recursively: bool = False) → int

Returns the total number of files within within the directory.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting number may vary depending on the value of parameter recursively.

get_contents(recursively: bool = False, show_abs_path: bool = False) → list[str]

Returns a list containing the paths that correspond to the directory’s contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

The resulting list may vary depending on the value of parameter recursively.

get_file(path: str) → LocalFile[source]

Returns the file residing in the specified path as a LocalFile instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the file in question.

Raises

InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path does not point to a file within the directory.

get_metadata(file_path: str) → dict[str, str]

Returns a dictionary containing any metadata associated with the file in question.

Parameters: file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
Raises: InvalidFileError – The provided path does not point to a file within the directory.

get_name() → Optional[str]: Returns the name of the directory, or None if it’s the root directory.

get_path() → str: Returns the path to the directory.

get_size(recursively: bool = False) → int

Returns the total sum of the sizes of all files within the directory, in bytes.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting size may vary depending on the value of parameter recursively.

get_subdir(path: str) → LocalDir[source]

Returns the directory residing in the specified path as a LocalDir instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the subdirectory in question.

Raises

InvalidPathError – The provided path does not exist.
InvalidDirectoryError – The provided path does not point to a subdirectory within the directory.

get_uri() → str[source]: Returns the directory’s URI.

is_file(path: str) → bool

Returns True if the provided path points to a file, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

ls(recursively: bool = False, show_abs_path: bool = False) → None

Lists the contents of the directory.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

Empty directories will not be considered if parameter recursively is set to True.

path_exists(path: str) → bool

Returns True if the provided path exists within the directory, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

set_metadata(file_path: str, metadata: dict[str, str]) → None

Associates the provided metadata with the file located in the given path.

Parameters

file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.
InvalidFileError – The provided path does not point to a file within the directory.

transfer_to(dst: _Directory, recursively: bool = False, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, filter: Optional[Callable[[str], bool]] = None, suppress_output: bool = False) → bool

Copies all files within this directory into the destination directory. Returns True if all files were successfully transferred, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
overwrite (bool) – Indicates whether to overwrite any files if they already exist. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting files. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
filter (Callable[[str], bool] | None) – A filter function to be used in order to exclude certain files from being transferred. This function is considered to receive a single parameter, namely the file’s absolute path within the source directory, and to return a boolean value, based on which it is determined whether the file is to be filtered out during the transfer (True) or not (False). Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.

traverse(recursively: bool = False, show_abs_path: bool = False) → Iterator[str]

Returns an iterator capable of traversing the directory by going through the paths of its contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Indicates whether it should be displayed the absolute or the relative path of the contents. Defaults to False.

Note

The resulting iterator may vary depending on the value of parameter recursively.

class fluke.storage.LocalFile(*args, **kwargs)[source]

This class represents a file which resides within the local file system.

Parameters

path (str) – A path pointing to the file.

Raises

InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path points to a directory.

cat(encoding: str = 'utf-8') → None

Prints the contents of a file as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

get_metadata() → dict[str, str]: Returns a dictionary containing any metadata associated with the file.

get_name() → str: Returns the file’s name.

get_path() → str: Returns the file’s absolute path.

get_size() → int: Returns the file’s size in bytes.

get_uri() → str[source]: Returns the file’s URI.

read() → bytes: Reads and returns the file’s contents in bytes.

read_chunks(chunk_size: int = 8192) → Iterator[bytes]

Returns an iterator capable of going through the file’s contents as distinct chunks of bytes.

Parameters: chunk_size (int) – The file chunk size in bytes. Defaults to 8192.

read_lines(chunk_size: Optional[int] = None, encoding: str = 'utf-8') → Iterator[str]

Returns an iterator capable of going through the file line-by-line.

Parameters

chunk_size (int | None) – If not None, then the file is read in distinct chunks, whose size are equal to this parameter value. Defaults to None.
encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.

Note

See Standard Encodings for all available encodings.

read_range(start: Optional[int], end: Optional[int]) → bytes

Returns the file contents that correspond to the provided byte range.

Parameters

start (int | None) – The point in file from which to begin reading bytes. If None, then begin reading from the start of the file.
end (int | None) – The point in file at which to stop reading bytes. If None, then stop reading at the end of the file.

read_text(encoding: str = 'utf-8') → str

Reads and returns the file’s contents as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

set_metadata(metadata: dict[str, str]) → None

Associates the provided metadata with this file.

Parameters

metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.

transfer_to(dst: _Directory, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, suppress_output: bool = False) → bool

Copies the file into the provided directory. Returns True if everything went as expected, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
overwrite (bool) – Indicates whether to overwrite the file if it already exists. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting file. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.

class fluke.storage.RemoteDir(*args, **kwargs)[source]

This class represents a directory which resides within a remote file system.

Parameters

auth (RemoteAuth) – A RemoteAuth instance used for authenticating with a remote machine via the SSH protocol.
path (str) – The path pointing to the directory.
cache (bool) – Indicates whether it is allowed for any fetched data to be cached for faster subsequent access. Defaults to False.
create_if_missing (bool) – If set to True, then the directory to which the provided path points will be automatically created in case it does not already exist, instead of an exception being thrown. Defaults to False.

Raises

InvalidPathError – The provided path does not exist.
InvalidDirectoryError – The provided path does not point to a directory.

close() → None: Closes all open connections.

count(recursively: bool = False) → int

Returns the total number of files within within the directory.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting number may vary depending on the value of parameter recursively.

get_contents(recursively: bool = False, show_abs_path: bool = False) → list[str]

Returns a list containing the paths that correspond to the directory’s contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

The resulting list may vary depending on the value of parameter recursively.

get_file(path: str) → RemoteFile[source]

Returns the file residing in the specified path as a RemoteFile instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the file in question.

Raises

InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path does not point to a file within the directory.

get_hostname() → str[source]: Returns the name of the host in which this directory resides.

get_metadata(file_path: str) → dict[str, str]

Returns a dictionary containing any metadata associated with the file in question.

Parameters: file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
Raises: InvalidFileError – The provided path does not point to a file within the directory.

get_name() → Optional[str]: Returns the name of the directory, or None if it’s the root directory.

get_path() → str: Returns the path to the directory.

get_size(recursively: bool = False) → int

Returns the total sum of the sizes of all files within the directory, in bytes.

Parameters: recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
Note: The resulting size may vary depending on the value of parameter recursively.

get_subdir(path: str) → RemoteDir[source]

Returns the subdirectory residing in the specified path as a RemoteDir instance.

Parameters

path (str) – Either the absolute path or the path relative to the directory of the subdirectory in question.

Raises

InvalidPathError – The provided path does not exist.
InvalidDirectoryError – The provided path does not point to a subdirectory within the directory.

get_uri() → str[source]: Returns the directory’s URI.

is_cacheable() → bool: Returns True if directory has been defined as cacheable, else returns False.

is_file(path: str) → bool

Returns True if the provided path points to a file, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

ls(recursively: bool = False, show_abs_path: bool = False) → None

Lists the contents of the directory.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Determines whether to include the contents’ absolute path or their path relative to this directory. Defaults to False.

Note

Empty directories will not be considered if parameter recursively is set to True.

open() → None: Opens all necessary connections.

path_exists(path: str) → bool

Returns True if the provided path exists within the directory, else returns False.

Parameters: path (str) – Either an absolute path or a path relative to the directory.

purge() → None: If cacheable, then purges the directory’s cache, else does nothing.

set_metadata(file_path: str, metadata: dict[str, str]) → None

Associates the provided metadata with the file located in the given path.

Parameters

file_path (str) – Either the absolute path or the path relative to the directory of the file in question.
metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.
InvalidFileError – The provided path does not point to a file within the directory.

transfer_to(dst: _Directory, recursively: bool = False, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, filter: Optional[Callable[[str], bool]] = None, suppress_output: bool = False) → bool

Copies all files within this directory into the destination directory. Returns True if all files were successfully transferred, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
overwrite (bool) – Indicates whether to overwrite any files if they already exist. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting files. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
filter (Callable[[str], bool] | None) – A filter function to be used in order to exclude certain files from being transferred. This function is considered to receive a single parameter, namely the file’s absolute path within the source directory, and to return a boolean value, based on which it is determined whether the file is to be filtered out during the transfer (True) or not (False). Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.

traverse(recursively: bool = False, show_abs_path: bool = False) → Iterator[str]

Returns an iterator capable of traversing the directory by going through the paths of its contents.

Parameters

recursively (bool) – Indicates whether the directory is to be traversed recursively or not. If set to False, then only those files that reside directly within the directory are to be considered. If set to True, then all files are considered, no matter whether they reside directly within the directory or within any of its subdirectories. Defaults to False.
show_abs_path (bool) – Indicates whether it should be displayed the absolute or the relative path of the contents. Defaults to False.

Note

The resulting iterator may vary depending on the value of parameter recursively.

class fluke.storage.RemoteFile(*args, **kwargs)[source]

This class represents a file which resides within a remote machine’s file system.

Parameters

auth (RemoteAuth) – A RemoteAuth instance used for authenticating with a remote machine via the SSH protocol.
path (str) – A path pointing to the file.
cache (bool) – Indicates whether it is allowed for any fetched data to be cached for faster subsequent access. Defaults to False.

Raises

InvalidPathError – The provided path does not exist.
InvalidFileError – The provided path points to a directory.

cat(encoding: str = 'utf-8') → None

Prints the contents of a file as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

close() → None: Closes all open connections.

get_hostname() → str[source]: Returns the name of the host in which this fileresides.

get_metadata() → dict[str, str]: Returns a dictionary containing any metadata associated with the file.

get_name() → str: Returns the file’s name.

get_path() → str: Returns the file’s absolute path.

get_size() → int: Returns the file’s size in bytes.

get_uri() → str[source]: Returns the file’s URI.

is_cacheable() → bool: Returns True if file has been defined as cacheable, else returns False.

open() → None: Opens all necessary connections.

purge() → None: If cacheable, then purges the file’s cache, else does nothing.

read() → bytes: Reads and returns the file’s contents in bytes.

read_chunks(chunk_size: int = 8192) → Iterator[bytes]

Returns an iterator capable of going through the file’s contents as distinct chunks of bytes.

Parameters: chunk_size (int) – The file chunk size in bytes. Defaults to 8192.

read_lines(chunk_size: Optional[int] = None, encoding: str = 'utf-8') → Iterator[str]

Returns an iterator capable of going through the file line-by-line.

Parameters

chunk_size (int | None) – If not None, then the file is read in distinct chunks, whose size are equal to this parameter value. Defaults to None.
encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.

Note

See Standard Encodings for all available encodings.

read_range(start: Optional[int], end: Optional[int]) → bytes

Returns the file contents that correspond to the provided byte range.

Parameters

start (int | None) – The point in file from which to begin reading bytes. If None, then begin reading from the start of the file.
end (int | None) – The point in file at which to stop reading bytes. If None, then stop reading at the end of the file.

read_text(encoding: str = 'utf-8') → str

Reads and returns the file’s contents as text.

Parameters: encoding (str) – The encoding with which to decode the bytes. Defaults to utf-8.
Note: See Standard Encodings for all available encodings.

set_metadata(metadata: dict[str, str]) → None

Associates the provided metadata with this file.

Parameters

metadata (dict[str, str]) – A dictionary containing the metadata that are to be associated with the file.

Raises

NonStringMetadataKeyError – The provided metadata dictionary contains at least one non-string key.
NonStringMetadataValueError – The provided metadata dictionary contains at least one non-string value.

transfer_to(dst: _Directory, overwrite: bool = False, include_metadata: bool = False, chunk_size: Optional[int] = None, suppress_output: bool = False) → bool

Copies the file into the provided directory. Returns True if everything went as expected, else returns False.

Parameters

dst (_Directory) – A _Directory class instance, which represents the transfer operation’s destination.
overwrite (bool) – Indicates whether to overwrite the file if it already exists. Defaults to False.
include_metadata (bool) – Indicates whether any existing metadata are to be assigned to the resulting file. Defaults to False.
chunk_size (int | None) – If not None, then files are transferred in chunks, whose size are equal to this parameter value. Defaults to None.
suppress_output (bool) – If set to True, then suppresses all output. Defaults to False.

Raises

InvalidChunkSizeError – Transferring files in chunks of the given size is not supported by the specified destination.