Storages Documentation

This module defines all the storage systems supported by uriutils.

class uriutils.storages.URIBytesOutput(uri_obj)

A BytesIO object for output that flushes content to the remote URI on close.

__init__(uri_obj)

close()

name

class uriutils.storages.BaseURI(storage_args={})

This is the base URI storage object that is inherited by the different storage systems. It defines the methods and operations that can be “conducted” on a URI. Almost all of these methods have to be implemented by a storage class.

SUPPORTED_SCHEMES = []

Defines the schemes supported by this storage system.

VALID_STORAGE_ARGS = []

The set of storage_args keyword arguments that is handled by this storage system.

__init__(storage_args={})

Parameters:storage_args (dict) – Arguments that will be applied to the storage system for read/write operations

dir_exists()

Check if the URI exists as a directory.

Returns:True if URI exists as a directory Return type:bool

download_file(filename)

Download the binary content stored in the URI for this object directly to local file.

Parameters:filename (str) – Filename on local filesystem

exists()

Returns:True if URI exists Return type:bool

get_content()

Returns:the bytestring stored at this object’s URI Return type:bytes

get_metadata()

Returns:the metadata associated with this object’s URI Return type:dict

join(path)

Similar to os.path.join() but returns a storage object instead.

Parameters:path (str) – path to join on to this object’s URI Returns:a storage object Return type:BaseURI

list_dir()

List the contents of a directory.

make_dir()

Create a directory.

classmethod parse_uri(uri, storage_args={})

Parses the URI and return an instantiation of the storage system if it is supported.

Parameters:

• uri (str) – URI to check
• storage_args (dict) – Keyword arguments to pass to the underlying storage object

Returns:

None if this storage system does not support uri.

put_content(content)

Parameters:content (bytes) – Content to write to this object’s URI

upload_file(filename)

Upload the binary content in filename to the URI for this object.

Parameters:filename (str) – Filename on local filesystem

Local filesystem

class uriutils.storages.FileURI(filepath, storage_args={})

Storage system for local filesystem.

Parameters:

• filepath (str) – Local file path
• storage_args (dict) – Keyword arguments that are passed to open()

SUPPORTED_SCHEMES = set(['', 'file'])

Supported schemes for FileURI.

AWS Simple Storage Service

class uriutils.storages.S3URI(bucket, key, storage_args={})

Storage system for AWS S3.

SUPPORTED_SCHEMES = set(['s3'])

Supported schemes for S3URI.

VALID_STORAGE_ARGS = ['CacheControl', 'ContentDisposition', 'ContentEncoding', 'ContentLanguage', 'ContentLength', 'ContentMD5', 'ContentType', 'Expires', 'GrantFullControl', 'GrantRead', 'GrantReadACP', 'GrantWriteACP', 'Metadata', 'ServerSideEncryption', 'StorageClass', 'WebsiteRedirectLocation', 'SSECustomerAlgorithm', 'SSECustomerKey', 'SSEKMSKeyId', 'RequestPayer', 'Tagging']

Storage arguments allowed to pass to S3.Client methods.

__init__(bucket, key, storage_args={})

Parameters:

• bucket (str) – Bucket name
• key (str) – Key to file
• storage_args (dict) – Keyword arguments that are passed to S3.Client

exists()

Uses HEAD requests for efficiency.

get_metadata()

Uses HEAD requests for efficiency.

list_dir()

Non-recursive file listing.

Returns:A generator over files in this “directory” for efficiency.

make_dir()

Ignored for S3.

Google Cloud Storage

class uriutils.storages.GoogleCloudStorageURI(bucket, key, storage_args={})

Storage system for Google Cloud storage.

SUPPORTED_SCHEMES = set(['gcs', 'gs'])

Supported schemes for GoogleCloudStorageURI.

VALID_STORAGE_ARGS = ['chunk_size', 'encryption_key']

Storage arguments allowed to pass to google.cloud.storage.client methods.

__init__(bucket, key, storage_args={})

Parameters:

• bucket (str) – Bucket name
• key (str) – Key to file
• storage_args (dict) – Keyword arguments that are passed to google.cloud.storage.client

exists()

Uses HEAD requests for efficiency.

get_metadata()

Uses HEAD requests for efficiency.

list_dir()

Non-recursive file listing.

Returns:A generator over files in this “directory” for efficiency.

put_content(content)

The default content type is set to application/octet-stream and content encoding set to None.

HTTP

class uriutils.storages.HTTPURI(url, raise_for_status=True, method=None, storage_args={})

Storage system for HTTP/HTTPS.

SUPPORTED_SCHEMES = set(['http', 'https'])

Supported schemes for HTTPURI.

VALID_STORAGE_ARGS = ['params', 'headers', 'cookies', 'auth', 'timeout', 'allow_redirects', 'proxies', 'verify', 'stream', 'cert', 'method']

Keyword arguments passed to requests.request().

__init__(url, raise_for_status=True, method=None, storage_args={})

Parameters:

• uri (str) – HTTP URI.
• raise_for_status (str) – Raises a requests.RequestException when the response status code is not 2xx (i.e., calls requests.Request.raise_for_status())
• method (str) – Overrides the default method for all HTTP operations.
• storage_args (dict) – Keyword arguments that are passed to requests.request()

dir_exists()

Makes a HEAD requests to the URI.

Returns:True if status code is 2xx.

make_dir()

Ignored.

put_content(content)

Makes a PUT request with the content in the body.

Raise:An requests.RequestException if it is not 2xx.

AWS Simple Notification Service

class uriutils.storages.SNSURI(topic_name, region, storage_args={})

Storage system for AWS Simple Notification Service.

SUPPORTED_SCHEMES = set(['sns'])

Supported schemes for SNSURI.

VALID_STORAGE_ARGS = ['Subject', 'MessageAttributes', 'MessageStructure']

Keyword arguments passed to SNS.Client.publish().

__init__(topic_name, region, storage_args={})

Parameters:

• topic_name (str) – Name of SNS topic for publishing; it can be either an ARN or just the topic name (thus defaulting to the current role’s account)
• region (str) – AWS region of SNS topic (defaults to current role’s region)
• storage_args (dict) – Keyword arguments that are passed to SNS.Client.publish()

dir_exists()

Not supported.

download_file(filename)

Not supported.

exists()

Returns:True if the SNS topic exists

get_content()

Not supported.

put_content(content)

Publishes a message straight to SNS.

Parameters:content (bytes) – raw bytes content to publish, will decode to UTF-8 if string is detected