Storages Documentation

This module defines all the storage systems supported by uriutils.

class uriutils.storages.URIBytesOutput(uri_obj)[source]

A BytesIO object for output that flushes content to the remote URI on close.

__init__(uri_obj)[source]
close()[source]
name
class uriutils.storages.BaseURI(storage_args={})[source]

This is the base URI storage object that is inherited by the different storage systems. It defines the methods and operations that can be “conducted” on a URI. Almost all of these methods have to be implemented by a storage class.

SUPPORTED_SCHEMES = []

Defines the schemes supported by this storage system.

VALID_STORAGE_ARGS = []

The set of storage_args keyword arguments that is handled by this storage system.

__init__(storage_args={})[source]
Parameters:storage_args (dict) – Arguments that will be applied to the storage system for read/write operations
dir_exists()[source]

Check if the URI exists as a directory.

Returns:True if URI exists as a directory
Return type:bool
download_file(filename)[source]

Download the binary content stored in the URI for this object directly to local file.

Parameters:filename (str) – Filename on local filesystem
exists()[source]
Returns:True if URI exists
Return type:bool
get_content()[source]
Returns:the bytestring stored at this object’s URI
Return type:bytes
get_metadata()[source]
Returns:the metadata associated with this object’s URI
Return type:dict
join(path)[source]

Similar to os.path.join() but returns a storage object instead.

Parameters:path (str) – path to join on to this object’s URI
Returns:a storage object
Return type:BaseURI
list_dir()[source]

List the contents of a directory.

make_dir()[source]

Create a directory.

classmethod parse_uri(uri, storage_args={})[source]

Parses the URI and return an instantiation of the storage system if it is supported.

Parameters:
  • uri (str) – URI to check
  • storage_args (dict) – Keyword arguments to pass to the underlying storage object
Returns:

None if this storage system does not support uri.

put_content(content)[source]
Parameters:content (bytes) – Content to write to this object’s URI
upload_file(filename)[source]

Upload the binary content in filename to the URI for this object.

Parameters:filename (str) – Filename on local filesystem

Local filesystem

class uriutils.storages.FileURI(filepath, storage_args={})[source]

Storage system for local filesystem.

Parameters:
  • filepath (str) – Local file path
  • storage_args (dict) – Keyword arguments that are passed to open()
SUPPORTED_SCHEMES = set(['', 'file'])

Supported schemes for FileURI.

AWS Simple Storage Service

class uriutils.storages.S3URI(bucket, key, storage_args={})[source]

Storage system for AWS S3.

SUPPORTED_SCHEMES = set(['s3'])

Supported schemes for S3URI.

VALID_STORAGE_ARGS = ['CacheControl', 'ContentDisposition', 'ContentEncoding', 'ContentLanguage', 'ContentLength', 'ContentMD5', 'ContentType', 'Expires', 'GrantFullControl', 'GrantRead', 'GrantReadACP', 'GrantWriteACP', 'Metadata', 'ServerSideEncryption', 'StorageClass', 'WebsiteRedirectLocation', 'SSECustomerAlgorithm', 'SSECustomerKey', 'SSEKMSKeyId', 'RequestPayer', 'Tagging']

Storage arguments allowed to pass to S3.Client methods.

__init__(bucket, key, storage_args={})[source]
Parameters:
  • bucket (str) – Bucket name
  • key (str) – Key to file
  • storage_args (dict) – Keyword arguments that are passed to S3.Client
exists()[source]

Uses HEAD requests for efficiency.

get_metadata()[source]

Uses HEAD requests for efficiency.

list_dir()[source]

Non-recursive file listing.

Returns:A generator over files in this “directory” for efficiency.
make_dir()[source]

Ignored for S3.

Google Cloud Storage

class uriutils.storages.GoogleCloudStorageURI(bucket, key, storage_args={})[source]

Storage system for Google Cloud storage.

SUPPORTED_SCHEMES = set(['gcs', 'gs'])

Supported schemes for GoogleCloudStorageURI.

VALID_STORAGE_ARGS = ['chunk_size', 'encryption_key']

Storage arguments allowed to pass to google.cloud.storage.client methods.

__init__(bucket, key, storage_args={})[source]
Parameters:
exists()[source]

Uses HEAD requests for efficiency.

get_metadata()[source]

Uses HEAD requests for efficiency.

list_dir()[source]

Non-recursive file listing.

Returns:A generator over files in this “directory” for efficiency.
put_content(content)[source]

The default content type is set to application/octet-stream and content encoding set to None.

HTTP

class uriutils.storages.HTTPURI(url, raise_for_status=True, method=None, storage_args={})[source]

Storage system for HTTP/HTTPS.

SUPPORTED_SCHEMES = set(['http', 'https'])

Supported schemes for HTTPURI.

VALID_STORAGE_ARGS = ['params', 'headers', 'cookies', 'auth', 'timeout', 'allow_redirects', 'proxies', 'verify', 'stream', 'cert', 'method']

Keyword arguments passed to requests.request().

__init__(url, raise_for_status=True, method=None, storage_args={})[source]
Parameters:
  • uri (str) – HTTP URI.
  • raise_for_status (str) – Raises a requests.RequestException when the response status code is not 2xx (i.e., calls requests.Request.raise_for_status())
  • method (str) – Overrides the default method for all HTTP operations.
  • storage_args (dict) – Keyword arguments that are passed to requests.request()
dir_exists()[source]

Makes a HEAD requests to the URI.

Returns:True if status code is 2xx.
make_dir()[source]

Ignored.

put_content(content)[source]

Makes a PUT request with the content in the body.

Raise:An requests.RequestException if it is not 2xx.

AWS Simple Notification Service

class uriutils.storages.SNSURI(topic_name, region, storage_args={})[source]

Storage system for AWS Simple Notification Service.

SUPPORTED_SCHEMES = set(['sns'])

Supported schemes for SNSURI.

VALID_STORAGE_ARGS = ['Subject', 'MessageAttributes', 'MessageStructure']

Keyword arguments passed to SNS.Client.publish().

__init__(topic_name, region, storage_args={})[source]
Parameters:
  • topic_name (str) – Name of SNS topic for publishing; it can be either an ARN or just the topic name (thus defaulting to the current role’s account)
  • region (str) – AWS region of SNS topic (defaults to current role’s region)
  • storage_args (dict) – Keyword arguments that are passed to SNS.Client.publish()
dir_exists()[source]

Not supported.

download_file(filename)[source]

Not supported.

exists()[source]
Returns:True if the SNS topic exists
get_content()[source]

Not supported.

put_content(content)[source]

Publishes a message straight to SNS.

Parameters:content (bytes) – raw bytes content to publish, will decode to UTF-8 if string is detected