In this page you will find documentation about the different Object Storage clients that are available. Object Storage can be addressed with the S3 and Swift protocols.

More detailed information on the compatibility of the Ceph RGW S3 implementation may be found at https://docs.ceph.com/en/latest/radosgw/s3/

Authentication

In order to use the service you need a user name and a password. Ceph RGW supports so-called v3 authentication. For v3 authentication, you have Keystone projects, also knows as tenants. Unless we agree upon something else, your project name is the same as your user name. In v3 authentication there are domains for both users and projects. If you have an account in SURFsara’s Central User Administration (CUA) system and have been granted access to the service, then both your project domain as your user domain is CuaUsers. When you only have a local account, then both domains are equal to Default.

In order to be able to use the S3 protocol, you need to get to create so-called EC2 credentials, i.e. an access key and a secret key. How this works is described in EC2 credentials for S3.

Command line clients

• S3

  • S3cmd

  • AWS S3 client (awscli)

  • S5cmd, for parallel file transfers

  • rclone

• Swift

  • Python SWIFT client

  • Curl

  • rclone

GUI's

• Cyberduck

Mounted file systems

• S3QL: It is possible to mount object storage as a file system with s3ql both using the S3 and Swift protocol. S3QL has features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival. The s3ql page has more information.

• S3FS: It is possible to mount object storage as a file system with s3fs using the S3 protocol. The s3fs page has more information.

• Rclone: Using rclone mount you can use rclone to mount SWIFT using the S3 and Swift protocol.

• Goofys: Goofys is a tool written in golang to mount S3 object stores. Information may be found at the goofys page.

Backups

Duplicity: Duplicity is a backup tool that amongst others supports cloud storage systems. More information about this can be found at the duplicity page.

Synchronisation

Rclone: Rclone is the rsync for cloud storage. Here is more information on how to use rclone on the rclone page.

Libraries and APIs

• Python library: The python library page gives you information on the python-swiftclient library.

• REST API: SWIFT offers a REST API. Information about this API and some examples are described at the API page.

• Boto3: Boto3 is the AWS SDK for python developed by Amazon. It contains of course an S3 part to access object stores. At boto3 has information on how to use it.