Installation
This page will provide information on how to install the python swift client and how to use it. The page Installation Instructions of the Python SWIFT Client on Linux tells you how to install the client on various flavours of Linux.
For windows it is rather similar. You can read about installation on windows at Installation Instructions of the Python SWIFT Client on Windows
For information on how to install de swift command line client on Mac OSX, please, have a look at: Installation Instructions of the Python SWIFT Client on OSX.
Containers versus buckets
OpenStack uses the concept of containers rather than buckets. However, these are technically the same. You can access buckets created with S3 using the Swift protocol and vice versa.
Usage
Now the usage of the swift commandline tools is like:
swift [options] <command> [--help] [<command options>]
More details and examples are provided below.
Authentication
The following environment variables are useful to set if you don’t want them to provide them all the time on the command line.
export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=<my project> export OS_USERNAME=<user name> export OS_PASSWORD=<password> export OS_AUTH_URL=https://objectstore.surf.nl:5000/v3 export OS_IDENTITY_API_VERSION=3
This holds for local keystone users. Users using their account in the SURF Central User Administration (CUA) through keystone need the specify the following:
export OS_PROJECT_DOMAIN_NAME=CuaUsers export OS_USER_DOMAIN_NAME=CuaUsers
for the OS_PROJECT_DOMAIN_NAME and OS_USER_DOMAIN_NAME environment variables. Apart from using your user name and password, it is also possible to generate a token that is valid for 24 hours. This may be handy if you are running the script elsewhere on a batch system and you don’t want to send you username and password with your batch job. You can use this token to access your data in SWIFT.
Setting the environment variables as shown above holds for Linux and Mac OSX. For Windows you may want to have a look at: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables?view=powershell-7.3
You can get a token in the following way:
$ swift stat -v StorageURL: https://objectstore.surf.nl/swift/v1/AUTH_45d98b7704ad4bcabefeda0ae3dc2547 Auth Token: gAAAAABmlQvr29-vzK-CmAduMdX1NYts3My6ueqe5GnDKqkoB0lpd0e8QWMkraZ3crOak4cymKseAKSoxIgOwj_MCf4LxSoUSIKBBYPZy5zBcrIeQYe0__Xbu5Jci0JOKVlF1vqp-tI45Syt758YYhcm9JmkB9PvePLLrh-eGb3xaUNoIIJBYtM Account: AUTH_45d98b7704ad4bcabefeda0ae3dc2547 Containers: 30 Objects: 1203 Bytes: 92361494529 Containers in policy "default-placement": 30 Objects in policy "default-placement": 1203 Bytes in policy "default-placement": 92361494529 Objects in policy "default-placement-bytes": 0 Bytes in policy "default-placement-bytes": 0 X-Timestamp: 1721043956.87719 X-Account-Bytes-Used-Actual: 92362452992 X-Trans-Id: tx000006b019c5ac17b37a9-0066950beb-abc2f-default X-Openstack-Request-Id: tx000006b019c5ac17b37a9-0066950beb-abc2f-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 Strict-Transport-Security: max-age=15768000
What you need is the StorageURL and the Auth Token. You can use these two to run the swift commands for the next 24 hours without supplying your user name and password.
swift --os-auth-token <TOKEN> --os-storage-url <STORAGE URL> [options] <command> [--help] [<command options>]
If you already know the storage url and only need the token you could also use:
$ openstack token issue +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | expires | 2024-07-16T11:50:49+0000 | | id | gAAAAABmlQ0Zd1UISb-fUix14WUTvueBcw7cxEApr2ed-Cvfss94eSE1rt1WOSXv2iOu3E6M67mWTa-iMmuUEPaaylf96pc1rRtEPIKWr29tgdnFieTkyCiyvJddvy_CqqHeBiordpZ29zuMRTyKgmvLDHX46BvgGrUZSDdKkZyVbikY7QqRfW4 | | project_id | 45d98b7704ad4bcabefeda0ae3dc2547 | | user_id | 4118ca9c8c544928a73042fd7be0e3b0 | +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Create a container
A container can be created by the following command:
swift post mycontainer
List the containers in an account
The containers in an account can be listed like:
swift list
List contents of a container
The objects in a container can be listed like:
swift list mycontainer
Upload/Download an object to/from a container
swift upload mycontainer myobject
If the container mycontainer does not exist yet, then it will be created. By default, the client will verify the checksum during the upload. Downloading an object from a container goes as follows:
swift download mycontainer myobject
Downloading the whole content of a container is done by:
swift download mycontainer
Throwing containers and objects away
Delete one object from a container:
swift delete mycontainer myobject
Delete a container with all objects in it:
swift delete mycontainer
Getting metadata
Container metadata can be obtained in the following manner:
$ swift stat mycontainer Account: AUTH_45d98b7704ad4bcabefeda0ae3dc2547 Container: mycontainer Objects: 2 Bytes: 1233704 Read ACL: Write ACL: Sync To: Sync Key: X-Timestamp: 1721044468.54386 X-Container-Bytes-Used-Actual: 1241088 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Mon, 15 Jul 2024 11:54:28 GMT X-Trans-Id: tx00000ef0afe9d397bb7b6-0066950e34-abc2f-default X-Openstack-Request-Id: tx00000ef0afe9d397bb7b6-0066950e34-abc2f-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 Strict-Transport-Security: max-age=15768000
Bytes is the total number of bytes of all object in the container, Objects is the number of objects in the container.
Object metadata can be obtained by the following command:
$ swift stat mycontainer myobject.zip Account: AUTH_45d98b7704ad4bcabefeda0ae3dc2547 Container: mycontainer Object: myobject.zip Content Type: application/zip Content Length: 4157 Last Modified: Mon, 15 Jul 2024 11:59:03 GMT ETag: b6548553bac130ebb6395dd229c198a4 Meta Mtime: 1721044728.183286 Accept-Ranges: bytes X-Timestamp: 1721044743.20849 X-Trans-Id: tx000007c682544412e9b27-0066950f0d-9270f-default X-Openstack-Request-Id: tx000007c682544412e9b27-0066950f0d-9270f-default Strict-Transport-Security: max-age=15768000
Content Length is the size in bytes and ETag is the md5 checksum of the object.
Set your own metadata for containers and objects
To set metadata for an container use this:
$ swift post mycontainer --meta foo:bar
$ swift stat mycontainer | grep Meta
Meta Foo: bar
Setting metadata for an object works in an identical fashion.
Copying objects
It is possible to copy objects. This goes as follows:
$ swift copy --destination /copycontainer/copied.zip mycontainer myobject.zip created container copycontainer mycontainer/myobject.zip copied to /copycontainer/copied.zip
Also the object’s metadata will be copied, unless you use the --fresh-metadata flag.
Renaming containers and objects
Important
It is NOT possible to rename a container. This means that you have to think really well about naming containers before you upload a PB of data.
It is possible to rename an object but not in the classical sense. First you need to copy an object using, for example, the method above and then throw the original object away.
Uploading large files (>5GB)
It is only possible to upload objects with the size of at most 5GB in one go to SWIFT. It is possible to up and download larger objects when the large object is uploaded in chunks. For the python SWIFT client you can upload an object larger than 5GB in the following way:
swift upload --use-slo -S <chunk size in bytes> mycontainer myobject
For example uploading a 6GB file using 1GB chunks:
$ swift upload --use-slo -S 1000000000 mycontainer 6GB.object 6GB.object segment 2 6GB.object segment 1 6GB.object segment 4 6GB.object segment 3 6GB.object segment 0 6GB.object segment 5 6GB.object
For downloading you can just proceed as usual.
There are Dynamic Large Objects and Static Large Objects when it comes to large object uploads. The curl page has some information on this. Both type of objects have their use cases. Dynamic Large Objects may have issues with end to end integrity of data which Static Large Objects don’t. Therefore we recommend to use the --use-slo flag.
Object versioning
You can store multiple versions of your content so that you can recover from unintended overwrites. Object versioning is an easy way to implement version control, which you can use with any type of content.
The first thing you have to do is create a container where old versions of objects are stored. You can choose an arbitrary name.
swift post mycontainer_versions
Then you have to update the container and tell swift where to store the older versions:
swift post mycontainer -H "X-Versions-Location:mycontainer_versions"
If you upload an object to a container and after that, upload a newer version of an object to the same container. The older version of the object is placed an a separate container. In this case that container would be mycontainer_versions under a name like:
<hexadecimal length of object name><object name><timestamp>
If you throw the latest version of the object away, the second latest version of the object is placed back into the container.
Below is an extended example:
$ swift post mycontainer $ swift post mycontainer_versions $ swift post mycontainer -H "X-Versions-Location:mycontainer_versions" $ echo "hello" > myobject $ swift upload mycontainer myobject myobject $ echo "goodbye" > myobject $ swift upload mycontainer myobject myobject $ swift list mycontainer_versions 008myobject/1721047418.750494 $ swift download mycontainer myobject myobject [auth 0.080s, headers 0.166s, total 0.166s, 0.000 MB/s] $ cat myobject goodbye $ swift delete mycontainer myobject myobject $ swift download mycontainer myobject myobject [auth 0.082s, headers 0.172s, total 0.172s, 0.000 MB/s] $ cat myobject hello
Object expiration
You can set objects to expire. This means that object will be automatically deleted after a certain period of time. More information on this may be found at: https://docs.openstack.org/swift/latest/api/object-expiration.html.
ACLs
You can set ACLs on containers. Using container ACLs you grant different levels of access to individual containers. More information on this is available at: https://docs.openstack.org/swift/latest/overview_acl.html.