awscli
In this page you will find documentation about the Amazon S3 client.
Installation
See the instructions here: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
Configuration
First you need to create a file ~/.aws/config with the following content:
[default]
output = json
addressing_style = virtual
endpoint_url = https://objectstore.surf.nl
signature_version = s3v4
payload_signing_enabled = true
region = default
Apart from that, you have to create the file ~/.aws/credentials where your access and secret key lives. Such a file looks something like this:
[default]
aws_access_key_id = a11da7a5afea11da7a5afea11da7a5af
aws_secret_access_key = badf00ddeadbeefdeadbeefbadf00d11
Where you have to put in your own access and secret key. Don’t forget to:
chmod 600 ~/.aws/credentials
Finally you have to set the environment variables AWS_RESPONSE_CHECKSUM_VALIDATION and AWS_REQUEST_CHECKSUM_CALCULATION to "when_required". When you use linux you can do this by:
export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
Usage
Information on how to use the aswcli may be found at: s3 and s3api.
S3 Bucket versioning
It is possible to enable S3 bucket versioning. This can be enabled per bucket. If you do this then when you overwrite an existing object, then you can still retrieve the previously overwritten version. In addition, when you accidentally delete an object, then you can still restore the deleted object when you have bucket versioning enabled.
This means that you may end up with a lot of data in buckets where there is a lot of overwriting and deleting going on.
More detailed information on this may be found at: s3versioning. Below we will provide an example on how to do this with the AWS commandline client.
Enabling and suspending bucket versioning
You can enable and verify versioning on the bucket in the following way:
$ aws s3 mb s3://my-versioned-bucket
$ aws s3api put-bucket-versioning --bucket my-versioned-bucket --versioning-configuration Status=Enabled
$ aws 3api get-bucket-versioning --bucket my-versioned-bucket
{
"Status": "Enabled",
"MFADelete": "Disabled"
}
It is not possible to turn off bucket versioning completely. However, it is possible to suspend it. This can be done by:
aws s3api put-bucket-versioning --bucket my-versioned-bucket --versioning-configuration Status=Suspended
Restoring a deleted object
Now if you, by accident, delete an object, then you can still retrieve it. Suppose we do the following:
$ aws s3 ls s3://my-versioned-bucket 2024-07-15 10:43:17 6 important.object $ aws s3 rm s3://my-versioned-bucket/important.object delete: s3://my-versioned-bucket/important.object
What you can do next is list the versions of the object that are available:
$ aws s3api list-object-versions --bucket my-versioned-bucket
{
"Versions": [
{
"ETag": "\"aee97cb3ad288ef0add6c6b5b5fae48a\"",
"Size": 6,
"StorageClass": "STANDARD",
"Key": "important.object",
"VersionId": "OQeCh4SDbYnkJRxioar-VSCK506mGnX",
"IsLatest": false,
"LastModified": "2024-07-15T08:43:17.674Z",
"Owner": {
"DisplayName": "johndoe",
"ID": "45d98b7704ad4bcabefeda0ae3dc1547$45d98b7704ad4bcabefeda0ae3dc1547"
}
}
],
"DeleteMarkers": [
{
"Owner": {
"DisplayName": "johndoe",
"ID": "45d98b7704ad4bcabefeda0ae3dc1547$45d98b7704ad4bcabefeda0ae3dc1547"
},
"Key": "important.object",
"VersionId": "kWjO5q9HKr3cMqu7S2T-o5ZiGAsAmOC",
"IsLatest": true,
"LastModified": "2024-07-15T08:44:44.593Z"
}
]
}
Here you see a so-called “Delete Marker” indicating that the object has been deleted. In order to restore the deleted object you simply have to delete this Delete Marker and you will have your object back. This is done in the following way:
$ aws s3api delete-object --bucket my-versioned-bucket --key important.object --version-id "kWjO5q9HKr3cMqu7S2T-o5ZiGAsAmOC"
{
"DeleteMarker": true,
"VersionId": "kWjO5q9HKr3cMqu7S2T-o5ZiGAsAmOC"
}
$ aws s3 ls s3://my-versioned-bucket
2024-07-15 10:43:17 6 important.object
Working with multiple versions
Suppose we are uploading different versions of a file with the same name as shown below:
$ echo "Good morning" > versioned.object $ aws s3 cp versioned.object s3://my-versioned-bucket/ upload: ./versioned.object to s3://my-versioned-bucket/versioned.object $ echo "Good afternoon" > versioned.object $ aws s3 cp versioned.object s3://my-versioned-bucket/ upload: ./versioned.object to s3://my-versioned-bucket/versioned.object $ echo "Good evening" > versioned.object $ aws s3 cp versioned.object s3://my-versioned-bucket/ upload: ./versioned.object to s3://my-versioned-bucket/versioned.object
After that we can list these versions like so:
$ aws s3api list-object-versions --bucket my-versioned-bucket
{
"Versions": [
{
"ETag": "\"fbd4309eb57f3d71a573201934425d10\"",
"Size": 13,
"StorageClass": "STANDARD",
"Key": "versioned.object",
"VersionId": "3y5N.enY2vlYqKxNvbkqJbBUEjuhrjV",
"IsLatest": true,
"LastModified": "2024-07-15T09:09:22.170Z",
"Owner": {
"DisplayName": "johndoe",
"ID": "45d98b7704ad4bcabefeda0ae3dc1547$45d98b7704ad4bcabefeda0ae3dc1547"
}
},
{
"ETag": "\"b0ef7a3470a76f1c1cd568e221705738\"",
"Size": 15,
"StorageClass": "STANDARD",
"Key": "versioned.object",
"VersionId": "b4iYFsWwzsuSGywYkw861pRs2Yonxm9",
"IsLatest": false,
"LastModified": "2024-07-15T09:09:10.781Z",
"Owner": {
"DisplayName": "johndoe",
"ID": "45d98b7704ad4bcabefeda0ae3dc1547$45d98b7704ad4bcabefeda0ae3dc1547"
}
},
{
"ETag": "\"8b20cea7247b2a56e2ffae98bc7d01fe\"",
"Size": 13,
"StorageClass": "STANDARD",
"Key": "versioned.object",
"VersionId": ".4UBkgIgTOd4MHsr7NIbbJpt40t-3w8",
"IsLatest": false,
"LastModified": "2024-07-15T09:08:56.437Z",
"Owner": {
"DisplayName": "johndoe",
"ID": "45d98b7704ad4bcabefeda0ae3dc1547$45d98b7704ad4bcabefeda0ae3dc1547"
}
}
]
}
Then we can get the most current version:
$ aws s3 cp s3://my-versioned-bucket/versioned.object versioned.object download: s3://my-versioned-bucket/versioned.object to ./versioned.object jeanmb@jeanmb-laptop:~/Desktop$ cat versioned.object Good evening
You can retrieve an earlier version:
$ aws s3api get-object --bucket my-versioned-bucket --key versioned.object --version-id ".4UBkgIgTOd4MHsr7NIbbJpt40t-3w8" versioned.object
{
"AcceptRanges": "bytes",
"LastModified": "Mon, 15 Jul 2024 09:08:56 GMT",
"ContentLength": 13,
"ETag": "\"8b20cea7247b2a56e2ffae98bc7d01fe\"",
"VersionId": ".4UBkgIgTOd4MHsr7NIbbJpt40t-3w8",
"ContentType": "binary/octet-stream",
"Metadata": {}
}
$ cat versioned.object
Good morning
Listing version information
You can list all available versions of a particular object including the version id, modification time and if it is the most current version or not in the following manner:
$ aws s3api list-object-versions --bucket my-versioned-bucket | jq -r '.Versions[] | select(.Key=="versioned.object") .Key+";"+.VersionId+";"+.LastModified+";"+(.IsLatest|tostring)' versioned.object;3y5N.enY2vlYqKxNvbkqJbBUEjuhrjV;2024-07-15T09:09:22.170Z;true versioned.object;b4iYFsWwzsuSGywYkw861pRs2Yonxm9;2024-07-15T09:09:10.781Z;false versioned.object;.4UBkgIgTOd4MHsr7NIbbJpt40t-3w8;2024-07-15T09:08:56.437Z;false
A maximum of 1000 objects are returned at one time. If you have more than 1000 objects in a bucket you may want to use a script like versions_listing.sh
This script displays the object name, version id, last modification time stamp, if it is the current version and if it is a delete marker.
The usage is:
./versions_listing.sh <bucket name> [object name]
If the object name is provided, it will list information on this particular object and if not, then it will list information on all objects in the bucket.
Cleaning up a versioned bucket
If you want to completely remove a bucket with versioning enabled, then you need to cleanup all versions of the objects and delete markers first. After that you can remove the bucket using:
aws s3 rb s3://<bucket name>
You can use this script to delete all versions in a bucket: delete_all_versions.py
Please use at your own risk.