This section of the documentation is under review.

Multi-user buckets

Information on this usecase may be found at the multi-user buckets page.

Serving Static Web Pages

SWIFT offers the possibility to serve data in containers as a static web site. The staticweb page has more.

Uploading large files

When you want to upload a small number of large files (>1GB) then s5cmd would be a good choice. This tools uses parallelism in its uploads. Not only in terms of many concurrent uploads of a large number of files but it parallelises the upload of a single large file as well resulting in a very good performance. This can be done using the -c or --concurrency flag. This works as follows:


Uploading large data
s5cmd --endpoint-url https://objectstore.surf.nl cp -c 64 /path/to/my/huge/file s3://mybucket/mybigfile

In the example above the large file is concurrently uploaded in 64 chunks. But you may play around with this value to see what gives you the best performance.

Uploading many small files

When you want to upload a folder containing many small files, then rclone and s5cmd are suitable tools.

Rclone

Rclone uses by default 4 parallel transfers when copying a whole directory with files. For large files this may be fine but if the directory contains many small files (<10MB) it may be worthwhile to use a large number of concurrent transfers. For example, something like this:

rclone copy --transfers=100 -P /path/to/folder/with/small/files S3:mybucket

s5cmd

s5cmd --numworkers 256 --endpoint-url https://objectstore.surf.nl cp --destination-region NL /path/to/folder/with/small/files s3://mybucket

Here numworkers which is the number of parallel transfers, is by default 256. This is already quite suitable for the many small files use case. But also here, you may play around a bit with this value to see what’s best.

  • No labels