To migrate data between two instances (for example, between researchdrive.surfsara.nl and a branded instance), we advice to use rclone, which you can download here: https://rclone.org/downloads/


Define data source and destination

Generate a WebDAV Password

Before you can use WebDAV, go through the following steps how to generated you WebDAV password  first.


Configure rclone

After installing rclone, begin by opening a command line terminal

Start the rclone cofigurator:

rclone config
rclone config


First, give a clear name to your remote:

rclone config
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> source-rdrive


Now rclone will give you a (long) list of possible storages. Select 'Webdav'

rclone config
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Alias for a existing remote
   \ "alias"
 2 / Amazon Drive
   \ "amazon cloud drive"
(...)
50 / Union merges the contents of several upstream fs
   \ (union)
51 / Uptobox
   \ (uptobox)
52 / WebDAV
   \ (webdav)
(...)
Storage> 52


Now copy/paste the full URL from the webdav settings (that you got earlier)

rclone config
URL of http host to connect to
Choose a number from below, or type in your own value
 1 / Connect to example.com
   \ "https://example.com"
url> https://researchdrive.surfsara.nl/remote.php/nonshib-webdav/


Select owncloud

rclone config
Name of the Webdav site/service/software you are using
Choose a number from below, or type in your own value
 1 / Fastmail Files
   \ (fastmail)
 2 / Nextcloud
   \ (nextcloud)
 3 / Owncloud
   \ (owncloud)
 4 / Sharepoint Online, authenticated by Microsoft account
   \ (sharepoint)
vendor> 2


Copy/paste the username from the webdav settings (that you got earlier)

rclone config
User name
user> example_user


Copy/paste the password from the webdav settings (that you got earlier)

rclone config
Password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> y
Enter the password:
password:
Confirm the password: ********
password: ********


Depending on your rclone version, it may ask for advanced config or bearer token here. You do not need those:

rclone config
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n


Check the configuration and confirm if it is correct

rclone config
Remote config
--------------------
[source-rdrive]
type = webdav
url = https://researchdrive.surfsara.nl/remote.php/nonshib-webdav/
vendor = nextcloud
user = example_user
pass = *** ENCRYPTED ***
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y


Quit the configurator

rclone config
Current remotes:

Name                 Type
====                 ====
source-rdrive        webdav

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

repeat steps

You have to repeat the above steps for the destination environment


Testing your configuration

If all went well, you have a source and a destination defined:

rclone config
rclone listremotes
source-rdrive:
target-example:


You should be able to list your files:

rclone config
rclone ls "source-rdrive:/source_project (Projectfolder)/"
rclone ls "target-example:/destination_project (Projectfolder)/"


Migrating the data

See rclone performance considerations for some important performance tips!

Timeout

It is important to set the '--timeout' option high enough. As a rule of thumb, set it to 10 minutes for every GB of the biggest file in a collection. So if the biggest file you want to upload in a collection is 10GB, set --timeout 100m

This may look ridiculously large, but it provides a safe margin to avoid problems with timeout issues.

And you should be able to transfer your files:

rclone config
rclone -vvv copy --timeout 360m --use-cookies --transfers 16 "source-rdrive:/source_project (Projectfolder)/" "target-example:/destination_project (Projectfolder)/"





  • No labels