All data on the Data Archive is managed by the Data Migration Facility (DMF). When you upload your data, it is stored on a DMF managed file system. This file system is an “online” disk pool that is easy and fast for users to access and retrieve their data from. However, the data will not stay there. The DMF server will automatically trigger a migration procedure where the files are individually scheduled for migration to the offline tape medium based on their size and age. Once all the data has been safely stored on two separate tapes, the data on the DMF file system may be removed to create space on the disk pool for incoming or outgoing data.


The DMF commands are a set of commands that can be used to track the status of the data and to move data between the disk pool and tape. The following DMF commands are available on the Data Archive Service node and Snellius:

  • dmput: Migrate files from the staging area to tape.

  • dmget: Migrate files or parts of files (also called staging) from tape to the staging area.

  • dmcopy: Copy all or part of an offline file to a destination file.

  • dmls: List files and directories and show the status of the files.

  • dmattr: Test for shell scripts to see whether a file is online or offline

  • dmfind: Search for files without having to retrieve them from tape


Below you will find how the DMF commands can assist you in various tasks on the archive. For full documentation about the DMF commands, see the following online manual pages.

Checking File Status

To determine the status of the files within the current folder use dmls:

dmls -l
-rw------- 1 user1    user1     632792 Jul 26 1999 (OFL) file1
-rw------- 1 user1    user1     632792 Jul 27 1999 (OFL) file2
-rw------- 1 user1    user1      15884 Jul 27 1999 (REG) file3
-rw------- 1 user1    user1     632792 Aug 2 1999 (DUL) file4
-rw------- 1 user1    user1     632792 Jun 19 23:20 (MIG) file5


Available states:

  • Regular (REG): user files residing only on disk

  • Migrating (MIG): files that are being copied from disk to tape

  • Unmigrating (UNM): files which are being copied from tape to disk

  • Migrated files can be either of the following:

    • Dual-state (DUL): data resides both online and offline

    • Offline (OFL): data is no longer on disk


Directory Status

Although directories are shown with statuses, they will always remain regular (REG) as they are not stored on tape. They are only defined in the file system itself.


You can also search for files in a given DMF state to, for example, determine which files need to be staged for retrieval using dmfind:

dmfind . -state ofl  -print
-rw-------    1 user1    user1     632792 Jul 26  1999 (OFL) file1
-rw-------    1 user1    user1     632792 Jul 27  1999 (OFL) file2

Changing File Status

Staging Files

Staging refers to moving the data from tape to disk for subsequent processing. This is necessary for all file processing except for file deletion.

Retrieve a Specific File

Use dmget to process the given file:

dmget -a <filename>

 

Check that the desired file is now available on tape in DUL state:

dmls -l <filename>

Retrieve Multiple Files at Once

Use dmget and a wildcard to process all the matching files in a single command:

cd <folder>
dmget -a *


Check that the desired files are now available on tape in DUL state:

dmls -l <folder>

Retrieve an entire folder at once

The input for dmget is always filenames, so to stage all the files in a directory (and it’s subdirectories) you can combine dmfind and dmget:

dmfind <folder> -state REG -o -stage MIG | dmget -a

Retrieve Files Remotely from the Archive to Snellius/Lisa

dmftar can directly unpack and restore files remotely from the Data Archive (provided key-based authenticationand key-forwarding have been enabled). For more details about dmftar see Data Archive: Effective archive file management.

To preview the contents of archive file:

dmftar -t -f <login>@archive.surfsara.nl:<data>.dmftar


To restore a whole archive file use:

dmftar -x -f <login>@archive.surfsara.nl:<data>.dmftar


To restore a single file from the archive file remotely use:

dmftar -x -f <login>@archive.surfsara.nl:<data>.dmftar/ <data>/<data1>/data1-1.dat

Storing Files

The size and number of files stored on the archive are critical to it’s performance. For more details about this and how to store files see Data Archive: Effective archive file management. As part of your workflow, you may want DMF to migrate files to tape using dmput or check migration status using dmls.


It is generally best to let the DMF system decide when a file needs to be written to DUL or OFL status. We recommend discussing your use case with an advisor before using these commands. You can submit a ticket through the service desk.

Migrate a file to tape

Migrate the given file onto tape (state DUL):

dmput <filename>


Migrate the given file onto tape and remove it from the disk pool (state OFL):

dmput -r <filename>

  

Check that the desired files are now available on tape in DUL/OFL state:

dmls -l <file>

Migrate a directory to tape

The input for dmput is always filenames, so to migrate all the files in a directory (and it’s subdirectories) to tape you can combine dmfind and dmput:

dmfind . -state REG | dmput


To remove files from the diskpool (i.e. OFL instead of DUL state) you can add the -r parameter to the dmput command. To force the command to wait until the migration is complete you can add the -w parameter to the dmput command.

Removing files

Removing files can be done using the regular remove command, no special dm command is required.

Remove a specific file

Use rm to remove the file:

rm <filename>

Remove an entire folder

Use rm to recursively remove the file:

rm -r <folder>

Table of contents