All data on the Data Archive is managed by ScoutAM (Archive Manager) which interacts with the ScoutFS (File System). This solution is provided by Versity. When you upload your data, it is stored on a ScoutFS managed file system. This file system is an “online” disk pool that is easy and fast for users to access and retrieve their data from. However, the data will not stay there. ScoutAM will automatically trigger a migration procedure where the files are individually scheduled for migration to the offline tape medium based on criteria that optimise file storage and retrieval. Once all the data has been safely stored on tape, the data on the disk file system may be removed to create space on the disk pool for incoming or outgoing data.

The DAcommands are a set of commands that can be used to track the status of the data and to move data between the disk pool and tape.

The DAcommands have been designed as close as possible to DMF commands link, the old command-line tools. For a detailed migration guide see the Migration page.

Data Archive Storage Guidelines

Your login account comes with:

  • 50MB of personal storage in your home directory
  • A symbolic link called archive inside your home directory (e.g., /home/username/archive)

IF you are part of other projects you will also find in your home directory:

  • Symbolic links to your project directories that point to the actual data storage location
  • Each project you have access to gets its own link

Example:

/home/username/              (Your 50MB home directory)
          ├── archive              (Link to your archive /archive/username)
          ├── project-alpha        (Link to Project Alpha's folder)
          └── project-beta         (Link to Project Beta's folder)

When you access /home/username/archive, you're actually accessing your archive folder. Similarly,
when you access /home/username/project-alpha, you're actually accessing your team's project-alpha project folder.


The following commands are available on the Data Archive Service node and Snellius:

  • darelease: Migrate files from the staging area to tape.

  • daget: Migrate files or parts of files (also called staging) from tape to the staging area.

  • dals: List files and directories and show the status of the files.

Below you will find how the DAcommands can assist you in various tasks on the archive. For full documentation about the DAcommands, see the follwing DAcommands Docs.

Checking File Status

To determine the status of the files within the current folder use dals:

dals -l
-rw------- 1 user1    user1     632792 Jul 26 1999 (OFL) file1
-rw------- 1 user1    user1     632792 Jul 27 1999 (OFL) file2
-rw------- 1 user1    user1      15884 Jul 27 1999 (REG) file3
-rw------- 1 user1    user1     632792 Aug 2 1999 (DUL) file4
-rw------- 1 user1    user1     632792 Jun 19 23:20 (QUE) file5

Available states:

  • Regular (REG): user files residing only on disk

  • Queued (QUE): files that are being copied from disk to tape

  • Staging (STG): files which are being copied from tape to disk

  • Migrated files can be either of the following:

    • Dual-state (DUL): data resides both online and offline

    • Offline (OFL): data is no longer on disk

Directory status

Although directories are shown with statuses, they will always remain regular (REG) as they are not stored on tape. They are only defined in the file system itself.

Changing File Status

Staging Files

Staging refers to moving the data from tape to disk for subsequent processing. This is necessary for all file processing except for file deletion.

Retrieve a Specific File

Use daget to process the given file:

daget <filename>

Check that the desired file is now available on tape in DUL state:

dals -l <filename>

Retrieve Multiple Files at Once

Use daget and a wildcard to process all the matching files in a single command:

cd <folder>
daget *

Check that the desired files are now available on tape in DUL state:

dals -l <folder>

Retrieve an entire folder at once

To stage all files in a directory and its subdirectories:

daget -r <folder>

Retrieve Files Remotely from the Archive to Snellius/other remote systems

dmftar can directly unpack and restore files remotely from the Data Archive (provided key-based authentication and key-forwarding have been enabled). For more details about dmftar see Data Archive: Effective archive file management.

To preview the contents of archive file:

dmftar -t -f <login>@archive.surfsara.nl:<data>.dmftar


To restore a whole archive file use:

dmftar -x -f <login>@archive.surfsara.nl:<data>.dmftar


To restore a single file from the archive file remotely use:

dmftar -x -f <login>@archive.surfsara.nl:<data>.dmftar/ <data>/<data1>/data1-1.dat

Storing Files

The size and number of files stored on the archive are critical to its performance. As part of your workflow, you may want to migrate files to tape using darelease or check migration status using dals.

Important : It is generally best to let the Archive system decide when a file needs to be written to DUL or OFL status. We recommend discussing your use case with an advisor before using these commands. You can submit a ticket through the service desk.

Migrate a file to tape

Migrate the given file onto tape:

darelease <filename>

Check that the desired files are now available on tape in DUL/OFL state:

dals -l <file>

Migrate a directory to tape

To migrate all the files in a directory (and its subdirectories) to tape:

darelease <directory>

Important : this will only work with files that have DUL state , i.e., data that resides both online and offline. When a file has just been uploaded, it has a REG state. For performance and optimization, it's best that you let the backend stack decide when to migrate to tape.

Removing files

Removing files can be done using the regular remove command, no special command is required.

Remove a specific file

Use rm to remove the file:

rm <filename>

Remove an entire folder

Use rm to recursively remove the file:

rm -r <folder>


  • No labels