Meant for:

  • iRODS admins
  • iRODS users

Requirements:


We use the offline facilities that the SURF Data Archive service offers. For reference, we keep here a link to the DMF commands, which display, among other information, the status that a file is in: Data Archive: DMF commands#CheckingFileStatus

Data objects stored through iRODS on SURF Data Archive will (in time) also be put offline on tape according to the policy dictated by the SURF Data Archive service. When that has occurred and you try to get/download the file using iCommands or any other iRODS client, the get request gives an error, because iRODS rules will flag that the data is offline and iRODS will cut the connection as it can not be retrieved without staging the file back online.

Do you wish to download offline data?

iget /your/object/in/iRODS

This command will fail when the file is offline. You can check the status of a file and bring it online in the following sections.

Checking the status of a file

Check the metadata of the object for additional information:

$ imeta ls -d /your/object/in/iRODS
AVUs defined for dataObj /your/object/in/iRODS:
attribute: SURF-BFID
value: 5362ssdfsdf000232100031b2af83532 #not a real value
units: 
----
attribute: SURF-TIME
value: 233143446291 #not a real value
units: 
----
attribute: SURF-DMF
value: OFL
units:

The metadata attribute 'SURF-DMF: OFL' indicates that the data object is offline and 'SURF-BFID' and 'SURF-TIME' are additional metadata used by the iRODS ruleset to process the status of the data object.

In order to stage the data object to ONLINE, so that you can download the file, you can use the DMF tools installed on the iRODS resource server which is connected to the SURF Data Archive.

Pay attention!

Note that the metadata of the iRODS object will not be updated until another get request is made. i.e. it will not be updated by default when it is online and the metadata information might not be up to date. See the last section below.

Bringing a file back from offline

With a simple iRODS rule (as below), we can initiate the staging of the file from offline (OFL) to online (DUL):

Note

In the following code, the *dataPath variable actually is pointing to a physical path in the file system (i.e.: not an iRODS logical path)

arcGet{
    msiExecCmd("dmget", *dataPath, "your-irods-instance.irods.surfsara.nl", "", "", *dmRes);
    msiGetStdoutInExecCmdOut(*dmRes,*Out)
    writeLine("stdout",*Out);
}
INPUT *dataPath="/nfs/archivelinks/irtestuser/your/object/in/iRODS"
OUTPUT ruleExecOut

Note that the argument, `*dataPath`,  for iRODS Rule is not the logical iRODS path, but the physical path of the object. This argument can be a space-separated list of objects:

arcGet{
    msiExecCmd("dmget", *dataPath, "your-irods-instance.irods.surfsara.nl", "", "", *dmRes);
    msiGetStdoutInExecCmdOut(*dmRes,*Out)
    writeLine("stdout",*Out);
}
INPUT *dataPath="/nfs/archivelinks/irtestuser/your/object/in/iRODS1 /nfs/archivelinks/irtestuser/your/object/in/iRODS2 /nfs/archivelinks/irtestuser/your/object/in/iRODS3"
OUTPUT ruleExecOut

Pay attention!

Note that the hostname "your-irods-instance.irods.surfsara.nl" is not necessarily the main endpoint of your iRODS service (typically the iCAT server), it is the hostname of the iRODS resource server where the SURF Data Archive is attached. For example, it could be something like myorg-resc.irods.surfsara.nl or myorg-data.irods.surfsara.nl.

Either you can invoke the rule manually:

irule -F arcGet.r
irule -F arcGet.r "*dataPath='/some/other/physical/path1 /yet/another/path2'"

or you can use this simple rule as a component in a more complex iRODS rule structure.

Viewing the file's status in the file system

If you want to discover when an object is online you can poll its status, bypassing the iRODS metadata, using this rule:

arcAttr{
    msiExecCmd("dmattr", *dataPath, "your-irods-instance.irods.surfsara.nl", "", "", *dmRes);
    msiGetStdoutInExecCmdOut(*dmRes,*Out)
    writeLine("stdout",*Out);
}
INPUT *dataPath="/nfs/archivelinks/irtestuser/your/object/in/iRODS"
OUTPUT ruleExecOut

if you write it in a file called dmattr.r and execute it, you will get an output similar to this:

irule -F dmattr.r "*dataPath='/nfs/archivelinks/irtestuser/your/object/in/iRODS'"
615ae5d000000000001eee85+DUL+14528512+14526536

The acronym "DUL" indicates that it is online. If "UNM" appears, it means that the system is in the process of staging the object online (i.e., the copy is still in progress).

Viewing the file's status in the iRODS metadata

Once a file has been offline and successfully brought back online, the iRODS metadata will not yet reflect the new status.

You can see that a file is successfully brought back online by looking at the file's status in the file system (as explained in the previous section).

In this scenario, a new iget will succeed, and then the iRODS metadata will also reflect the new status:

iget /your/object/in/iRODS

You can now check the new status in the iRODS metadata:

$ imeta ls -d /your/object/in/iRODS
...
attribute: SURF-DMF
value: DUL
...

See that the new value is DUL.

  • No labels