As Snellius is a new system and was newly installed and configured some things might not be fully working yet and are still in the process of being set up. Here, we list the issues that are known to us and that you don't have to report to the Service Desk. Of course, if you encounter issues on Snellius not listed here then please let us know (through the Service Desk).
Resolved issues (updated 22-11-22 21:20)
- The out-of-memory issues with the foss-versions of QuantumESPRESSO, VASP and CP2K have been resolved
Infiniband and file system performance
We found that the infiniband connections are not always stable, and may be underperforming. Similarly, the GPFS file systems (home, project and scratch) are not performing as expected. Reading/writing large files performs as expected, but reading/writing many small files is slower than expected. Among other things, this can affect the time it takes to start a binary, run commands, etc.
We are looking into both issues.
Using NCCL for GPU <=> GPU communication
NCCL is a communication library that offers optimized primitives for inter-GPU communication. We have found that it often hangs during initialization on Snellius. Probability of a hang during init increases with the amount of GPUs in the allocation. The issue is that NCCL sets up its communication using an ethernet-based network interface. By default, it selects the 'ib-bond0' interface, which supports IP over the infiniband network in Snellius. This interface seems to be experiencing issues however.
As a workaround, you can configure NCCL to use the traditional ethernet interface, which on Snellius GPU nodes is called 'eno1np0', by exporting the following environment variable
Note that if you use mpirun as launcher, you should make sure that it gets exported to the other nodes in the job too
(note that when launching your parallel application with srun, your environment gets exported automatically, so the 2nd step is not needed).
Impact on performance of this workaround is expected to be minimal: the traditional ethernet interface is only used to initialize the connection. Any further NCCL communication between nodes is performed using native infiniband.
Cartopy: ibv_fork_init() warning
Users can encounter the following warning message, when import "cartopy" and "netCDF" modules in Python:
The issue is similar to the one reported here. The warning will disappear if "cartopy" is imported before "netCDF".
Another solution is to disable OFI before running the python script:
Attaching to a process with GDB can fail
gdb -p <pid> (or the equivalent
attach <pid> command in gdb) to attach to a process running in a SLURM job, you might encounter errors or warnings related to executable and library files than cannot be opened:
Such issues will also prevent symbols from being resolved correctly, making debugging really difficult.
The reason that this happens is that processes in a SLURM job get a slightly different view of file system mounts (using a so-called namespace). When you want to attach GDB to a running process and use SSH to log into the node where the process is running, the
gdb process will not be in the same namespace, causing GDB to have issues to directly access the binary (and its libraries) you're trying to debug.
The workaround is to use a slightly different method for attaching to the process:
$ gdb <executable>
(gdb) set sysroot /
(gdb) attach <pid>
For the example above, to attach to
/usr/bin/sleep (PID 1054730) the steps would become:
Project space quota exceeded on Snellius
During the data-migration of the project spaces contents from Cartesius to Snellius, there was a considerable time (weeks) in which there was no freeze on the (Cartesius) source side, in which users kept working, causing files to be created, and sometimes also deleted. During the resync runs, the migration software has kept things in sync by "soft deleting" files that were migrated but at a later stage appeared to be no longer present on the Catesius side. That is: rather than actually removing them, the AFM software moved them into a .ptrash directory.
Depending on the productivity and rate of data turnover of users in the last weeks of Cartesius, this can add up and cause "quota pressure" for users. The .ptrash directories are owned by root. Snellius system administration has to delete them, users cannot do it themselves. Several quota-related service desk tickets may have this as their root cause - but only for project spaces. Home directories were migrated in a different manner and do not have this issue.
Allocating multiple GPU nodes
Normally, batch scripts like
Should get you an allocation with 2 GPU nodes, 8 gpus, and 4 MPI tasks per node. However, right now, there is an issue related to specifying an amount of GPUs larger than 4: jobs with the above SBATCH arguments that use OpenMPI and call srun or mpirun will hang.
Instead of specifying the total number of GPUs, please specify the number of GPUs per node, combined with the number of nodes instead. E.g.
This will give you the desired allocation with a total of 2 GPU nodes, 8 gpus, and 4 MPI tasks per node, and the srun (or mpirun) will not hang.