SURF User Knowledge Base
Space shortcuts
Page tree

Introduction

PyCharm is a popular IDE for the Python programming language, and is developed by Jetbrains.
Remote development means running the back-end of a development environment on a remote device other than the user's personal device.

Remote development using PyCharm is officially supported by Jetbrains, although still in Beta at the time of writing this document.


Prerequisites

It is assumed that:

  • The user is already somewhat familiar with the PyCharm IDE
  • The user has access to either PyCharm Professional or PyCharm Edu
  • The user is comfortable manually editing configuration files

Table of Contents

Configuring your ssh connection to the batch nodes

In order to be able to run the PyCharm backend on a batch node, PyCharm needs to know how to connect to it. Since connecting directly to batch nodes is not possible, we need to configure the login node as a proxy in your SSH config.
Essentially, this means that your computer will SSH to the login node, which will then forward the connection to the batch node.

On Unix-based systems, your SSH config is typically located at ~/.ssh/config.
Similarly, on Windows, your SSH config will probably be located at %userprofile%\.ssh\config.

Add the following snippet there:

Snellius

Host snellius
  Port 22
  HostName snellius.surf.nl

Host tcn*
  HostName %h
  ProxyJump snellius

Changing node type

Change tcn to the name of the node 'flavour' you actually want to allocate. You can find a list of node flavours here

Lisa

Host lisa
  Port 22
  HostName lisa.surfsara.nl

Host r*n*
  HostName %h
  ProxyJump lisa

All Lisa nodes have the format r*n*, for example: r38n4

Allocate an interactive node

This process is more broadly described here, but it boils down to allocating an interactive node using the salloc command:

Interactive jobs
Alternatively, SLURM provides a way to allocate resources with the salloc command, e.g.:
salloc -n 32 -t 4:00:00

SLURM will inform you whether the node allocation has been succesfull:

robertsc@login4:~$ salloc -n 1 -c 4 -p shared -t 06:00:00
salloc: Pending job allocation 8485336
salloc: job 8485336 queued and waiting for resources
salloc: job 8485336 has been allocated resources
salloc: Granted job allocation 8485336
salloc: Waiting for resource configuration
salloc: Nodes r38n4 are ready for job

Pay attention to the name of the node (r38n4 in this example) that gets allocated to you, you'll need it later.

You will be fully charged for the allocated resources, even if you are not actively running any Python scripts through PyCharm. Consider how many cores you really need for testing your code, since with interactive work you are leaving resources idle most of the time.

Connect PyCharm to the allocated node

Click "Connect via SSH"

Click the icon next to the <New Connection> bar,

and then the button on the next page.

In the SSH configurations screen:

  1. Enter your username
  2. Fill in the node name as Host
  3. set "Authentication type" to "OpenSSH config and authentication agent"
  4. click "OK".

Select the added configuration from the drop-down menu, and then click "Check connection and Continue", after which a connection to the node will be made

The first time you connect to a specific node, you will be asked if you trust the RSA key. Press "Yes".

You will be asked to fill in your password twice: once to login to Snellius/Lisa, and once to login to the allocated node.

In the drop-down menu, select PyCharm as IDE.

Next, fill in the directory path of your project as "Project directory".

Click "Download and Start IDE".

PyCharm will download and launch itself on the interactive node.
If everything went right, we find ourselves in the familiar PyCharm environment:

It is worth pointing out that looking in the bottom right of our Development Environment, PyCharm displays the bits/second it is sending and receiving from our node.

As a double check, we can see that PyCharm launched itself on the node if we run the htop command on the interactive node:

Adding a Python interpreter to PyCharm

In general, adding a Python interpreter to your remote PyCharm project works no different than adding one to a local project.

To add a Python interpreter to your PyCharm project, first go to the general PyCharm settings, and then under "Project" go to "Python interpreter".
There, you can select the flavour of environment you're used to.

We will list some ways to add a Python interpreter to your project which may not immediately be intuitive.

System Python (i.e. from a module)

We need to find the path to the Python binary, which loaded when we execute module load Python/<xxx>.
We can find out its exact location with the which command like so:

$ module load 2021
$ module load Python/3.9.5-GCCcore-10.3.0
$ which python
/sw/arch/Debian10/EB_production/2021/software/Python/3.9.5-GCCcore-10.3.0/bin/python

Then, paste the above path at System Interpreter → triple dots '...' menu → Select Python Interpreter

Existing Conda environment on Lisa/Snellius

Equally, we can find out the path to our Python binary with the with the which command, after activating our conda environment:

$ module load 2021
$ module load Miniconda3/4.9.2
$ conda activate 2D-VQ-AE-2
$ which python
/home/robertsc/.conda/envs/2D-VQ-AE-2/bin/python

The correct conda executable path should select itself automatically.

Development

You should now be setup to work with your project!
Any time you execute any code, your code will be running on your allocated node.

Good luck!