Category Archives: Getting Started

04 Partitions (queues) of zeus HPC

There is only one queue/partition named “all” (default) on zeus HPC. For more details see this post:

Queues of zeus [updated June 2018] | EEC High Performance Computing (coventry.domains)

 

Alex Pedcenko

02 GUI HPC VNC jobs for Windows Clients

For running GUI jobs on HPC compute node (rather than on login node), you can use my new Windows HTA-scripts from here: aa3025/hpc (coventry.ac.uk). They will also work via University VPN (Anyconnect)

##### optional step – usually already done when your HPC account is created ########

Prior to all the following you need to set your VNC password on the HPC (you can also change your VNC password this way):

  • login via ssh to HPC,
  • issue the command “vncpasswd” to change your VNC password (VNC password can be different to your HPC password, but it makes life easier if it is the same as your HPC password). Use this password for connecting to your VNC sessions in future.
####################################################

You can use these “scripts” to launch VNC (Remote Desktop) job on one of the HPC compute nodes if you need access to GUI (Graphical User Interface)

These scripts allow establishing VNC Desktop session on a compute node of HPC : (see also VIDEO screen cast here)

or here:

0) Download the whole distribution from aa3025/hpc (coventry.ac.uk) You have at least to download 2 folders: “tools” and either “zeus” or “pluto” depending on which HPC you use. You can use git for synchronising the whole distribution:

git clone https://github.coventry.ac.uk/aa3025/hpc.git
Alternatively you can download the zip archive from : HERE

1) Depending which HPC you are about to use, choose a folder zeus or pluto to launch VNC session on required HPC. Before proceeding further  check that HPC has free (idle) compute nodes available (“sinfo” in HPC console or check Zeus website or Pluto website)

2) Launch only one script: 01_START_VNC_Session_….._1_node.hta (by double-clicking on it), a.k.a. “submission app”

3) Fill in your HPC username, password and the rest of info. Hit “Submit” button, the script will then submit the “VNC” job to the HPC queue in order to start “vncserver” process [Remote Desktop session] on the assigned compute node.

4) Wait for your VNC job to start in the queue of HPC (you will see job listing in the newly opened black “plink” console window), note the target node name (it is also presented in the last column of the small table at the bottom of the submission app).

5) Once VNC job has started (“Running” state, “R”) in the black console,  you can close it. Make sure your VNC job is in Running state (also specified by “R” against your job ID in the table at the bottom of the submission app, if it is still in “PD” state (means still “pending”), waiting for resources to be allocated).

6) Once you closed the first (job submission) app and black console window, the second script will start automatically to get the ssh-tunnel running between your PC and the allocated compute node and will connect you to the node’s VNC session — fill in the target node name from the previous step and your HPC password. The VNC screen number is always :1 (you can’t change it).

7) Now, the VNC password window will pop up — enter your VNC password there (theoretically it can be different from your HPC password). If you do not have VNC password, return to the top of this page (“Prior to all of the following…”) and set it up.

8) If all went well, the VNC Remote Desktop windows will be presented. You can toggle it to full screen by pressing F8 on your keyboard (amongst some other options in F8 as well).

By default the VNC job will last for 24 hours and will request 1 whole node, you can select different time and less CPUs than on the whole node or more than one compute node to be allocated.

More than one compute node. If you specify more than one node in VNC job submission app, the VNC Desktop session will start on the first allocated node, but the resources of all other allocated nodes will still be available from your VNC Desktop session.

VNC job termination

When the slurm VNC job expires (the time limit of your job [normally 24 hours if you did not change it] was reached), your VNC Desktop session will be terminated.

However, if you wish to terminate the VNC job before it expires (you do not use it any more and need to free the HPC resources), you can either just kill it from ssh console with “scancel JOBID” command, or by selecting “Logout” in your VNC Desktop session and your slurm job with VNC sessions will terminate.

If you just close the VNC viewer window on your PC –> the VNC Desktop session will remain running until termination time is reached or you cancel your VNC job. Make sure you do not waste HPC resources by having idle VNC sessions’ jobs hanging in the queue!

Alex Pedcenko

03 Disk Space and User Quotas

As you may be aware, we are experiencing constant disk space shortage on zeus HPC. To address this issue, the disk quotas have been introduced for user home folders. Each user is can be entitled (if space permitting) to 200 Gb of disk space. You can exceed this amount up to 1Tb for up to 7 days (grace period). After that you may no longer add files to your home folder until you clean it below original 200Gb threshold.
Remember that you should not use the HPC disk space for storing your files, please store only the files which are necessary for currently running jobs and/or post processing results etc..

For storing temporary files, job results and even submitting new jobs you can use fast scratch space here /beegfs/users/yourHPCusername

This space has no quota, but it is not backed up, so it should be only used for current jobs and projects, not for storing data!

 

You can look check how much your home folder is consuming here: http://zeus.coventry.ac.uk/space.php quotas’ status and disk usage are updated hourly.

What if I’m above the quota, but I still need my files and have no means to offload them from HPC?

At the moment we are regularly backing up the content of all /home/ folders onto external network drive (NAS drive). If you need to keep your files, which are currently in your home folder and are above the user quota and you have no means to store them anywhere else, please let me know (email me: aa3025@coventry.ac.uk) and I can disable backing up your home folder on zeus HPC. This way you will have bulk of your files already stored on the backup drive and can delete them from your zeus home folder, leaving only the files which are necessary for currently running jobs. When you delete the files from your home folder on Zeus, they will be automatically (with some delay of 1 day or so) deleted from the backup drive. So if you need to keep them, please let me know for disabling overwriting of the backup copy.

Then you can be given an access to backup drive to retrieve your files when needed.

Best Regards,
Alex Pedcenko

07 Midnight Commander for the Rescue of a Novice Linux User

If you just starting with Linux command line, the “Midnight Commander” file manager will be quite helpful for you to ease few simple tasks. You can launch it with a command mc You will see something like this:

mc-panels

 

The interface of “mc” are two panels with content of any 2 folders you navigate to. To switch between the panels (left/right) use TAB key on your keyboard. You can also use mouse if your ssh-session is capable of this. MC allows to do following things (key to execute in brackets, “F” refers to “function” keys – top row on your keyboard):

  1. Navigate through files and folders (just select folder you want to go with arrows and press [ENTER]). To go “up” one folder select “..” on top of the folder listing and press [ENTER].
  2. To select multiple files/folders on current panel use “Ins/0” key on your NUMPAD (small numericl panel on the right of the keyboard) panel. NumLock must be OFF!
  3. Select All files in folder (“+ and then Enter”). Can also select files of certain extension in the same way.
  4. View content of a file: select file and press (F3)
  5. Edit file: select file and press (F4)
  6. Copy file or folder from one folder to another (i.e. from one panel to the opposite): select file or folder and press (F5)
  7. Move files/folders (select and press F6)
  8. Make new folder in the current folder/panel: F7
  9. Delete file/folder: select file or folder you want to delete and press (F8) [ cannot be undone ]
  10. (F9) go into top menus if you do not have mouse functionality, navigate through using arrows.
  11. (F10) exits MC.

You can still use “mc” session as a command line session: just press Ctrl+O to toggle between command line and mc panels. If you start typing while in panel mode, the command appears underneath the panels. Press Enter to execute it in the currently active panel of mc. Typing “exit” will close mc. Type “cd” {ENTER} to change to the top of your home folder. Type “cd foldername” to change to the “foldername” (sometimes it is faster than using “arrows/PgUp/PgDown”  keys.

While in one of the panels try pressing (F2) you will see what you can do with currently selected file or folder (the most useful is compress). MC can also decompress various archives (zip/rar/tar/gzip/z… etc). Just select the achieve and press [ENTER] – it will show its content. Select the files you want to de-compress and copy them to the opposite panel.

If you going to use MC you will find much more features soon and wont be able to work in command line without it.

regards,

Alex Pedcenko

 

 

 

 

01 Connecting to HPC

OK, you’ve got an HPC account, what next?

If you have got the HPC account (on either zeus or pluto HPC), the first thing you need to do is to change your temporary password. For this, you need to log in to HPC terminal/console with your temporary password and issue the command passwd.

This can be done by establishing SSH connection with one of the HPC headnodes (either zeus.coventry.ac.uk or pluto.coventry.ac.uk depending on which HPC you have the account on). SSH is a secure shell protocol used to login to remote computers via command line. There are few ssh-client apps allowing you to do this.

Connecting in console with secure shell (ssh)

To do that you will need ssh-client software. The possibilities are:

  • Windows “native” ssh client (command line). If you have relatively recent copy of Windows 10, just open command window (Winkey+R: cmd.exe, Enter)
  • If you are using Linux or Mac computer you already have native ssh client.

In either of these cases just open a terminal and type

ssh yourhpcusername@zeus.coventry.ac.uk

or if you are using pluto HPC:

ssh yourhpcusername@pluto.coventry.ac.uk

Using PuTTY as ssh-client

PuTTY free ssh client (also available from AppsAnywhere https://appsanywhere.coventry.ac.uk), or can be downloaded and installed on your desktop PC e.g. from https://www.chiark.greenend.org.uk/~sgtatham/putty/. The documentation for using PuTTY can be found at http://www.chiark.greenend.org.uk/~sgtatham/putty/docs.html

Using Google Chrome as ssh-client

You can install ssh extension for both Google Chrome or MS Edge browsers:

Secure Shell – Chrome Web Store (google.com)

Using Windows 10 Linux subsystem (WSL) apps

Since recently you can install mini Linux distribution on your Windows 10 PC. What you need is to first install optional Windows component (Control Panel -> Programs -> Turn Windows Features on/off -> Windows Subsystem for Linux (WSL)). Once that done, reboot your PC, then go to Windows App store and get some Linux distro you are familiar with: Ubuntu, Debian etc and install it. Launch it, and you will be brought to full-functional Linux terminal, where you can install standard Linux console applications (for sure you may need “openssh-client” (Ubuntu) or “ssh” (Debian), e.g. for Ubuntu do “apt update; apt install openssh-client”). It is usefull to set up your WSL username the same as on HPC (saves time entering it all the time upon each ssh connection).

Connecting to the Desktop of HPC with VNC session (Remote Desktop like)

If you want to have a “Desktop” experience on HPC, once you have changed your temporary HPC password, you can use this way for VNC jobs/connection, see here : VNC jobs GUI HPC VNC jobs for Windows Clients | EEC High Performance Computing (coventry.domains)


Alex Pedcenko

03 Transferring Files to/from HPC

To submit a job to HPC queue, you most probably will need to copy some files (e.g. executables to run or your models, codes etc) to your HPC home folder. To do so you will need to connect remotely to the HPC file system. This can be done as described below.

Copying files via sFTP/ssh

You can copy files from/to your home folder on HPC via sftp connection. If you are using Linux local machine it can be easily done with “scp” command. Windows 10 since recently also has ssh/scp commands (from windows terminal).

Windows users can also use various sftp/scp client software, e.g. WinSCP.
Once you have downloaded and installed WinSCP, start it and connect to the headnode (e.g. zeus.coventry.ac.uk) with your HPC username and password. There is a traditional two-panel interface, one panel is your local file system, the other one is the remote one (e.g. your home folder on HPC). You can use F-keys (see bottom of the WinSCP window) or drag-and-drop files for download/upload operations.

Transferring files between HPC and your PC if you are outside the University network

Further documentation on WinSCP can be found on their website

Links: WinSCP website

css.php