Author Archives: admin - Page 3

MATLAB 2019a

Posted by on April 1, 2019 No comments

MATLAB 2019a is now available on zeus HPC. Use module load matlab/2019a and launch it with just matlab.

Toolboxes installed:

MATLAB                                                Version 9.6         (R2019a)
Simulink                                              Version 9.3         (R2019a)
Computer Vision Toolbox                               Version 9.0         (R2019a)
Control System Toolbox                                Version 10.6        (R2019a)
Curve Fitting Toolbox                                 Version 3.5.9       (R2019a)
DSP System Toolbox                                    Version 9.8         (R2019a)
Deep Learning Toolbox                                 Version 12.1        (R2019a)
Fixed-Point Designer                                  Version 6.3         (R2019a)
GPU Coder                                             Version 1.3         (R2019a)
HDL Coder                                             Version 3.14        (R2019a)
Image Acquisition Toolbox                             Version 6.0         (R2019a)
Image Processing Toolbox                              Version 10.4        (R2019a)
Instrument Control Toolbox                            Version 4.0         (R2019a)
MATLAB Coder                                          Version 4.2         (R2019a)
MATLAB Compiler                                       Version 7.0.1       (R2019a)
MATLAB Compiler SDK                                   Version 6.6.1       (R2019a)
Model Predictive Control Toolbox                      Version 6.3         (R2019a)
Optimization Toolbox                                  Version 8.3         (R2019a)
Parallel Computing Toolbox                            Version 7.0         (R2019a)
Partial Differential Equation Toolbox                 Version 3.2         (R2019a)
Sensor Fusion and Tracking Toolbox                    Version 1.1         (R2019a)
Signal Processing Toolbox                             Version 8.2         (R2019a)
Simscape                                              Version 4.6         (R2019a)
Simscape Driveline                                    Version 2.16        (R2019a)
Simscape Electrical                                   Version 7.1         (R2019a)
Simscape Fluids                                       Version 2.6         (R2019a)
Simscape Multibody                                    Version 6.1         (R2019a)
Simulink 3D Animation                                 Version 8.2         (R2019a)
Simulink Coder                                        Version 9.1         (R2019a)
Simulink Control Design                               Version 5.3         (R2019a)
Statistics and Machine Learning Toolbox               Version 11.5        (R2019a)
Symbolic Math Toolbox                                 Version 8.3         (R2019a)
Text Analytics Toolbox                                Version 1.3         (R2019a)
Vehicle Dynamics Blockset                             Version 1.2         (R2019a)
Vehicle Network Toolbox                               Version 4.2         (R2019a)

Let me know if any extra toolboxes are necessary.

Regards,

Alex

Zeus HPC Video channel

Posted by on January 9, 2019 No comments

Microsoft Stream Zeus HPC channel

You are welcome t to contribute if you want.

Alex Pedcenko

OpenMPI 4.0.0

Posted by on December 11, 2018 No comments

OpenMPI v.4.0.0 is available on zeus HPC. Two versions, compiled with intel compiler 2019:

module load openmpi/4.0.0/intel2019

and gcc-7.1.0

module load /openmpi/4.0.0/gcc7.1.0

There is also openmpi-2.1.5 compiled with gcc-7.1.0

module load openmpi/2.1.5/gcc7.1.0

Alex

04 Partitions (queues) of zeus HPC

Posted by on November 30, 2018 No comments

There is only one queue/partition named “all” (default) on zeus HPC. For more details see this post:

Queues of zeus [updated June 2018] | EEC High Performance Computing (coventry.domains)

 

Alex Pedcenko

Secure Shell extension for Google Chrome!

Posted by on November 28, 2018 No comments

Terminal emulator and SSH client is now available as a Google Chrome browser extension: run your ssh session to HPC from Chrome!

Secure Shell is an xterm-compatible terminal emulator and stand-alone ssh client for Chrome. It uses Native-Client to connect directly to ssh servers without the need for external proxies.

https://chrome.google.com/webstore/detail/secure-shell-extension/iodihamcpbpeioajjeobimgagajmlibd

Alex

GUI HPC VNC jobs for Mac/Linux Clients

Posted by on July 6, 2018 No comments
  1. not essential, but useful: create password-less access to zeus, in the terminal of your OSX/Linux do:
ssh-keygen

(just press Enter few times, do not enter any passwords or “secrets”)

Changing username to your HPC login name, do

ssh-copy-id username@zeus1.coventry.ac.uk

(enter HPC password as prompted, this will copy public ssh key created above to your zeus profile, now you can connect to zeus without entering password -> very useful for following steps). Only need to be done ONCE (per client computer)!

——————————————————————————————————————-

Next steps can be scripted, i.e. put code lines into shell script and call it later with different node name or number (node name can be passed as a parameter to the script)

2. Launch VNC server job in the zeus queue, say, on 1 GPU node asking for 1 K80 GPU and exclusive access to the node (remove –exclusive if the whole node is not needed)

 

ssh yourHPClogin@zeus1.coventry.ac.uk sbatch --exclusive -N1 --gres=gpu:K80:1 -t 24:00:00 /share/apps/slurm/vnc.slurm

(omit –gres=gpu:K80:1 if no GPUs needed)

(generally -N X in the above code line means ask for X compute nodes)

Check which node was assigned to your VNC job in the queue:

ssh username@zeus1.coventry.ac.uk squeue -u $USERNAME

Say the result of that is compute node zeus400 assigned to your VNC job (you can always check with “qstat” or “squeue -u $USER” on zeus or on zeus website (show queue)

 

3. Establish tunnel from your Mac to target node (e.g. zeus400) via zeus2.coventry.ac.uk (or zeus1):

ssh -L localhost:5901:zeus400:5901 zeus2.coventry.ac.uk

 

4. Now launch vncviewer on your Mac and connect to localhost:1 (that must bring you to the VNC session at zeus400:1

vncviewer localhost:1

(or any other way you launch it on Macs…)
Alex Pedcenko

02 GUI HPC VNC jobs for Windows Clients

Posted by on July 5, 2018 No comments

For running GUI jobs on HPC compute node (rather than on login node), you can use my new Windows HTA-scripts from here: aa3025/hpc (coventry.ac.uk). They will also work via University VPN (Anyconnect)

##### optional step – usually already done when your HPC account is created ########

Prior to all the following you need to set your VNC password on the HPC (you can also change your VNC password this way):

  • login via ssh to HPC,
  • issue the command “vncpasswd” to change your VNC password (VNC password can be different to your HPC password, but it makes life easier if it is the same as your HPC password). Use this password for connecting to your VNC sessions in future.
####################################################

You can use these “scripts” to launch VNC (Remote Desktop) job on one of the HPC compute nodes if you need access to GUI (Graphical User Interface)

These scripts allow establishing VNC Desktop session on a compute node of HPC : (see also VIDEO screen cast here)

or here:

0) Download the whole distribution from aa3025/hpc (coventry.ac.uk) You have at least to download 2 folders: “tools” and either “zeus” or “pluto” depending on which HPC you use. You can use git for synchronising the whole distribution:

git clone https://github.coventry.ac.uk/aa3025/hpc.git
Alternatively you can download the zip archive from : HERE

1) Depending which HPC you are about to use, choose a folder zeus or pluto to launch VNC session on required HPC. Before proceeding further  check that HPC has free (idle) compute nodes available (“sinfo” in HPC console or check Zeus website or Pluto website)

2) Launch only one script: 01_START_VNC_Session_….._1_node.hta (by double-clicking on it), a.k.a. “submission app”

3) Fill in your HPC username, password and the rest of info. Hit “Submit” button, the script will then submit the “VNC” job to the HPC queue in order to start “vncserver” process [Remote Desktop session] on the assigned compute node.

4) Wait for your VNC job to start in the queue of HPC (you will see job listing in the newly opened black “plink” console window), note the target node name (it is also presented in the last column of the small table at the bottom of the submission app).

5) Once VNC job has started (“Running” state, “R”) in the black console,  you can close it. Make sure your VNC job is in Running state (also specified by “R” against your job ID in the table at the bottom of the submission app, if it is still in “PD” state (means still “pending”), waiting for resources to be allocated).

6) Once you closed the first (job submission) app and black console window, the second script will start automatically to get the ssh-tunnel running between your PC and the allocated compute node and will connect you to the node’s VNC session — fill in the target node name from the previous step and your HPC password. The VNC screen number is always :1 (you can’t change it).

7) Now, the VNC password window will pop up — enter your VNC password there (theoretically it can be different from your HPC password). If you do not have VNC password, return to the top of this page (“Prior to all of the following…”) and set it up.

8) If all went well, the VNC Remote Desktop windows will be presented. You can toggle it to full screen by pressing F8 on your keyboard (amongst some other options in F8 as well).

By default the VNC job will last for 24 hours and will request 1 whole node, you can select different time and less CPUs than on the whole node or more than one compute node to be allocated.

More than one compute node. If you specify more than one node in VNC job submission app, the VNC Desktop session will start on the first allocated node, but the resources of all other allocated nodes will still be available from your VNC Desktop session.

VNC job termination

When the slurm VNC job expires (the time limit of your job [normally 24 hours if you did not change it] was reached), your VNC Desktop session will be terminated.

However, if you wish to terminate the VNC job before it expires (you do not use it any more and need to free the HPC resources), you can either just kill it from ssh console with “scancel JOBID” command, or by selecting “Logout” in your VNC Desktop session and your slurm job with VNC sessions will terminate.

If you just close the VNC viewer window on your PC –> the VNC Desktop session will remain running until termination time is reached or you cancel your VNC job. Make sure you do not waste HPC resources by having idle VNC sessions’ jobs hanging in the queue!

Alex Pedcenko

Update on queues of Zeus HPC

Posted by on June 5, 2018 No comments

Queuing model UPDATE June 2018

To simplify usage of different queues, we combined all nodes into a single default queue (slurm partition) “all”. The usage limits are now solely user-based, each user has default (for now) number of CPU*minutes that they can use at any time moment (subject to available resources). If this number of CPU*min is reached, the new jobs from this user will be put on queue until their running jobs will free resources. This is independent of type of compute nodes. During this initial stage we will try to adapt the default CPU*min allowance to suite better and more effective HPC usage. The simple principle behind this is that user can use more CPU cores but for less time, or, less CPU cores, but for longer time. The run time of the job to be submitted is determined by the value you set in –time or -t parameter during the submission of a job (e.g. -t 24:00:00).

If you require a particular type of compute nodes (CPU/GPU/Phi etc), this can be done in submission script or during the submission with sbatch command: by specifying an additional parameter “constraint”:

  • for 56 Intel Broadwell CPU based nodes (128GB RAM each) with 32xCPU-cores , specify --constraint=broadwell
  • for 144 Intel Nehalem CPU based nodes (48 GB RAM each) with 8xCPU-cores, specify --constraint=nehalem
  • for 18 Intel SandyBridge CPU based nodes (48 GB RAM) with 12xCPU-cores, do --constraint=sandy
  • for 1 x 32 CPU, 512GB RAM SMP node ask for --constraint=smp
  • for 10 nodes x 2 NVidia Kepler K20 GPUs, ask for --gres=gpu:K20:N (where N is the Nr of GPUs needed, max is 2 GPUs/node)
  • for 18 nodes x 2 NVidia Kepler K80 GPUs, ask for --gres=gpu:K80:N  (where N is the Nr of GPUs needed, max is 2 GPUs/node)
  • for N Intel Phi, ask for --gres=mic:N or --constraint=phi.

For more details on Zeus’s CPUs and nodes see this post: http://zeus.coventry.ac.uk/wordpress/?p=336

If you have no particular preference on the type of CPU or compute node and are running parallel job, please specify ONLY TOTAL Nr of CPUs required, NOT Nr of nodes!: SLURM will assign the nodes automatically.

e.g. if I need 64 CPUs in total for 24 hours on whatever available nodes I submit my slurm script with:

sbatch -n 64 -t 24:00:00 myslurmscriptname.slurm

if I need 64 CPUs in total for 48 hours on Broadwell-based nodes (32 CPUs/node) I submit my slurm script with:

sbatch -n 64 -t 48:00:00 --constraint=broadwell myslurmscriptname.slurm

Finally if I want 2 GPU nodes with 2 Nvidia Kepler K80 GPUs (1 GPUs/node) and 2 CPUs on each node (36 hours), I do something like:

sbatch -N2 --ntasks-per-node=1 --cpus-per-task=2 --gres=gpu:K80:1 -t 36:00:00 mygpuscript.slurm

Certainly some variations of these sbatch commands are possible, also these flags can be specified inside the slurm submission script itself. For full list of possible sbatch options see slurm docs: https://slurm.schedmd.com/sbatch.html

 

Alex Pedcenko

ONLINE SELF-PACED TRAINING with NVIDIA Deep Learning Institute Online Labs

Posted by on May 2, 2018 No comments

The NVIDIA Deep Learning Institute (DLI) offers hands-on training for developers, data scientists, and researchers looking to solve the world’s most challenging problems with deep learning and accelerated computing.

Choose from full-day courses to deploy an end-to-end project, or two-hour mini courses to learn a specific technology or technique.

 

Online Courses available:

  • Fundamentals of Deep Learning for Computer Vision
  • Fundamentals of Accelerated Computing with CUDA C/C++
  • Fundamentals of Accelerated Computing with CUDA Python
  • Fundamentals of Accelerated Computing with OpenACC
  • Deep Learning for Healthcare Image Analysis
  • Deep Learning for Healthcare Genomics

The link: https://developer.nvidia.com/dli/onlinelabs 

PLUTO HPC switched off

Posted by on March 23, 2018 No comments

for the weekend 23-25 March, Pluto HPC is switching off, due to the cooling failure in the HPC comms  room to prevent hardware damage.

Alex Pedcenko

css.php