EEC High Performance Computing

Here is a step-by-step guide on how to install and configure your Jupyter notebook on Zeus HPC.

1. Login to HPC’s login node and set up your user environment. This is necessary if you need to have additional python packages installed. First, we load the python module, for the operating system to “know” where the python interpreter and conda distribution is located. We also check which is the python version installed:

[aa3025@zeus2 ~]$ module load python/last
[aa3025@zeus2 ~]$ which python

/share/apps/Anaconda3_2020.11/bin/python
[aa3025@zeus2 ~]$ python --version

Python 3.8.8

2.  OK, now we can set up our own user environment, let’s call it “myenv” (arbitrary name)

[aa3025@zeus2 ~]$ conda create --name myenv

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

environment location: /home/aa3025/.conda/envs/myenv


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate myenv
#
# To deactivate an active environment, use
#
# $ conda deactivate

3. Now we can activate the environment we created

[aa3025@zeus2 ~]$ conda activate myenv
(myenv) [aa3025@zeus2 ~]$

4. Let’s install some python packages, which may be absent in the base environment. E.g. “jupyter” (can take some long tome):

(myenv) [aa3025@zeus2 ~]$ conda install jupyter

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

environment location: /home/aa3025/.conda/envs/myenv

added / updated specs:
- jupyter


The following NEW packages will be INSTALLED:

_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
_openmp_mutex pkgs/main/linux-64::_openmp_mutex-4.5-1_gnu
argon2-cffi pkgs/main/linux-64::argon2-cffi-20.1.0-py39h27cfd23_1
async_generator pkgs/main/noarch::async_generator-1.10-pyhd3eb1b0_0
attrs pkgs/main/noarch::attrs-21.2.0-pyhd3eb1b0_0
backcall pkgs/main/noarch::backcall-0.2.0-pyhd3eb1b0_0
bleach pkgs/main/noarch::bleach-4.0.0-pyhd3eb1b0_0
ca-certificates pkgs/main/linux-64::ca-certificates-2021.9.30-h06a4308_1
certifi pkgs/main/linux-64::certifi-2021.5.30-py39h06a4308_0
cffi pkgs/main/linux-64::cffi-1.14.6-py39h400218f_0
dbus pkgs/main/linux-64::dbus-1.13.18-hb2f20db_0
debugpy pkgs/main/linux-64::debugpy-1.4.1-py39h295c915_0
decorator pkgs/main/noarch::decorator-5.1.0-pyhd3eb1b0_0
.....
Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

5. OK, that’s done. Now, next time when we login to HPC or any HPC compute node and want to use our “myenv” environment, we just need to say in the shell:

module load python/last
conda activate myenv

We can also notice that installing last package, did upgrade our python interpreter:

(myenv) [aa3025@zeus1 ~]$ python --version
Python 3.9.7

6. [Optional] If we wanted to use a particular version of python in our “user” environment (e.g. some packages may require a exact version of python or of some other packages etc), we can mention that at the environment creation step:

conda create --name python2 python=2.7

If we messed up something, we can just remove our environment and start a new one again:

conda env remove --name myenv

6. Now, assuming all is fine, let’s create ipython kernel for Jupyter in our environment “myenv”.

(myenv) [aa3025@zeus2 ~]$ python -m ipykernel install --user --name=myenv
Installed kernelspec myenv in /home/aa3025/.local/share/jupyter/kernels/myenv

Now we can start Jupyter job, e.g. using

• slurm sbatch submission script: http://zeus/slurm/slurm_scripts/jupyter.slurm
• or using these scripts of mine: aa3025/hpc (coventry.ac.uk)) and use this kernel “myenv” in a Jupyter notebook.

Alex