Category Archives: Slurm Business

05 Submitting jobs to different nodes

Posted by on February 13, 2020 No comments

Since 2018 we have only one default slurm partition (queue) on zeus called “all”, see here

See also: examples of SLURM submission scripts

Submit to ‘Broadwell’ nodes

The hostnames of Broadwell nodes are zeus[300-343] & zeus[400-409] & zeus[500-501] (56 in total). They have 32 CPU-cores (Broadwell Xeon) and 128GB of RAM. There are several ways how you can use these nodes.

  • submit directly to compute node by specifying its hostname (not recommended, only if you need to use that exact node for some reason, e.g. having a reservation there): e.g. to request zeus300, in your slurm script use 
    #SBATCH -w zeus300
  • or request that particular node during submission 
    sbatch -w zeus300 slurmscriptname.slurm
  • or in your slurm submission script request constraint “broadwell” with whatever number of tasks you require, for example, we can request one task that can have access to all 32 CPUs of one broadwell node (to run SMP code for example) 
    #SBATCH --constraint=broadwell
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32

    or request 32 individual tasks (will allocate one task per CPU)

    #SBATCH --constraint=broadwell
#SBATCH -n32 -N1

    where -n32 is the total number of CPU’s requested over one node (-N1) (not Nr of CPUs per node)

Submit to ‘Sandybridge’ 12-CPU nodes nodes

GPU nodes of zeus (zeus[200-217]) have 12-core Sandybridge family CPU’s and NVIDIA K20 GPUs (2 per each node). You can use those nodes without GPU’s as well. To use these nodes do either of:

  • specify 
    --constraint=sandy

    or request k20 GPUs:

    --gres=gpu:K20:N

    (where N is number of GPUs you need)

    sbatch --constraint=sandy slurmscriptname.slurm
    sbatch -N1 --gres=gpu:K20:2 slurmscriptname.slurm
  • you can also request these particular nodes (zeus200…zeus217) to be allocated to your job, e.g. we can ask for 2 of these nodes:
  • sbatch -w zeus[200-201] slurmscriptname.slurm

     

Submit to Nehalem (8-CPU) nodes

These are old Nehalem CPU based nodes of Zeus HPC each having 8 CPU-cores and 48Gb of RAM: zeus[20-91] (floor 3 of ECB) and zeus[100-171] (Mezz floor of ECB)
If you want to use specifically these nodes you can use

sbatch --constraint=nehalem slurmscriptname.slurm

when submitting your job with sbatch or srun or specify it in your slurm sbatch script

#SBATCH --constraint=nehalem

You can specifically request a particular node (e.g. zeus34 and zeus56…72 in the example below.

sbatch -w zeus[34,56-72] slurmscriptname.slurm

Important thing to remember is that these nodes cannot allocate more trhan 8 tasks per node.

Request nodes with GPU processors

  • In your slurm submission script specify (e.g. to ask for 2 GPUs of any kind) 
    #SBATCH --gres=gpu:2
  • OR request particular kind of GPUs (K20 or K80) in your slurm submission script (below example asks for 1 K80 GPU): 
    #SBATCH --gres=gpu:k80:1
  • or e.g. 2 K20 GPUs on one node in exclusive mode (node will not be shared with other jobs if it has free resources (CPUs or GPUs that are not used by your job): 
    sbatch --gres=gpu:k20:2 -N1 --exclusive slurmscriptname.sh

Launching job without slurm submission script

  • You can run any executable, e.g. “myexecutable”, on the zeuse’s nodes, without any slurm submission scripts (job will be placed in queue while the job is running anyway). Here we request 2 nodes and the total of 12 CPUs for 10 minutes:
  • srun -t 00:10:00 -N2 -n12 ./myexecutable

    Note that when submitting job with “srun”, the command line will not be “released” until the job completes and the output is produced in STDOUT (unless specified otherwise). This approach is suitable if you expect your “job” to complete very quickly. this way you can also see the direct output of your executable in the terminal.

  • Yo can also reserve nodes for a specified time without launching any task on them (i.e. it will launch idle interactive bash session), the example below is asking for 2 Broadwell nodes (32 CPU/node, in exclusive mode (no resource sharing with other jobs)) for 12 hours: 
    salloc -N2 --constraint=broadwell --exclusive -t 12:00:00

    Similarly for GPUs, e.g. one Sandybridge node with 2 K20 GPUs for 30 minutes:

    salloc -N1 --gres=gpu:K20:2 --exclusive -t 30

    After submission of “salloc” job your console will start dedicated bash session in interactive mode, which will have all allocated resources available. When you leave that bash session (Ctrl-D or “exit”) your “salloc” job/allocation will be terminated in the queue. To avoid this (job termination on exiting bash session) you can use “screen” utility.

    For all possible “constraints”, see here: http://zeus.coventry.ac.uk/wordpress/?p=1094

  • Note that in any of the cases described above, if you do not specify the time required for your job, it will be assigned default 4 hours, after which the job will be terminated, you can do it by using “-t” flag (below we ask for 8 random CPUs for 8 hours
sbatch -n8 -t 8:00:00 slurmscriptname.slurm

Memory requirements when submitting a job

Posted by on January 15, 2020 No comments

Dear All

To avoid HPC users downing compute nodes, memory limits are introduced on zeus HPC.

If your jobs are not that memory-hungry, you probably will not notice this at all. By “memory-hungry” it is meant exceeding 4GB per CPU-core (default value).

If your job require more than that, you can request more memory using --mem=... or --mem-per-cpu=... [MB] parameter with sbatch.

E.g.

1) ask for “full” 48 GB of memory to be available to your job e.g. on Nehalem (8-CPU) nodes:

sbatch -n8 -N1 --mem=48000 -t=8:00:00 myslurmscript.slurm

In this case one node is requested, 8 tasks (CPUs) and total job memory is 48GB (48000 MB)

If this is not specified max memory would be 4GB x 8 CPUs = 32 GB

2) If you using 32-CPU Broadwell nodes, which have 128GB of RAM, the default value of 4GB/CPU is max possible. If you want more RAM per CPU, e.g. if you use only 2 CPU-tasks, but need all node memory allocated to them, you can do:

sbatch -n2 -N1 --mem-per-cpu=64000 ...

3) if you need more than 128GB per node, you can use SMP node (zeus15, max 512GB/node) by requesting “–constraint=smp” when submitting the job.

If you request amount of memory, exceeding physically available, job will fail to submit with error message “error: Memory specification can not be satisfied”, “error: Unable to allocate resources: Requested node configuration is not available”.

If you requested certain memory for your job (or left it with default 4GB/CPU), which is then exceeded during the run, slurm will terminate the job.

Before this measure, it was possible to “oversubscribe” (consume more memory than available RAM by using disk swap space) and make a node unresponsive/slow, which result in job termination anyway, but in certain cases led to node failure.

Regards

Alex

SLURM Cheat Sheet

Posted by on November 18, 2019 No comments

SLURM Cheat Sheet

02 GUI HPC VNC jobs for Windows Clients

Posted by on July 5, 2018 No comments

For running GUI jobs on HPC compute node (rather than on login node), you can use my new Windows HTA-scripts from here: aa3025/hpc (coventry.ac.uk). They will also work via University VPN (Anyconnect)

##### optional step – usually already done when your HPC account is created ########

Prior to all the following you need to set your VNC password on the HPC (you can also change your VNC password this way):

  • login via ssh to HPC,
  • issue the command “vncpasswd” to change your VNC password (VNC password can be different to your HPC password, but it makes life easier if it is the same as your HPC password). Use this password for connecting to your VNC sessions in future.
####################################################

You can use these “scripts” to launch VNC (Remote Desktop) job on one of the HPC compute nodes if you need access to GUI (Graphical User Interface)

These scripts allow establishing VNC Desktop session on a compute node of HPC : (see also VIDEO screen cast here)

or here:

0) Download the whole distribution from aa3025/hpc (coventry.ac.uk) You have at least to download 2 folders: “tools” and either “zeus” or “pluto” depending on which HPC you use. You can use git for synchronising the whole distribution:

git clone https://github.coventry.ac.uk/aa3025/hpc.git
Alternatively you can download the zip archive from : HERE

1) Depending which HPC you are about to use, choose a folder zeus or pluto to launch VNC session on required HPC. Before proceeding further  check that HPC has free (idle) compute nodes available (“sinfo” in HPC console or check Zeus website or Pluto website)

2) Launch only one script: 01_START_VNC_Session_….._1_node.hta (by double-clicking on it), a.k.a. “submission app”

3) Fill in your HPC username, password and the rest of info. Hit “Submit” button, the script will then submit the “VNC” job to the HPC queue in order to start “vncserver” process [Remote Desktop session] on the assigned compute node.

4) Wait for your VNC job to start in the queue of HPC (you will see job listing in the newly opened black “plink” console window), note the target node name (it is also presented in the last column of the small table at the bottom of the submission app).

5) Once VNC job has started (“Running” state, “R”) in the black console,  you can close it. Make sure your VNC job is in Running state (also specified by “R” against your job ID in the table at the bottom of the submission app, if it is still in “PD” state (means still “pending”), waiting for resources to be allocated).

6) Once you closed the first (job submission) app and black console window, the second script will start automatically to get the ssh-tunnel running between your PC and the allocated compute node and will connect you to the node’s VNC session — fill in the target node name from the previous step and your HPC password. The VNC screen number is always :1 (you can’t change it).

7) Now, the VNC password window will pop up — enter your VNC password there (theoretically it can be different from your HPC password). If you do not have VNC password, return to the top of this page (“Prior to all of the following…”) and set it up.

8) If all went well, the VNC Remote Desktop windows will be presented. You can toggle it to full screen by pressing F8 on your keyboard (amongst some other options in F8 as well).

By default the VNC job will last for 24 hours and will request 1 whole node, you can select different time and less CPUs than on the whole node or more than one compute node to be allocated.

More than one compute node. If you specify more than one node in VNC job submission app, the VNC Desktop session will start on the first allocated node, but the resources of all other allocated nodes will still be available from your VNC Desktop session.

VNC job termination

When the slurm VNC job expires (the time limit of your job [normally 24 hours if you did not change it] was reached), your VNC Desktop session will be terminated.

However, if you wish to terminate the VNC job before it expires (you do not use it any more and need to free the HPC resources), you can either just kill it from ssh console with “scancel JOBID” command, or by selecting “Logout” in your VNC Desktop session and your slurm job with VNC sessions will terminate.

If you just close the VNC viewer window on your PC –> the VNC Desktop session will remain running until termination time is reached or you cancel your VNC job. Make sure you do not waste HPC resources by having idle VNC sessions’ jobs hanging in the queue!

Alex Pedcenko

Update on queues of Zeus HPC

Posted by on June 5, 2018 No comments

Queuing model UPDATE June 2018

To simplify usage of different queues, we combined all nodes into a single default queue (slurm partition) “all”. The usage limits are now solely user-based, each user has default (for now) number of CPU*minutes that they can use at any time moment (subject to available resources). If this number of CPU*min is reached, the new jobs from this user will be put on queue until their running jobs will free resources. This is independent of type of compute nodes. During this initial stage we will try to adapt the default CPU*min allowance to suite better and more effective HPC usage. The simple principle behind this is that user can use more CPU cores but for less time, or, less CPU cores, but for longer time. The run time of the job to be submitted is determined by the value you set in –time or -t parameter during the submission of a job (e.g. -t 24:00:00).

If you require a particular type of compute nodes (CPU/GPU/Phi etc), this can be done in submission script or during the submission with sbatch command: by specifying an additional parameter “constraint”:

  • for 56 Intel Broadwell CPU based nodes (128GB RAM each) with 32xCPU-cores , specify --constraint=broadwell
  • for 144 Intel Nehalem CPU based nodes (48 GB RAM each) with 8xCPU-cores, specify --constraint=nehalem
  • for 18 Intel SandyBridge CPU based nodes (48 GB RAM) with 12xCPU-cores, do --constraint=sandy
  • for 1 x 32 CPU, 512GB RAM SMP node ask for --constraint=smp
  • for 10 nodes x 2 NVidia Kepler K20 GPUs, ask for --gres=gpu:K20:N (where N is the Nr of GPUs needed, max is 2 GPUs/node)
  • for 18 nodes x 2 NVidia Kepler K80 GPUs, ask for --gres=gpu:K80:N  (where N is the Nr of GPUs needed, max is 2 GPUs/node)
  • for N Intel Phi, ask for --gres=mic:N or --constraint=phi.

For more details on Zeus’s CPUs and nodes see this post: http://zeus.coventry.ac.uk/wordpress/?p=336

If you have no particular preference on the type of CPU or compute node and are running parallel job, please specify ONLY TOTAL Nr of CPUs required, NOT Nr of nodes!: SLURM will assign the nodes automatically.

e.g. if I need 64 CPUs in total for 24 hours on whatever available nodes I submit my slurm script with:

sbatch -n 64 -t 24:00:00 myslurmscriptname.slurm

if I need 64 CPUs in total for 48 hours on Broadwell-based nodes (32 CPUs/node) I submit my slurm script with:

sbatch -n 64 -t 48:00:00 --constraint=broadwell myslurmscriptname.slurm

Finally if I want 2 GPU nodes with 2 Nvidia Kepler K80 GPUs (1 GPUs/node) and 2 CPUs on each node (36 hours), I do something like:

sbatch -N2 --ntasks-per-node=1 --cpus-per-task=2 --gres=gpu:K80:1 -t 36:00:00 mygpuscript.slurm

Certainly some variations of these sbatch commands are possible, also these flags can be specified inside the slurm submission script itself. For full list of possible sbatch options see slurm docs: https://slurm.schedmd.com/sbatch.html

 

Alex Pedcenko

Process and Thread Affinity

Posted by on March 6, 2017 No comments

https://computing.llnl.gov/tutorials/openMP/ProcessThreadAffinity.pdf

Resource Management for Multi-Core/Multi-Threaded Usage

Posted by on March 6, 2017 No comments

https://slurm.schedmd.com/slurm_ug_2011/cons_res.pdf

EC STRUCTURAL SIMULATION HPC SUITE

Posted by on April 16, 2014 No comments

by Oliver Grimes
——————

MS-WORD *.doc version of this document.

EC_FEA_Run_Script

TAKE CARE AS MOST PARAMETERS ARE CASE SENSITIVE

The purpose of this script is to enable quick solver choice and settings for FEA runs on Coventry University HPC “ZEUS”. The following document details how to use and submit the script.

Currently Registered Solvers are:

LS-DYNA, OPTISTRUCT(_NL), RADIOSS, LS-OPT, LS-TASC, DIGIMAT_DYNA or MADYMO

You can find the most up to date EC_FEA_Run_Script.sh in your $HOME/RUN_SCRIPTS

HOW TO SUMBIT THE SCRIPT

  1. Ensure the script is in your run directory (ideally a clean directory)
  2. Places you input file (solver input deck) and any include folders in this folder.
  3. Set your preferred solver and settings as detailed in the following pages.
  4. The job is then ready to submit in either of the following ways:
  • In a terminal window enter 
    ./EC_FEA_Run_Script.sh
  • In a terminal window enter 
    sh EC_FEA_Run_Script.sh

Note: Wildcards can be used to save time ie. ./EC_FE* or sh EC* etc.

  1. Any errors on warnings on submitting the job will appear in the standard output of the terminal window. It is import to read these warnings as error as the job may make adjustments to the solver settings if incorrect.
  2. On submission an incrementally numbered SOLUTION_#### folder and SOLUTION_####.out file will be produced. This contains your job solution and output log summary (which is useful to monitor whilst the job is initialising to check it doesn’t error terminate). The SOLUTION_#### names (folder and out log file) which automatically increment by number if the job is resubmitted in the same directory. This can be useful when debugging job and keeping a record of changes.
  3. Check the job is running using the “squeue –u username” command in a terminal.

IMPORTANT: Do not change the SOLUTION_#### names until the job has finished!!

USEFUL COMMANDS

squeue

– View the job queue

squeue -u username

– View just your user jobs (replace username with your’s)

scancel JOB_ID

– Cancel a job by JOB_ID (found in the squeue)

scancel -u username

– Cancel all your user jobs (replace username with your’s)

sinfo

– View queue partitions status

sbatch

– Manually submit a SUBMISSION_SCRIPT

RUN SCRIPT Parameters and Settings

The following section details the run script settings.

Input Filename:

Enter your filename (include the file extension) in the input parameter. This must not contain spaces, neither must the run directory.

Eg. 

input="filename.ext"

Jobname:

Entry of the jobname is not essential but is very useful. The name entered under this parameter will appear as the name of the job in the job queue.

Eg. 

jobname="JOB_123"

Solver Choice:

Enter the FEA solver you require from the list of registered solvers. This parameter is case sensitive so be careful to copy it directly as in the list.

Currently Registered Solvers are:

LS-DYNA, OPTISTRUCT(_NL), RADIOSS, LS-OPT, LS-TASC, DIGIMAT_DYNA or MADYMO

Eg. 

solver=OPTISTRUCT

Number of Nodes for the job:

Enter the number of computer nodes required for the job. Most nodes on ZEUS contain 2 x Quad core processors (8 cores nodes) so two nodes provides 16 processors. Remember, more processors does not necessarily provide improvements in the runtime, some solvers don’t scale well. DO NOT WAIST HPC RESOURCE!! When running multiple jobs, running more jobs simultaneously on less processors will overall be quicker than with job stacked sequentially on more nodes.

Eg. 

NUM_NODES=3

(must be “default” or an integer)

Note: Unless the more_nodes_password is supplied the max number of nodes maybe capped. If the specified nodes is over the capped limit NUM_NODES will be automatically set to the max allowed. A warning will be issued in such a case.

Walltime:

The walltime is the maximum time you think the job will take to run. If unsure, be overly cautious as the job will be terminated if the walltime is reached, even if the job hasn’t finished. However when the HPC queue is busy specifying lengthy walltime’s will deprioritise the job and take longer to start up. Try to keep this realistic.

Eg. 

walltime=24:00:00

(for 24 hours)

ADVANCED Options

The second half of the EC_FEA_Run_Script.sh contain more settings for advanced users. Left these parameters as default using you know what they do.

For specific information on how these parameters relate to each solver see the APPENDIX section of these instructions.

#####################################################################
#####                                                           #####
#####                   ADVANCED OPTIONS                        #####
#####                                                           #####
#####################################################################

More Nodes Password:

Specify the more nodes password here if you have been given it by your supervisor. This will remove the student cap of the maximum number of nodes you can launch the job on.

Eg. 

more_nodes_password="password"

 

Number of CPU’s:

This option is to specify the number of processes (domains/MPI processes etc.) the entire job requires. If set as default the most efficient configuration for specified solver with be chosen.

Eg. 

NUM_NCPU=default

(must be “default” or an integer)

 

Number of Threads per process:

This option specifies the number of thread per process (specified above). For example a Hybrid job run multiple SMP processes using MPI, each SMP process will likely have multiple threads running using several cores. This parameter specifies in this example the number of threads/sub processes for the SMP process. Only set if you know what you are doing as job will run extremely slowly/inefficiently if set incorrectly. If set as default the most efficient configuration for specified solver with be chosen.

Eg. 

NUM_THREADS=default

(must be “default” or an integer)

 

Memory for the job:

Specifies the required memory for the job. This units of this input are solver dependant (could be words, megawords, megabytes). Ideally leave as default unless needed.

Eg. 

memory=default

(must be “default” or an integer)

Queue Partition:

To set a specific SLURM queue partition enter the required partition name using this parameter. Usually set as default which choses the most relevant queue for the solver choice (usually the “all” queue).

Eg. 

queue=default

(must be “default” or an existing queue partition name)

Parallel Processing Type:

This option let you choose your parallel processing job (assuming the corresponding solver in compatible and installed). The options are default, SMP, MPP, Hybrid (Using multiple SMP). Leave as default unless you know the job with be faster/more efficient using a different processing type.

Eg. 

parallel=default

Precision Type:

Sets the solver to process in either single or double precision (floating point length). Only require for some simulation type. Normally costs approximately 20-30% extra runtime. The default option is usually single precision.

Eg. 

precision=default

(must be “default”, single or double)

 

Solver Version:

A different version of the solver can be called be setting this parameter. The specific option relates to the version folder name in the /share/apps/EC_STRUCT_SIM/%solver%/%version folder%. Leave as default unless required.

Eg. 

version=default

(must be “default”, previous, latest, specific (ie. 971_R5.1.1))

Other Solver Command Line Options/Arguments:

If the user wants to provide any additional solver options/arguments then this parameter will add them to the end of the default command line.

Eg. 

OTHER_ARGS="-option argument=1"

DO NOT EDIT THE SCRIPT BEYOND THE ABOVE PARAMETER

 

APPENDIX – SOLVER SPECIFIC SETTINGS

The follow appendix details how the above settings relate to the specific solver settings.

APPENDIX A – LS-DYNA Run Script Settings.

Location: 

/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/LS-DYNA_Script

Required Modules:

######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/LS-DYNA/v971
#######################################################################

Default Options:

###################### SET LS-DYNA RUN PARAMETERS #####################
DEFAULT_NUM_NODES=2

DEFAULT_QUEUE=all

DEFAULT_NUM_NCPU_SMP=1

DEFAULT_NUM_NCPU_MPP=8

DEFAULT_NUM_NCPU_Hybrid=2

DEFAULT_NUM_THREADS_SMP=8

DEFAULT_NUM_THREADS_MPP=1

DEFAULT_NUM_THREADS_Hybrid=4

DEFAULT_MEMORY=777777777

DEFAULT_PARALLEL="MPP"

DEFAULT_PRECISION="single"

SOLVER_VERSIONS="971_5.1.1 971_6.1.1 971_6.1.2"

DEFAULT_VERSION=971_6.1.2

LATEST_VERSION=971_6.1.2

PREVIOS_VERSION=971_5.1.1

EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/LS-DYNA"

Solver Notes:

SMP – Runs one process (Shared memory parallel) using multiple threads.

  • Must be run on 1 x node only (will auto default to 1 otherwise)

MPP– Runs one serial process per core (Massively Parallel Processing) using HPMPI.

  • Total cores = NUM_NODES * NUM_NCPU

Hybrid – Runs parallel SMP processes (which can each run parallel threads) using HPMPI.

  • No. of SMP processes = NUM_NODES * NUM_NCPU / NUM_THREADS
  • No of threads per SMP process = NUM_THREADS (best to use a whole physical processor=4)
  • Total cores = No. of SMP processes * No. of thread per SMP process

APPENDIX B – LS-OPT Run Script Settings.

Location: 

/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/LS-OPT_Script

Required Modules:

######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/LS-DYNA/v971
#######################################################################

Default Options:

####################### SET LS-TASC RUN PARAMETERS #######################
LS_OPT_PATH=/share/apps/EC_STRUCT_SIM/LS-OPT/v5.0_84950
####################### SET LS-DYNA RUN PARAMETERS ####################### 

input=DynaOpt.inp

jobname=$(basename `pwd`)

DEFAULT_NUM_NODES=1

DEFAULT_QUEUE=all

DEFAULT_NUM_NCPU_SMP=1

DEFAULT_NUM_NCPU_MPP=8

DEFAULT_NUM_NCPU_Hybrid=2

DEFAULT_NUM_THREADS_SMP=8

DEFAULT_NUM_THREADS_MPP=1

DEFAULT_NUM_THREADS_Hybrid=4

DEFAULT_MEMORY=777777777

DEFAULT_PARALLEL="MPP"

DEFAULT_PRECISION="single"

SOLVER_VERSIONS="971_5.1.1 971_6.1.1 971_6.1.2"

DEFAULT_VERSION=971_6.1.2

LATEST_VERSION=971_6.1.2

PREVIOS_VERSION=971_5.1.1

EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/LS-DYNA"

APPENDIX C – LS-TASC Run Script Settings.

Location: 

/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/ LS-TASC_Script

Required Modules:

######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/LS-DYNA/v971
#######################################################################

Default Options:

####################### SET LS-TASC RUN PARAMETERS #######################
LS_OPT_PATH=/share/apps/EC_STRUCT_SIM/LS-TASC/lstasc_21_86715_x64_rhel6
####################### SET LS-DYNA RUN PARAMETERS #######################

input=DynaOpt.inp

jobname=$(basename `pwd`)

DEFAULT_NUM_NODES=1

DEFAULT_QUEUE=all

DEFAULT_NUM_NCPU_SMP=1

DEFAULT_NUM_NCPU_MPP=8

DEFAULT_NUM_NCPU_Hybrid=2

DEFAULT_NUM_THREADS_SMP=8

DEFAULT_NUM_THREADS_MPP=1

DEFAULT_NUM_THREADS_Hybrid=4

DEFAULT_MEMORY=777777777

DEFAULT_PARALLEL="MPP"

DEFAULT_PRECISION="single"

SOLVER_VERSIONS="971_5.1.1 971_6.1.1 971_6.1.2"

DEFAULT_VERSION=971_6.1.2

LATEST_VERSION=971_6.1.2

PREVIOS_VERSION=971_5.1.1

EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/LS-DYNA"

APPENDIX D – DIGIMAT_DYNA Run Script Settings.

Location: 

/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/DIGIMAT_DYNA_Script

Required Modules:

######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/LS-DYNA/v971
module load /share/apps/EC_STRUCT_SIM/modulefiles/digimat/v5.0.1
#######################################################################

Default Options:

 

####################### SET DIGIMAT RUN PARAMETERS #######################

DEFAULT_DIGI_DYNA_VERSION=V611

LATEST_DIGI_DYNA_VERSION=V611

PREVIOS_DIGI_DYNA_VERSION=V511

DIGIMAT2CAE="/share/apps/EC_STRUCT_SIM/DIGIMAT/v5.0.1/Digimat/Digimat2CAE/5.0.1"

DIGIMAT_BIN="/share/apps/EC_STRUCT_SIM/DIGIMAT/v5.0.1/Digimat/Digimat/5.0.1/exec"

####################### SET LS-DYNA RUN PARAMETERS #######################

DEFAULT_NUM_NODES=2

DEFAULT_QUEUE=all

DEFAULT_NUM_NCPU_SMP=1

DEFAULT_NUM_NCPU_MPP=8

DEFAULT_NUM_NCPU_Hybrid=2

DEFAULT_NUM_THREADS_SMP=8

DEFAULT_NUM_THREADS_MPP=1

DEFAULT_NUM_THREADS_Hybrid=4

DEFAULT_MEMORY=777777777

DEFAULT_PARALLEL="MPP"

DEFAULT_PRECISION="single"

SOLVER_VERSIONS="971_5.1.1 971_6.1.1"

DEFAULT_VERSION=971_6.1.1

LATEST_VERSION=971_6.1.1

PREVIOS_VERSION=971_5.1.1

EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/LS-DYNA"

APPENDIX E – MADYMO Run Script Settings.

Location: 

/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/MADYMO_Script

Required Modules:

######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/madymo/R7.5
#######################################################################

Default Options:

 

######################## SET MADYMO RUN PARAMETERS #######################

DEFAULT_NUM_NODES=2

DEFAULT_QUEUE=all

DEFAULT_NUM_NCPU_SMP=8

DEFAULT_NUM_NCPU_MPP=8

DEFAULT_NUM_NCPU_Hybrid=2

DEFAULT_NUM_THREADS_SMP=8

DEFAULT_NUM_THREADS_MPP=1

DEFAULT_NUM_THREADS_Hybrid=4

DEFAULT_MEMORY=777777777

DEFAULT_PARALLEL="MPP"

DEFAULT_PRECISION="single"

SOLVER_VERSIONS="madymo_75"

DEFAULT_VERSION=madymo_75

LATEST_VERSION=madymo_75

PREVIOS_VERSION=madymo_75

EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/MADYMO"

APPENDIX F – OPTISTRUCT Run Script Settings.

Location: 

/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT OPTISTRUCT_Script

Required Modules:

######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/optistruct/opt_12.0.1
#######################################################################

Default Options:

 

##################### SET OPTISTRUCT RUN PARAMETERS #####################

DEFAULT_NUM_NODES=1

DEFAULT_QUEUE=all

DEFAULT_NUM_NCPU_SMP=1

DEFAULT_NUM_NCPU_MPP=4

DEFAULT_NUM_NCPU_Hybrid=4

DEFAULT_NUM_THREADS_SMP=4

DEFAULT_NUM_THREADS_MPP=4

DEFAULT_NUM_THREADS_Hybrid=4

DEFAULT_MEMORY=

DEFAULT_PARALLEL="SMP"

DEFAULT_PRECISION="double"

SOLVER_VERSIONS="12.0 12.0.210"

DEFAULT_VERSION=12.0.210

LATEST_VERSION=12.0.210

PREVIOS_VERSION=12.0

SOLVER=$solver

EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/HYPERWORKS/v12.0.1/altair/scripts"

TMP_DIR="/tmp"

Solver Notes:

SMP – Runs one process (Shared memory parallel) using multiple threads (1 x node max)

  • Must be run on 1 x node only (will auto default to 1 otherwise)

MPP – Runs the same as MPP mode (no MPP mode in OPTISTRUCT)

Hybrid – Natively (to Altair/OPTISTRUCT) known as SPMD (Single Program Multi Domain)

  • Minimum on 3x processes (1 x Manager, 2 x Run jobs)
  • Runs multiple SMP processes (best 4 threads/cores) per SMP process.
  • NUM_NODES = No. of loadcases + 1 (for manager) / 2 (2 x processes per node)

Parallel run speed up with OPTISTRUCT can only be gained if the job has multiple loadcases. Ie. Each loadcase is run on a SMP process in parallel.

APPENDIX G – RADIOSS Run Script Settings.

Location: 

/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/RADIOSS_Script

Required Modules:

######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/radioss/rad_12.0.1
#######################################################################

Default Options:

 

####################### SET RADIOSS RUN PARAMETERS ####################### 

DEFAULT_NUM_NODES=1

DEFAULT_QUEUE=all

DEFAULT_NUM_NCPU_SMP=1

DEFAULT_NUM_NCPU_MPP=4

DEFAULT_NUM_NCPU_Hybrid=4

DEFAULT_NUM_THREADS_SMP=4

DEFAULT_NUM_THREADS_MPP=4

DEFAULT_NUM_THREADS_Hybrid=4

DEFAULT_MEMORY=

DEFAULT_PARALLEL="SMP"

DEFAULT_PRECISION="double"

SOLVER_VERSIONS="12.0 12.0.210"

DEFAULT_VERSION=12.0.210

LATEST_VERSION=12.0.210

PREVIOS_VERSION=12.0

SOLVER=$solver

EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/HYPERWORKS/v12.0.1/altair/scripts"

TMP_DIR="/tmp"

Submitting jobs sequence — long runs

Posted by on February 8, 2014 No comments

Its quite simple to increase your job’s run time if needed by using job dependencies without compromising on the priorities of the queue. We want to keep queue very flexible, and encourage users to submit relatively short re-startable jobs.

If you need to submit long re-startable job to the queue you can use job “dependency” option in slurm.

E.g. , say queue “all” has time limit of 24 hours, but you need 3×24 hours for your calculations you can submit you job (same submission script from the same job folder) 3 times:

sbatch  myslurmscript.sh

Submitted batch job 81962

sbatch -d afterok:81962 myslurmscript.sh

Submitted batch job 81963

sbatch -d afterok:81963 myslurmscript.sh

there are also “afterany” and “afternotok” varieties of dependencies.

In this way 1st job 81962 will start and run, the other two will wait in the queue. After the 1st job is completed, next will start… and so on. Each will have 24 hours run time.

This would work if the code you are running is re-startable and can continue execution from the point where the previous its instance stopped. When queue server (SLURM) terminates the job, e.g. due to time limit has reached, it will send SIGTERM signal to your code’s processes on the nodes. You code can receive this signal and save all data needed for it to restart from the same position. After 120 seconds SLURM will send SIGKILL signal and processes will terminate. Normally 2 minutes is quite enough time to save all data to the disk needed to restart the code…

Regards,
Alex Pedcenko

Queues of zeus [updated June 2018]

Posted by on January 16, 2014 No comments

Queuing model UPDATE June 2018

To simplify usage of different queues, we combined all nodes into a single default queue (slurm partition) “all”. The usage limits are now solely user-based, each user has default (for now) number of CPU*minutes that they can use at any time moment. If this number of CPU*min is reached, the new jobs from this user will be put on queue until their running jobs will free resources. This is independent of type of compute nodes.

If you require a particular type of compute nodes (CPU/GPU/Phi etc), this can be done in submission script or during the submission with sbatch command: by specifying an additional parameter “constraint”:

    • for 32xCPU Broadwell nodes, specify --constraint=broadwell (54 nodes in total, 128GB RAM each)
    • for Nehalem (8-CPU) nodes, specify --constraint=nehalem (144 nodes in total, 48 GB RAM each)
    • for SandyBridge CPUs (12 CPUs/node), do --constraint=sandy(18 nodes in total, 48 GB RAM each)
    • for 512GB RAM SMP node (32 CPUs max) ask for --constraint=smp (1 node in total)
    • for Nx NVidia Kepler K20 GPUs, ask for --gres=gpu:K20:N (18 nodes x 2 GPUs in total)
    • for Nx NVidia Kepler K80 GPUs, ask for --gres=gpu:K80:N (10 nodes x 2 GPUs in total)
    • for N Intel Phi, ask for --gres=mic:N or --constraint=phi (2 nodes in total, also have 32 CPUs each).

If you have no particular preference on type of CPU or compute node and are running parallel job, please specify ONLY TOTAL Nr of CPUs required, NOT Nr of nodes!: SLURM will assign nodes automatically

e.g. if I need 64 CPUs in total for 24 hours on whatever available nodes I submit my slurm script with:

sbatch -n 64 -t 24:00:00 myslurmscriptname.slurm

if I need 64 CPUs in total for 48 hours on Broadwell-based nodes (32 CPUs/node) I submit my slurm script with:

sbatch -n 64 -t 48:00:00 --constraint=broadwell myslurmscriptname.slurm

Finally if I want 2 GPU nodes with 2 Nvidia Kepler K80 GPUs (1 GPUs/node) and 2 CPUs on each node, I do something like:

sbatch -N2 --ntasks-per-node=1 --cpus-per-task=2 --gres=gpu:K80:1 mygpuscript.slurm

full list of possible sbatch options is available in slurm docs: https://slurm.schedmd.com/sbatch.html

Note that if you do not specify the required time for your job (e.g. for 8 hours “… -t 8:00:00 …”), your job will be terminated after default time of 4 hours.

 

 

 

 

 

[Historical info below, we don’s have different queues at the moment]

Zeus has few queues (partitions)

  • “all” — default queue includes all the nodes of zeus from Nr 22 up: zeus[22-91,100-171,200-217] Wall time limit: 24 hours in this queue
  • “long” — Queue for long jobs (>24 hours>) include nodes zeus[22-80] infinite time limit
  • “GPU” — Queue for jobs on 18 Sandybridge nodes zeus[200-217] which have GPUs (two K20 each) Wall time limit in this queue is 36 hours
  • “SMP” — Queue for one node zeus5, 32 cores (4×8), 512Gb of RAM, Wall time: 48 Hours
  • “debug” — zeus[20-21] — for short tests, max wall time is 20 min.

%%%%%%%%%%%%% November 2016 UPDATE %%%%%%%%%%%%%%%%%

After major upgrade new partitions/queues are added see all the queues in the output of sinfo command. See current queue limitations here.

New queues:

New Broadwell CPU based nodes:

  • 44 nodes with 2 x Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz (32 CPU-cores/node) 128Gb RAM:   queue Broadwell
  • 10 nodes with 2 x Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz (32 CPU-cores/node) 128Gb RAM + 2x NVidia K80 GPU:  queue NGPU
  • 2 nodes  with 2xIntel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz (32 CPU-cores/node) 128Gb RAM + Xeon Phi coprocessor SE10/7120: queue Phi

%%%%%%%%%%%%% %%%%%%%%%%% %%%%%%%%%%%%%%%%%

to submit to either queue (below is “SMP” queue, queue name is case sensitive!):

sbatch --partition SMP slurmscriptname

If you do not specify walltime when submitting the job, your job will be terminated after 4 hours (either queue, apart from “debug”)!

To see the state of all queues:

sinfo

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
debug        up      20:00      2   idle zeus[20-21]
all*         up 1-00:00:00     25    mix zeus[22-35,37-40,74-80]
all*         up 1-00:00:00    134   idle zeus[41-73,81-91,100-171,200-217]
all*         up 1-00:00:00      1   down zeus36
long         up   infinite     25    mix zeus[22-35,37-40,74-80]
long         up   infinite     33   idle zeus[41-73]
long         up   infinite      1   down zeus36
GPU          up 1-12:00:00     18   idle zeus[200-217]
SMP          up 2-00:00:00      1   idle zeus5

To see the running jobs in all the queues do:

squeue

[aa3025@zeus3 ~]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            258451       GPU      lm0   ab7585  R   21:29:33     18 zeus[200-217]
            258486       GPU s455_tes  bakern6  R   12:20:38      1 zeus200
            258489       SMP     bash   ab1083  R      43:51      1 zeus5
            258460       all     Dyna  arestil  R   20:03:59      8 zeus[38-45]
            258485       all  Dropped  malinl2  R   12:48:08      3 zeus[27-29]
            258484       all  Ev_0.4m  malinl2  R   12:48:33      3 zeus[23-24,26]
            258466       all Lotus_MI massawaj  R   18:41:18      3 zeus[31-33]
            258478       all carcrash  lsdyna5  R   17:05:46      1 zeus72
            258431      long    duct1   dmitry  R   23:41:26      4 zeus[126-129]
            251602      long    duct1   dmitry  R 2-22:27:44      4 zeus[118-120,122]
            253428      long    duct1   dmitry  R 2-00:48:01      3 zeus[155,157-158]
            253427      long    duct1   dmitry  R 2-10:29:58      3 zeus[123-125]
            251518      long convnn_1  hermesm  R 5-16:03:16      1 zeus54
            251517      long convnn_1  hermesm  R 5-16:04:17      1 zeus54
            251407      long p1822999  bakern6  R 6-18:29:31      1 zeus56
            251406      long p1822300  bakern6  R 6-18:30:51      1 zeus59

to see only your jobs do:

squeue -u $USER

Listing properties of a job

to see detailed properties of a job, do
scontrol show jobid 258486

scontrol show jobid 258486

JobId=258486 Name=s455_test
   UserId=bakern6(581) GroupId=bakern6(581)
   Priority=13 Account=(null) QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=0 Restarts=0 BatchFlag=1 ExitCode=0:0
   RunTime=12:31:26 TimeLimit=1-12:00:00 TimeMin=N/A
   SubmitTime=2015-02-18T21:59:46 EligibleTime=2015-02-18T21:59:46
   StartTime=2015-02-18T21:59:46 EndTime=2015-02-20T09:59:46
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   Partition=GPU AllocNode:Sid=zeus3:9863
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=zeus200
   BatchHost=zeus200
   NumNodes=1 NumCPUs=1 CPUs/Task=1 ReqS:C:T=*:*:*
   MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) Gres=(null) Reservation=(null)
   Shared=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/home/bakern6/new/vortex_solve/job_455_1000_50.q
   WorkDir=/home/bakern6/new/vortex_solve
css.php