The total number of nodes (or CPUs) you can use depend on how long your job has to run (i.e. in which queue/partition it was submitted):

[queues are listed from higher to lower priority, i.e. shorter queues have higher priority in the waiting list!]

for up to 4-hours short jobs:

• “short4” queue has unlimited number  of CPUs for up to 4 hours  and can use nodes from any of the queues! (Hint: do not specify how many nodes you need, just specify how many CPU-cores you need for your job, i.e. for 1000 CPU-cores for 4 hours “sbatch -p short4 -n 1000 -t 4:00:00  submitscript.slurm”). Default time (if you do not specify walltime) is 1 hr.

for up to 12hours jobs:

• You can use up to 144 nodes x 8 CPUs  of “all12”  queue for up to 12 hours.

For up to 24hrs jobs:

• You can use up to 80 nodes x 8 CPUs  in “all” queue (640 CPUs) for up to 24 hours.
• You can also use 10 nodes x 32 CPUs of Broadwell queue (another 320 CPUs, use less nodes per job, these are “fat” nodes! ) for up to 36 hours
• You can also use 20 nodes x 8 CPUs of “all48” (another 160 CPUs) queue for up to 48 hours (lower priority than “all“)

For up to 36hrs jobs:

• You can use up to 10 nodes x 32CPUs in Broadwell queue
• You can use up to 5 nodes x 32CPUs (160 CPUs and 10 K80 GPUs) in NGPU queue
• You can use up to 18 nodes x 12 CPUs in (+ 36 K20 GPUs)  in GPU queue
• You can use 1 SMP node (32 CPUs and 512 GB RAM) in SMP queue

(specialized queues SMP,GPU,NGPU have higher priority in the waiting list, i.e. if you need to use GPUs on these nodes, you have higher “weight”)

For up to 48hrs jobs:

• You can use 20 nodes x 8 CPUs (160 CPUs in total) in queue “all48” for up to 48 hours

For >48 hrs long jobs:

• You can use up to 20 nodes (160 CPUs) in queue “long” for unlimited job time

There is a standing reservation of all nodes in “all” queue and “Broadwell+NGPU+Phi” nodes for this Sunday 13/11/16 from 0:00 to 12:00, which is needed for conducting more performance tests before commissioning of HPC. So If your submitted job spans through this time period you will get this message as a reason for “queueing”.

Alex Pedcenko