LS-DYNA

LS-Dyna R11

to submit LS-Dyna R11 job to SLURM queue you need to load the appropriate module first:

module load lsdyna/R11_XXX

and (if you use parallel runs)

module load lsdyna/pmpi

Then you can select appropriate solver variety (MPP or SMP [parallel or not-parallel (but multi-threaded), DP or SP [double or single precision]]
using the following command:

EXECUTABLE=`ls $LSDYNA_HOME/R11_142770/$EXEC_TYPE/$PREC/*$CPU*`

  • where variable LSDYNA_HOME is the location of LSDYNA (it is defined in lsdyna/R11_XXX, modulefile that you loaded so you don’t have to set it yourself)
  • EXEC_TYPE=”MPP”   (or “SMP”)
  • PREC=”DP”   (or SP) is precision
  • CPU = “sse” (or “avx2”, use avx2 for Broadwell CPUs only, older ones do not support these CPU optimizations)

The example slurm file, which automates all this procedure can be found at  http://zeus/slurm/slurm_scripts/lsdyna_R11_xxx.slurm

This SLURM file can be launched as ( e.g. on 64 CPUs of any kind for 12 hours )

sbatch -n64 -t 12:00:00 lsdyna_R11_xxx.slurm

If you want to use AVX2 solvers (i.e. you selected  CPU=”avx2″) in slurm file, chose Broadwell 932-CPU) nodes:

sbatch -n64 -t 12:00:00 -- constraint=Broadwell lsdyna_R11_xxx.slurm

LS-Dyna 9

SLURM file for LS-Dyna 9.1.0 submission is  here lsdyna.

You have to put the submission script e.g. “lsdyna.slurm” in your case folder with all simulation files, which are necessary for the solver to run.
E.g. I put some test key-file of LS-DYNA “YARI_2_YARI_STD.key” and the submission script in the same separate folder, say, /home/myusername/lsdyna/test1/.

The script below is for 9.0.1 version of LS-DYNA, otherwise define the version required at the line of slurm script below, in variable LSDYNA_VER=”…”
You can see all versions available on Zeus by listing content of folder where all executables reside.

ls /share/apps/EC_STRUCT_SIM/LS-DYNA/

You can also put your own executable in your home folder and use that instead, changing its location path in the submission script below.

Two modulefiles are required for LSDYNA to work, one is general module load lsdyna/971, which defines license server location etc, and the other defining the version of MPI used (in pre-8 versions of LSDYNA we used HPMPI) in 8+ it is IBM Platform MPI, so module load lsdyna/pmpi is needed in your slurm script. Note that this script do not specify number of nodes or CPUs required, see note below.

#!/bin/bash

#SBATCH --time=12:00:00
#SBATCH --job-name="lsdyna9"

HOSTFILE=`srun /bin/hostname`
echo $HOSTFILE > hostlist.txt

WORKDIR=${SLURM_SUBMIT_DIR}
LOGFILE=${SLURM_SUBMIT_DIR}/${SLURM_JOBID}_log.txt

module purge
module load lsdyna/971
module load lsdyna/pmpi

LSDYNA_VER="971_9.0.1"

EXEC_TYPE="MPP" #or SMP or Hybrid
PREC="DP" # DP or SP  (double or single precision)

EXECUTABLE="$LSDYNA_HOME/$LSDYNA_VER/$EXEC_TYPE/$PREC/*.exe"

KEYFILE="YARI_2_YARI_STD" # keyfile WITHOUT extension .key

keyfile=${WORKDIR}/${KEYFILE}

MPIRUN=`which mpirun`

export MPI_ROOT=$(dirname $(dirname "$MPIRUN"))

ARGS="i=${keyfile}.key O=${keyfile}.otf G=${keyfile}.ptf D=${keyfile}.dpf F=${keyfile}.thf U=${keyfile}.xtf memory=500m"   #500m memory words is 4Gb per core

NPROCS=$SLURM_NTASKS

echo "Nodelist: ${SLURM_JOB_NODELIST}" >> ${LOGFILE}_nodes

CMD="${MPIRUN} -hostfile ${WORKDIR}/hostlist.txt ${EXECUTABLE} ${ARGS}"

echo "Launching: $CMD" >> ${LOGFILE}_cmd

$CMD | tee ${SLURM_JOBID}_out.log

Submit to the queue with e.g.

sbatch -N2 -n32 --ntasks-per-node=16 -t 24:00:00 -p Broadwell --exclusive lsdyna.slurm

The above command reserves 24 hours for 32 CPU cores of two Broadwell nodes, 16 CPU-cores per node. Trials show that using more cores per node (each Broadwell node has 2×16 CPU-cores in 2 CPU-sockets) in MPP version of LSDYNA results in deterioration of the overall performance. So using fewer CPU-cores per node gives the same speed of solution as using more than 16 cores (or improvement is barely noticeable).

Alex Pedcenko

css.php