by Oliver Grimes
——————
MS-WORD *.doc version of this document.
EC_FEA_Run_Script
TAKE CARE AS MOST PARAMETERS ARE CASE SENSITIVE
The purpose of this script is to enable quick solver choice and settings for FEA runs on Coventry University HPC “ZEUS”. The following document details how to use and submit the script.
Currently Registered Solvers are:
LS-DYNA, OPTISTRUCT(_NL), RADIOSS, LS-OPT, LS-TASC, DIGIMAT_DYNA or MADYMO
You can find the most up to date EC_FEA_Run_Script.sh in your $HOME/RUN_SCRIPTS
HOW TO SUMBIT THE SCRIPT
- Ensure the script is in your run directory (ideally a clean directory)
- Places you input file (solver input deck) and any include folders in this folder.
- Set your preferred solver and settings as detailed in the following pages.
- The job is then ready to submit in either of the following ways:
- In a terminal window enter
./EC_FEA_Run_Script.sh
- In a terminal window enter
sh EC_FEA_Run_Script.sh
Note: Wildcards can be used to save time ie. ./EC_FE* or sh EC* etc.
- Any errors on warnings on submitting the job will appear in the standard output of the terminal window. It is import to read these warnings as error as the job may make adjustments to the solver settings if incorrect.
- On submission an incrementally numbered SOLUTION_#### folder and SOLUTION_####.out file will be produced. This contains your job solution and output log summary (which is useful to monitor whilst the job is initialising to check it doesn’t error terminate). The SOLUTION_#### names (folder and out log file) which automatically increment by number if the job is resubmitted in the same directory. This can be useful when debugging job and keeping a record of changes.
- Check the job is running using the “squeue –u username” command in a terminal.
IMPORTANT: Do not change the SOLUTION_#### names until the job has finished!!
USEFUL COMMANDS
squeue
– View the job queue
squeue -u username
– View just your user jobs (replace username with your’s)
scancel JOB_ID
– Cancel a job by JOB_ID (found in the squeue)
scancel -u username
– Cancel all your user jobs (replace username with your’s)
sinfo
– View queue partitions status
sbatch
– Manually submit a SUBMISSION_SCRIPT
RUN SCRIPT Parameters and Settings
The following section details the run script settings.
Input Filename:
Enter your filename (include the file extension) in the input parameter. This must not contain spaces, neither must the run directory.
Eg.
input="filename.ext"
Jobname:
Entry of the jobname is not essential but is very useful. The name entered under this parameter will appear as the name of the job in the job queue.
Eg.
jobname="JOB_123"
Solver Choice:
Enter the FEA solver you require from the list of registered solvers. This parameter is case sensitive so be careful to copy it directly as in the list.
Currently Registered Solvers are:
LS-DYNA, OPTISTRUCT(_NL), RADIOSS, LS-OPT, LS-TASC, DIGIMAT_DYNA or MADYMO
Eg.
solver=OPTISTRUCT
Number of Nodes for the job:
Enter the number of computer nodes required for the job. Most nodes on ZEUS contain 2 x Quad core processors (8 cores nodes) so two nodes provides 16 processors. Remember, more processors does not necessarily provide improvements in the runtime, some solvers don’t scale well. DO NOT WAIST HPC RESOURCE!! When running multiple jobs, running more jobs simultaneously on less processors will overall be quicker than with job stacked sequentially on more nodes.
Eg.
NUM_NODES=3
(must be “default” or an integer)
Note: Unless the more_nodes_password is supplied the max number of nodes maybe capped. If the specified nodes is over the capped limit NUM_NODES will be automatically set to the max allowed. A warning will be issued in such a case.
Walltime:
The walltime is the maximum time you think the job will take to run. If unsure, be overly cautious as the job will be terminated if the walltime is reached, even if the job hasn’t finished. However when the HPC queue is busy specifying lengthy walltime’s will deprioritise the job and take longer to start up. Try to keep this realistic.
Eg.
walltime=24:00:00
(for 24 hours)
ADVANCED Options
The second half of the EC_FEA_Run_Script.sh contain more settings for advanced users. Left these parameters as default using you know what they do.
For specific information on how these parameters relate to each solver see the APPENDIX section of these instructions.
#####################################################################
##### #####
##### ADVANCED OPTIONS #####
##### #####
#####################################################################
More Nodes Password:
Specify the more nodes password here if you have been given it by your supervisor. This will remove the student cap of the maximum number of nodes you can launch the job on.
Eg.
more_nodes_password="password"
Number of CPU’s:
This option is to specify the number of processes (domains/MPI processes etc.) the entire job requires. If set as default the most efficient configuration for specified solver with be chosen.
Eg.
NUM_NCPU=default
(must be “default” or an integer)
Number of Threads per process:
This option specifies the number of thread per process (specified above). For example a Hybrid job run multiple SMP processes using MPI, each SMP process will likely have multiple threads running using several cores. This parameter specifies in this example the number of threads/sub processes for the SMP process. Only set if you know what you are doing as job will run extremely slowly/inefficiently if set incorrectly. If set as default the most efficient configuration for specified solver with be chosen.
Eg.
NUM_THREADS=default
(must be “default” or an integer)
Memory for the job:
Specifies the required memory for the job. This units of this input are solver dependant (could be words, megawords, megabytes). Ideally leave as default unless needed.
Eg.
memory=default
(must be “default” or an integer)
Queue Partition:
To set a specific SLURM queue partition enter the required partition name using this parameter. Usually set as default which choses the most relevant queue for the solver choice (usually the “all” queue).
Eg.
queue=default
(must be “default” or an existing queue partition name)
Parallel Processing Type:
This option let you choose your parallel processing job (assuming the corresponding solver in compatible and installed). The options are default, SMP, MPP, Hybrid (Using multiple SMP). Leave as default unless you know the job with be faster/more efficient using a different processing type.
Eg.
parallel=default
Precision Type:
Sets the solver to process in either single or double precision (floating point length). Only require for some simulation type. Normally costs approximately 20-30% extra runtime. The default option is usually single precision.
Eg.
precision=default
(must be “default”, single or double)
Solver Version:
A different version of the solver can be called be setting this parameter. The specific option relates to the version folder name in the /share/apps/EC_STRUCT_SIM/%solver%/%version folder%. Leave as default unless required.
Eg.
version=default
(must be “default”, previous, latest, specific (ie. 971_R5.1.1))
Other Solver Command Line Options/Arguments:
If the user wants to provide any additional solver options/arguments then this parameter will add them to the end of the default command line.
Eg.
OTHER_ARGS="-option argument=1"
DO NOT EDIT THE SCRIPT BEYOND THE ABOVE PARAMETER
APPENDIX – SOLVER SPECIFIC SETTINGS
The follow appendix details how the above settings relate to the specific solver settings.
APPENDIX A – LS-DYNA Run Script Settings.
Location:
/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/LS-DYNA_Script
Required Modules:
######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/LS-DYNA/v971
#######################################################################
Default Options:
###################### SET LS-DYNA RUN PARAMETERS #####################
DEFAULT_NUM_NODES=2
DEFAULT_QUEUE=all
DEFAULT_NUM_NCPU_SMP=1
DEFAULT_NUM_NCPU_MPP=8
DEFAULT_NUM_NCPU_Hybrid=2
DEFAULT_NUM_THREADS_SMP=8
DEFAULT_NUM_THREADS_MPP=1
DEFAULT_NUM_THREADS_Hybrid=4
DEFAULT_MEMORY=777777777
DEFAULT_PARALLEL="MPP"
DEFAULT_PRECISION="single"
SOLVER_VERSIONS="971_5.1.1 971_6.1.1 971_6.1.2"
DEFAULT_VERSION=971_6.1.2
LATEST_VERSION=971_6.1.2
PREVIOS_VERSION=971_5.1.1
EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/LS-DYNA"
Solver Notes:
SMP – Runs one process (Shared memory parallel) using multiple threads.
- Must be run on 1 x node only (will auto default to 1 otherwise)
MPP– Runs one serial process per core (Massively Parallel Processing) using HPMPI.
- Total cores = NUM_NODES * NUM_NCPU
Hybrid – Runs parallel SMP processes (which can each run parallel threads) using HPMPI.
- No. of SMP processes = NUM_NODES * NUM_NCPU / NUM_THREADS
- No of threads per SMP process = NUM_THREADS (best to use a whole physical processor=4)
- Total cores = No. of SMP processes * No. of thread per SMP process
Location:
/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/LS-OPT_Script
Required Modules:
######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/LS-DYNA/v971
#######################################################################
Default Options:
####################### SET LS-TASC RUN PARAMETERS #######################
LS_OPT_PATH=/share/apps/EC_STRUCT_SIM/LS-OPT/v5.0_84950
####################### SET LS-DYNA RUN PARAMETERS #######################
input=DynaOpt.inp
jobname=$(basename `pwd`)
DEFAULT_NUM_NODES=1
DEFAULT_QUEUE=all
DEFAULT_NUM_NCPU_SMP=1
DEFAULT_NUM_NCPU_MPP=8
DEFAULT_NUM_NCPU_Hybrid=2
DEFAULT_NUM_THREADS_SMP=8
DEFAULT_NUM_THREADS_MPP=1
DEFAULT_NUM_THREADS_Hybrid=4
DEFAULT_MEMORY=777777777
DEFAULT_PARALLEL="MPP"
DEFAULT_PRECISION="single"
SOLVER_VERSIONS="971_5.1.1 971_6.1.1 971_6.1.2"
DEFAULT_VERSION=971_6.1.2
LATEST_VERSION=971_6.1.2
PREVIOS_VERSION=971_5.1.1
EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/LS-DYNA"
Location:
/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/ LS-TASC_Script
Required Modules:
######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/LS-DYNA/v971
#######################################################################
Default Options:
####################### SET LS-TASC RUN PARAMETERS #######################
LS_OPT_PATH=/share/apps/EC_STRUCT_SIM/LS-TASC/lstasc_21_86715_x64_rhel6
####################### SET LS-DYNA RUN PARAMETERS #######################
input=DynaOpt.inp
jobname=$(basename `pwd`)
DEFAULT_NUM_NODES=1
DEFAULT_QUEUE=all
DEFAULT_NUM_NCPU_SMP=1
DEFAULT_NUM_NCPU_MPP=8
DEFAULT_NUM_NCPU_Hybrid=2
DEFAULT_NUM_THREADS_SMP=8
DEFAULT_NUM_THREADS_MPP=1
DEFAULT_NUM_THREADS_Hybrid=4
DEFAULT_MEMORY=777777777
DEFAULT_PARALLEL="MPP"
DEFAULT_PRECISION="single"
SOLVER_VERSIONS="971_5.1.1 971_6.1.1 971_6.1.2"
DEFAULT_VERSION=971_6.1.2
LATEST_VERSION=971_6.1.2
PREVIOS_VERSION=971_5.1.1
EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/LS-DYNA"
Location:
/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/DIGIMAT_DYNA_Script
Required Modules:
######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/LS-DYNA/v971
module load /share/apps/EC_STRUCT_SIM/modulefiles/digimat/v5.0.1
#######################################################################
Default Options:
####################### SET DIGIMAT RUN PARAMETERS #######################
DEFAULT_DIGI_DYNA_VERSION=V611
LATEST_DIGI_DYNA_VERSION=V611
PREVIOS_DIGI_DYNA_VERSION=V511
DIGIMAT2CAE="/share/apps/EC_STRUCT_SIM/DIGIMAT/v5.0.1/Digimat/Digimat2CAE/5.0.1"
DIGIMAT_BIN="/share/apps/EC_STRUCT_SIM/DIGIMAT/v5.0.1/Digimat/Digimat/5.0.1/exec"
####################### SET LS-DYNA RUN PARAMETERS #######################
DEFAULT_NUM_NODES=2
DEFAULT_QUEUE=all
DEFAULT_NUM_NCPU_SMP=1
DEFAULT_NUM_NCPU_MPP=8
DEFAULT_NUM_NCPU_Hybrid=2
DEFAULT_NUM_THREADS_SMP=8
DEFAULT_NUM_THREADS_MPP=1
DEFAULT_NUM_THREADS_Hybrid=4
DEFAULT_MEMORY=777777777
DEFAULT_PARALLEL="MPP"
DEFAULT_PRECISION="single"
SOLVER_VERSIONS="971_5.1.1 971_6.1.1"
DEFAULT_VERSION=971_6.1.1
LATEST_VERSION=971_6.1.1
PREVIOS_VERSION=971_5.1.1
EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/LS-DYNA"
Location:
/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/MADYMO_Script
Required Modules:
######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/madymo/R7.5
#######################################################################
Default Options:
######################## SET MADYMO RUN PARAMETERS #######################
DEFAULT_NUM_NODES=2
DEFAULT_QUEUE=all
DEFAULT_NUM_NCPU_SMP=8
DEFAULT_NUM_NCPU_MPP=8
DEFAULT_NUM_NCPU_Hybrid=2
DEFAULT_NUM_THREADS_SMP=8
DEFAULT_NUM_THREADS_MPP=1
DEFAULT_NUM_THREADS_Hybrid=4
DEFAULT_MEMORY=777777777
DEFAULT_PARALLEL="MPP"
DEFAULT_PRECISION="single"
SOLVER_VERSIONS="madymo_75"
DEFAULT_VERSION=madymo_75
LATEST_VERSION=madymo_75
PREVIOS_VERSION=madymo_75
EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/MADYMO"
Location:
/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT OPTISTRUCT_Script
Required Modules:
######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/optistruct/opt_12.0.1
#######################################################################
Default Options:
##################### SET OPTISTRUCT RUN PARAMETERS #####################
DEFAULT_NUM_NODES=1
DEFAULT_QUEUE=all
DEFAULT_NUM_NCPU_SMP=1
DEFAULT_NUM_NCPU_MPP=4
DEFAULT_NUM_NCPU_Hybrid=4
DEFAULT_NUM_THREADS_SMP=4
DEFAULT_NUM_THREADS_MPP=4
DEFAULT_NUM_THREADS_Hybrid=4
DEFAULT_MEMORY=
DEFAULT_PARALLEL="SMP"
DEFAULT_PRECISION="double"
SOLVER_VERSIONS="12.0 12.0.210"
DEFAULT_VERSION=12.0.210
LATEST_VERSION=12.0.210
PREVIOS_VERSION=12.0
SOLVER=$solver
EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/HYPERWORKS/v12.0.1/altair/scripts"
TMP_DIR="/tmp"
Solver Notes:
SMP – Runs one process (Shared memory parallel) using multiple threads (1 x node max)
- Must be run on 1 x node only (will auto default to 1 otherwise)
MPP – Runs the same as MPP mode (no MPP mode in OPTISTRUCT)
Hybrid – Natively (to Altair/OPTISTRUCT) known as SPMD (Single Program Multi Domain)
- Minimum on 3x processes (1 x Manager, 2 x Run jobs)
- Runs multiple SMP processes (best 4 threads/cores) per SMP process.
- NUM_NODES = No. of loadcases + 1 (for manager) / 2 (2 x processes per node)
Parallel run speed up with OPTISTRUCT can only be gained if the job has multiple loadcases. Ie. Each loadcase is run on a SMP process in parallel.
Location:
/share/apps/EC_STRUCT_SIM/RUN_SCRIPTS/MASTER_SCRIPT/RADIOSS_Script
Required Modules:
######################## LOAD REQUIRED MODULES ########################
module purge
module load /share/apps/EC_STRUCT_SIM/modulefiles/radioss/rad_12.0.1
#######################################################################
Default Options:
####################### SET RADIOSS RUN PARAMETERS #######################
DEFAULT_NUM_NODES=1
DEFAULT_QUEUE=all
DEFAULT_NUM_NCPU_SMP=1
DEFAULT_NUM_NCPU_MPP=4
DEFAULT_NUM_NCPU_Hybrid=4
DEFAULT_NUM_THREADS_SMP=4
DEFAULT_NUM_THREADS_MPP=4
DEFAULT_NUM_THREADS_Hybrid=4
DEFAULT_MEMORY=
DEFAULT_PARALLEL="SMP"
DEFAULT_PRECISION="double"
SOLVER_VERSIONS="12.0 12.0.210"
DEFAULT_VERSION=12.0.210
LATEST_VERSION=12.0.210
PREVIOS_VERSION=12.0
SOLVER=$solver
EXECUTABLE_DIR="/share/apps/EC_STRUCT_SIM/HYPERWORKS/v12.0.1/altair/scripts"
TMP_DIR="/tmp"
Recent Comments