If your OpenFOAM problem does a lot of I/O (read/write) operations during the run and especially if you have a huge mesh, you may want to run the case from the local harddrives of the compute nodes (SSDs), instead of using your network-attached home folder (all users’ homes are shared across the compute nodes via NFS network-shares). The submission script below demonstrates how to do that:
- it will first copy the whole content of your case folder (so make sure its “clean” prior to submission to save time of copying) to the local harddrives of compute nodes’ in “/scratch/$USER” folder;
- it will launch the chosen OpenFOAM solver on the compute nodes;
- once the job finishes, the calculation results are copied back to the initial case folder in your home (the one you submitted the job from).
Beware, if you terminate the run before the end, the results will not be copied back to your home folder. In this case must use the “proper” termination of the OpenFOAM run, i.e. to update your controlDict dictionary to read: stopAt writeNow; instead of endTime. The solver will stop the iterations after writing data for the current time step and the rest of the slurm script, which deals with copying the results back to your home folder will be executed.
OK, the slurm script:
#!/bin/bash
#SBATCH --partition=all
#SBATCH --nodes=18
#SBATCH --time=8:00:00
#SBATCH --job-name="foam"
#SBATCH --ntasks-per-node=8
source /share/apps/OpenFOAM/start.sh
#./Allclean
# in case you want to launch reconstructPar in parallel automatically
# on job end (check that you have preconstructPar.slurm in your case folder):
echo "sbatch --dependency=afterok:${SLURM_JOBID} preconstructPar.slurm" > reconstructor.sh
chmod +x reconstructor.sh
#---------------------------------------------------------
WORKDIR=$SLURM_SUBMIT_DIR
echo "mydir1:" $SLURM_SUBMIT_DIR
. $WM_PROJECT_DIR/bin/tools/RunFunctions
application=`getApplication`
runApplication blockMesh
# do whatever you have to here, e.g.
# cp ${WORKDIR}/0/T.org ${WORKDIR}/0/T
### or
# runApplication setFields
decomposePar # now decomposing case
#-------------------------Now the magic begins -----------
WORKDIROLD=${WORKDIR}
WORKDIR=/scratch/$USER/$SLURM_JOBID # local folder on the nodes
pdsh -w ${SLURM_NODELIST} rm -fr /scratch/$USER
pdsh -w ${SLURM_NODELIST} mkdir -p ${WORKDIR}
####echo "mydir2:" $WORKDIROLD
pdsh -w ${SLURM_NODELIST} cp -r ${WORKDIROLD}/processor* ${WORKDIR}/
pdsh -w ${SLURM_NODELIST} cp -r ${WORKDIROLD}/constant ${WORKDIR}/
pdsh -w ${SLURM_NODELIST} cp -r ${WORKDIROLD}/system ${WORKDIR}/
pdsh -w ${SLURM_NODELIST} cp -r ${WORKDIROLD}/0 ${WORKDIR}/
cd ${WORKDIR}
mydir=`pwd`
echo "changed to $mydir form $WORKDIROLD"
#------------------------- Generate hosts file --------------
HOSTS=`srun /bin/hostname|sort`
echo ${HOSTS} | sed 's/ /n/g'> ${WORKDIROLD}/hostfile.txt
## Nr of CPU's in total we reserved
NPROCS=$(( $SLURM_JOB_NUM_NODES * $SLURM_NTASKS_PER_NODE ))
#-------------------------Launching the job -----------------
MPIRUN=/opt/mpi/bullxmpi/1.2.4.3/bin/mpirun
EXECUTABLE=${application}
ARGS="-parallel"
command="${MPIRUN} -np ${NPROCS} -hostfile ${WORKDIROLD}/hostfile.txt ${EXECUTABLE} ${ARGS}"
echo ${command} >> $WORKDIROLD/debug.log
#----------------------- launch the executable ---------------
${command} >> $SLURM_SUBMIT_DIR/debug.log
#-------------------------Copy the results back to original folder----
pdsh -w $SLURM_NODELIST cp -rf ${WORKDIR}/processor* ${WORKDIROLD}/
sleep 3
# clean the scratch folder on the nodes:
pdsh -w ${SLURM_NODELIST} rm -fr $WORKDIR
cd ${WORKDIROLD}
This script will also create “reconstructor.sh” file for reconstruction of the decomposed fields. Straight after submitting the above slurm script do:
./reconstructor.sh
This will place reconstructPar job into the queue which will be delayed until your OpenFOAM case completes.
Alex Pedcenko
0 Comments.