reconstructPar in parallel on local node or in slurm queue

When your parallel case finished solution, reconstructPar will assemble OpenFoam case from multiple processor* folders into one. It does so, by default in serial facion: i.e. using only one CPU core. That can take very long time.

The way to seed it up is to chop your time instants data into bunches: each bunch for each CPU will contain sequence of time intervals to reconstruct. This can be done with the following script launched from case folder:

#!/bin/bash

# parameters
dt=1.0;   # OpenFoam solution write interval [seconds], leave as decimal! (i.e. dt = deltaT * writeInterval)
start=0;  # initial time [seconds]
stop=201; # end time [seconds]

# do not change below
# how many reconstructPar processes to launch (e.g. Nr of CPUs we have)

cpus=$(cat /proc/cpuinfo | grep processor | wc -l);

##### magic: ###### picking time instants for each cpu
step=`expr $dt*$cpus | bc`
last=`expr $step-$dt | bc`
for i in `seq $start $dt $last`;do
   times=$(seq -s, $i $step $stop)
   reconstructPar -time $times &
done

This script above can be launched on e.g. single machine/compute node.

 

reconstructPar as a parallel job

If you have a ginormous case though, you may want to launch reconstructPar in the slurm queue as a separate job. Then you can use the following slurm script (save it as e.g. “reconstruct.sh”, make it executable (chmod +x reconstruct.sh) and submit it as SLURM array task, e.g. as 12 array tasks with:

sbatch --array=0-11 reconstruct.sh

Or if you want one task per each compute node (e.g. memory considerations):

sbatch --array=0-11 -N1  --exclusive reconstruct.sh

Listing of reconstruct.sh:

#!/bin/bash
#SBATCH -n 1

module load openfoam/6 # or whichever way you activate your openfoam

dt=1.0;   # OpenFoam solution write interval [seconds], leave as decimal! (i.e. dt = deltaT * writeInterval)
start=0;  # initial time [seconds]
stop=201; # end time [seconds]

# do not change below
# how many reconstructPar processes to launch (e.g. Nr of CPUs we have)

cpus=$SLURM_ARRAY_TASK_COUNT;

##### magic: ###### picking time instants for each cpu
step=`expr $dt*$cpus | bc`
last=`expr $step-$dt | bc`

declare -a times=()
cpu=0
for i in `seq $start $dt $last`;do
   times[$cpu]="reconstructPar -time "$(seq -s, $i $step $stop)
   cpu=$(( cpu + 1 ))
done

# launch one reconstruct per each array task
${times[$SLURM_ARRAY_TASK_ID]}

Alex Pedcenko

css.php