MaxClusterUsage
Maximum Usage of ITSC Cluster Resource¶
Info
Contributed by masterzhen119 on 2021-10-14.
As PhD members in Statistics, we have two types of ITSC source.
- University level: Everyone owns 10 running jobs without limitation of cores simutanously.
- Division level: Members from Statistic division could run 30 cores (or 30 jobs) at most.
Sometimes, when you have multiple tests in parallel like 100. Each one will take more than 1 hour. The wise thought is to use both resource of University and Division level. Here comes the way: we want to use bash file to allocate first 10 jobs to University level and the rest for STAT. The bash file is:
#!/bin/bash
dir="/lustre/project/Stat/path/of/names/of/genes"
cd /lustre/project/Stat/code/job/path/
num=1
c=11
for i in $(ls $dir)
do
if (($num < $c))
then
sbatch --export=p=${i} try3.job
echo $num
else
sbatch -p stat -q stat --export=p=${i} try3.job
fi
pwd
num=$[$num+1]
sleep 1
done
In this way, we can pass the gene name to job file, and the corresponding job file is:
#!/bin/bash
#SBATCH -J job_name
#SBATCH -N 1 -c 1
#SBATCH --output= job_name-%j-%a_out.log
#SBATCH --error= job_name-%j-%a_err.log
cd /lustre/project/Statcode/job/path/
python3 example.py $p
We have pass the name to python. The rest thing you need to do :
import sys
gene_name = sys.argv[1]