Submitting array jobs to the Sun Grid Engine

From Research Computing

The following is an example array job. More details on this to follow.

[flengyel@nept sge]$ cat array.job
#!/bin/bash 
#$ -cwd
#$ -S /bin/bash
#$ -o /tmp/$USER.$JOB_ID.$TASK_ID

echo "NOTE: submit this job as qsub -t 1-3 array.job"
echo Your program text would normally go here.
echo Use an  output file such as above
echo to write into  the /tmp directory:
echo NFS file usage will be reduced.
echo The temporary output can be moved
echo at the end of the program, as below.
echo This workaround was needed, since SGE  
echo syntax within \#\$ is  limited:
echo arrays do not work within \#\$.
echo However, shell commands work normally
echo provided the -S command is used above.
echo 
echo NOTE: \$TASK_ID is used with -o 
echo but  \$SGE_TASK_ID works here instead.
echo

# This is the syntax to define arrays of names
# The entry to ignore is the zeroth.
arr=( 
  ignorethis.txt 
  firstfile.txt 
  secondfile.txt 
  thirdfile.txt  
)
 
echo /tmp/$USER.$JOB_ID.$SGE_TASK_ID 
# copy file in /tmp to job submission directory $SGE_CWD_PATH using the
# file name determined by $SGE_TASK_ID, which in this case is in {1,2,3}
mv /tmp/$USER.$JOB_ID.$SGE_TASK_ID  $SGE_CWD_PATH/${arr[$SGE_TASK_ID]}