Submitting OpenMPI jobs to the Sun Grid Engine

From Research Computing

My thanks to Zhao Chen for supplying the example code for this documentation. We'll show how to compile an OpenMPI program and submit to the cluster. You should be logged into neptune.gc.cuny.edu to ensure that your environment variables are set up to use OpenMPI.

Here is a sample MPI program, called sum.cpp

#include<iostream.h>
#include<mpi.h>

int main(int argc, char ** argv){
  int mynode, totalnodes;
  int sum,startval,endval,accum;
  MPI_Status status;

  MPI_Init(&argc,&argv);
  MPI_Comm_size(MPI_COMM_WORLD, &totalnodes);
  MPI_Comm_rank(MPI_COMM_WORLD, &mynode);
	
  sum = 0;
  startval = 1000*mynode/totalnodes+1;
  endval =   1000*(mynode+1)/totalnodes;

  for(int i=startval;i<=endval;i=i+1)
     sum = sum + i;

  if(mynode!=0)
    MPI_Send(&sum,1,MPI_INT,0,1,MPI_COMM_WORLD);
  else
    for(int j=1;j<totalnodes;j=j+1){
      MPI_Recv(&accum,1,MPI_INT,j,1,MPI_COMM_WORLD, &status);
      sum = sum + accum;
  }
   
  
  if(mynode == 0)
    cout << "The sum from 1 to 1000 is: " << sum << endl;	
  
  MPI_Finalize();
  
}

The program is compiled with mpiCC (on neptune.gc.cuny.edu):

[flengyel@nept OPENMPI]$ mpiCC -o sum sum.cpp
[flengyel@nept OPENMPI]$ 

There were no errors, and a program called sum was created:

[flengyel@nept OPENMPI]$ ls -l sum
-rwxr-xr-x 1 flengyel domusers 964160 Apr 14 13:08 sum

We can first test this program on neptune before submitting to the cluster by running it:

[flengyel@nept OPENMPI]$ ./sum
The sum from 1 to 1000 is: 500500
[flengyel@nept OPENMPI]$ 

However, we will want to submit it to the computational cluster. We'll need a batch file for this, as follows. We'll assume you know how to create and edit files on Linux.

#!/bin/bash
#$ -S /bin/bash
#$ -q x86_64.q
#$ -N sum
#$ -pe ompi 2
#$ -cwd

. /usr/local/sge/default/common/settings.sh
export PATH=/home/nept/apps64/openmpi/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/nept/apps64/openmpi/lib
mpirun --prefix /home/nept/apps64/openmpi -v -np $NSLOTS ./sum

You can use the quad.q for quad core machines in place of x86_64.q, which is for dual core 64-bit machines. The script should be executable:

[flengyel@nept OPENMPI]$ chmod +x sum.sh
[flengyel@nept OPENMPI]$ ls -l sum.sh
-rwxr-xr-x 1 flengyel domusers 331 Apr 14 13:13 sum.sh

Now we can submit the job using qsub. See the instructions on SGE for more details.

[flengyel@nept OPENMPI]$ qsub sum.sh Your job 19758 ("sum") has been submitted