Research Computing - Recent changes [en]

Research Computing - Recent changes [en] 2009-11-20T18:42:11Z Track the most recent changes to the wiki on this page. MediaWiki 1.4.1 Research Computing 2009-11-16T04:48:09Z 2009-11-16T04:48:09 2009-11-16T04:48:09Z

WikiSysop foobar Getting Started 2009-11-16T04:47:16Z 2009-11-16T04:47:16 2009-11-16T04:47:16Z

New page<div>The CUNY Research Computing cluster is composed of a few servers and a number of computing nodes. To start, we show an example to submit a binary executable job to the cluster. To use the cluster, you must have an account on the Graduate Center Research Computer Cluster. If you need one, please follow the application process [http://research.gc.cuny.edu/wiki/index.php/How_to_apply_for_a_computational_cluster_account here]. Now suppose you have the log in to the cluster, 1. From a secure shell terminal, e.g., [http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe putty] under windows or a Linux / Mac terminal ssh -l <your username> neptune.gc.cuny.edu 2. Now you are on a linux shell environment. Make a new directory for your program and create your program. mkdir myfirstproject cd myfirstproject vim example.cpp 3. Finish writing your java program and save it to the cluster. Compile locally at the neptune server by g++ example.cpp -o example 4. NOW, you are NOT suppose to run your executable ''example'' on the cluster, but rather, you need to submit it as a script job to the cluster. To do this, you need a job execution script. vim example.sh and put the following lines in the shell script #$ -S /bin/bash #$ -q quad.q #$ -m abe #$ -M youremail@yourdomain.com #$ -o $HOME/myfirstproject/example_result #$ -e $HOME/myfirstproject/example_exception $HOME/myfirstproject/example The script tells the cluster a few things. #$ -S /bin/bash #$ -q quad.q The above lines tell the system the script should be executed using shell commands in /bin/bash, you could change this to other shell (e.g., C shell) by specifying the corresponding shell path. It also tells the job should be run on the quad.q queue of the cluster, which is a set of computing nodes, each composed of Intel Q9550 quad core CPU and 8GB of memory. The queue name can also be x86_64.q, which are of specification Intel Core 2 Duo 8400 CPU and 4GB of memory. #$ -m abe #$ -M youremail@yourdomain.com The two lines above tell you when your job begins to execute, ends, or aborts (due to some exception), the system should send you an email to youremail@yourdomain.com (change this to your own email). #$ -o $HOME/myfirstproject/example_result #$ -e $HOME/myfirstproject/example_exception The above lines tell the system to redirect the system standard stream to file $HOME/myfirstproject/example_result, and system standard error stream to $HOME/myfirstproject/example_exception, where $HOME is your home directory on the cluster. You program is supposed to output the result in standard output stream (e.g., std::cout in C++, and system.out.print in java). If there was any exception, e.g., segmentation fault, you should check the standard error file ''example_exception''. The last line, which does not begin with a "#" sign, is the actual script to be executed by the cluster. $HOME/myfirstproject/example 5. After finish the script file, you could now submit it to the cluster by qsub example.sh to the cluster. Watch out your email when the job is done (or aborted) and check your result at "example_result" or reason for aborting at "example_exception". Have fun.</div>

WikiSysop foobar Research Computing 2009-11-16T04:46:49Z 2009-11-16T04:46:49 2009-11-16T04:46:49Z

/* If you are new to the computational cluster */ <table border='0' width='98%' cellpadding='0' cellspacing='4' class='diff'> <tr> <td colspan='2' width='50%' align='center' class='diff-otitle'>Revision as of 04:46, 16 Nov 2009</td> <td colspan='2' width='50%' align='center' class='diff-ntitle'>Current revision</td> </tr> <tr><td colspan="2" align="left">Line 33:</td> <td colspan="2" align="left">Line 33:</td></tr> <tr><td> </td><td class='diff-context'>In March 2008, the Graduate Center became the twenty-first institution state-wide – and the first CUNY campus – to join the [http://www.nysgrid.org New York State Grid (NYSGrid)]. NYSGrid was formed in 2006 when member institutions united to facilitate information and technology exchange, to develop an advanced technological infrastructure across the state, and to provide access to computational and data resources on a statewide grid basis. NYSGrid itself is a "virtual organization" within the [http://www.opensciencegrid.org Open Science Grid], a national, distributed computing grid for data-intensive research, supported by the [http://www.nsf.gov National Science Foundation] and the U.S. Department of Energy's Office of Science. These resources provide CUNY researchers with extraordinary access to high-performance computing facilities.</td><td> </td><td class='diff-context'>In March 2008, the Graduate Center became the twenty-first institution state-wide – and the first CUNY campus – to join the [http://www.nysgrid.org New York State Grid (NYSGrid)]. NYSGrid was formed in 2006 when member institutions united to facilitate information and technology exchange, to develop an advanced technological infrastructure across the state, and to provide access to computational and data resources on a statewide grid basis. NYSGrid itself is a "virtual organization" within the [http://www.opensciencegrid.org Open Science Grid], a national, distributed computing grid for data-intensive research, supported by the [http://www.nsf.gov National Science Foundation] and the U.S. Department of Energy's Office of Science. These resources provide CUNY researchers with extraordinary access to high-performance computing facilities.</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td>-</td><td class='diff-deletedline'>==Getting Started==</td><td colspan="2"> </td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td>-</td><td class='diff-deletedline'>The CUNY Research Computing cluster is composed of a few servers and a number of computing nodes. To start, we show an example to submit a binary executable job to the cluster. To use the cluster, you must have an account on the Graduate Center Research Computer Cluster. If you need one, please follow the application process [http://research.gc.cuny.edu/wiki/index.php/How_to_apply_for_a_computational_cluster_account here].</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>Now suppose you have the log in to the cluster,</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>1. From a secure shell terminal, e.g., [http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe putty] under windows or a Linux / Mac terminal</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> ssh -l <your username> neptune.gc.cuny.edu</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>2. Now you are on a linux shell environment. Make a new directory for your program and create your program.</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> mkdir myfirstproject</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> cd myfirstproject</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> vim example.cpp</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>3. Finish writing your java program and save it to the cluster. Compile locally at the neptune server by</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> g++ example.cpp -o example</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>4. NOW, you are NOT suppose to run your executable ''example'' on the cluster, but rather, you need to submit it as a script job to the cluster. To do this, you need a job execution script.</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> vim example.sh</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>and put the following lines in the shell script</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -S /bin/bash</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -q quad.q</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> </td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -m abe</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -M youremail@yourdomain.com</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> </td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -o $HOME/myfirstproject/example_result</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -e $HOME/myfirstproject/example_exception</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> $HOME/myfirstproject/example</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>The script tells the cluster a few things.</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -S /bin/bash</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -q quad.q</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>The above lines tell the system the script should be executed using shell commands in /bin/bash, you could change this to other shell (e.g., C shell) by specifying the corresponding shell path. It also tells the job should be run on the quad.q queue of the cluster, which is a set of computing nodes, each composed of Intel Q9550 quad core CPU and 8GB of memory. The queue name can also be x86_64.q, which are of specification Intel Core 2 Duo 8400 CPU and 4GB of memory.</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -m abe</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -M youremail@yourdomain.com</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>The two lines above tell you when your job begins to execute, ends, or aborts (due to some exception), the system should send you an email to youremail@yourdomain.com (change this to your own email).</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -o $HOME/myfirstproject/example_result</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> #$ -e $HOME/myfirstproject/example_exception</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>The above lines tell the system to redirect the system standard stream to file $HOME/myfirstproject/example_result, and system standard error stream to $HOME/myfirstproject/example_exception, where $HOME is your home directory on the cluster. You program is supposed to output the result in standard output stream (e.g., std::cout in C++, and system.out.print in java). If there was any exception, e.g., segmentation fault, you should check the standard error file ''example_exception''.</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>The last line, which does not begin with a "#" sign, is the actual script to be executed by the cluster.</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> $HOME/myfirstproject/example</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>5. After finish the script file, you could now submit it to the cluster by</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'> qsub example.sh</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>to the cluster. Watch out your email when the job is done (or aborted) and check your result at "example_result" or reason for aborting at "example_exception".</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>Have fun.</td><td colspan="2"> </td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>==Research Computing Fellows==</td><td> </td><td class='diff-context'>==Research Computing Fellows==</td></tr> <tr><td colspan="2" align="left">Line 103:</td> <td colspan="2" align="left">Line 49:</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>If you don't find something that should be here, or if you find something that shouldn't be here, contact the [http://mailhide.recaptcha.net/d?k=01q5B1PJPOMuUdcyvLgRqGnw==&c=DtNMMNDZw6457tsk2ivk40OcITofyVEd3y4J7FoLomI= Assistant Director for Research Computing].</td><td> </td><td class='diff-context'>If you don't find something that should be here, or if you find something that shouldn't be here, contact the [http://mailhide.recaptcha.net/d?k=01q5B1PJPOMuUdcyvLgRqGnw==&c=DtNMMNDZw6457tsk2ivk40OcITofyVEd3y4J7FoLomI= Assistant Director for Research Computing].</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>= Getting Help =</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>"Who you gonna call? Ghostbusters!"</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>=== If you are new to the computational cluster ===</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>If you're new to the computational cluster, see </td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>[[Getting Started]] and the sections on</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>[http://research.gc.cuny.edu/wiki/index.php/Research_Computing:Services#Computational_Cluster the computational cluster] and the section on [http://research.gc.cuny.edu/wiki/index.php/Research_Computing:Services#SUN_Grid_Engine_Documentation SUN Grid Engine Documentation] for information on submitting and monitoring jobs using the SGE job manager.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>=== If you are new to Linux ===</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>We can suggest a few online tutorials. The [http://www.fsid.cvut.cz/cz/U201/LINUX.HTML Quick Linux Tutorial] covers basic commands for interacting with a Linux System. </td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>A more comprehensive series of tutorials is available from Dr. Bob's [http://lowfatlinux.com/ Lowfat Linux] site.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>=== If something isn't working ===</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>If an online trouble isn't covered online, write to the [mailto:helpdesk@gc.cuny.edu Help Desk]. Please include the following data:</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>* Your username</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>* the command you were running</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>* the job id (if applicable)</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>* error messages encountered</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>You can cut and paste error messages into your email message.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>'''IMPORTANT:''' Please inform the help desk that your request concerns research computing.</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>=Citations=</td><td> </td><td class='diff-context'>=Citations=</td></tr> </table>

WikiSysop foobar Research Computing:Services 2009-11-16T03:55:30Z 2009-11-16T03:55:30 2009-11-16T03:55:30Z

/* If something isn't working */ <table border='0' width='98%' cellpadding='0' cellspacing='4' class='diff'> <tr> <td colspan='2' width='50%' align='center' class='diff-otitle'>Revision as of 03:55, 16 Nov 2009</td> <td colspan='2' width='50%' align='center' class='diff-ntitle'>Current revision</td> </tr> <tr><td colspan="2" align="left">Line 19:</td> <td colspan="2" align="left">Line 19:</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>=== If something isn't working ===</td><td> </td><td class='diff-context'>=== If something isn't working ===</td></tr> <tr><td>-</td><td class='diff-deletedline'>If an online trouble isn't covered online, write to the [mailto:helpdesk@gc.cuny.edu Help Desk]. [[How to report a trouble]]</td><td>+</td><td class='diff-addedline'>If an online trouble isn't covered online, write to the [mailto:helpdesk@gc.cuny.edu Help Desk]. Please include the following data:</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>* Your username</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>* the command you were running</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>* the job id (if applicable)</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>* error messages encountered</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>You can cut and paste error messages into your email message.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> </td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>'''IMPORTANT:''' Please inform the help desk that your request concerns research computing.</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>= Projects =</td><td> </td><td class='diff-context'>= Projects =</td></tr> </table>

WikiSysop foobar How to report a trouble 2009-11-16T03:52:09Z 2009-11-16T03:52:09 2009-11-16T03:52:09Z

WikiSysop foobar Research Computing:Services 2009-11-16T03:51:43Z 2009-11-16T03:51:43 2009-11-16T03:51:43Z

WikiSysop foobar Research Computing 2009-11-16T03:29:58Z 2009-11-16T03:29:58 2009-11-16T03:29:58Z

/* Scheduled Maintenance */ <table border='0' width='98%' cellpadding='0' cellspacing='4' class='diff'> <tr> <td colspan='2' width='50%' align='center' class='diff-otitle'>Revision as of 03:29, 16 Nov 2009</td> <td colspan='2' width='50%' align='center' class='diff-ntitle'>Current revision</td> </tr> <tr><td colspan="2" align="left">Line 111:</td> <td colspan="2" align="left">Line 111:</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>The following publications acknowledge the Graduate Center's parallel computational facilities.</td><td> </td><td class='diff-context'>The following publications acknowledge the Graduate Center's parallel computational facilities.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>'''What the 2007 TAC Market Design Game tells us about effective auction mechanisms''' Jinzhong Niu, Kai Cai, Simon Parsons, Peter McBurney and Enrico H. Gerding. [http://www.springerlink.com/content/102852/?p=6558021367b646b994b7528c8c91be9c&pi=0 Autonomous Agents and Multi-Agent Systems] 10.1007/s10458-009-9110-0, 2009 [http://www.springerlink.com/content/r7vg71wj0q000539 Abstract]</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td colspan="2" align="left">Line 198:</td> <td colspan="2" align="left">Line 201:</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>= Scheduled Maintenance =</td><td> </td><td class='diff-context'>= Scheduled Maintenance =</td></tr> <tr><td>-</td><td class='diff-deletedline'>Memorial Day software upgrades May 25, 2009</td><td>+</td><td class='diff-addedline'>Software and hardware upgrades Nov 24--Nov 25, 2009</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td>-</td><td class='diff-deletedline'>Note: the cur server was affected by a routine up2date. New SELinux rules interfered</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>with the MySQL database, causing web sites to report database connection errors.</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>This problem has been addressed.</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>Friday February 27, 2009 from 5PM to 9PM</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'></td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>* upgrade of the cluster network from 100mb/sec to 1Gb/sec.</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>* modification of the SGE port configuration</td><td colspan="2"> </td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>ALL JOBS WILL BE TERMINATED. We will do this as gracefully</td><td> </td><td class='diff-context'>ALL JOBS WILL BE TERMINATED. We will do this as gracefully</td></tr> <tr><td>-</td><td class='diff-deletedline'>as we can, by issuing a qdel -u all command at 5PM Friday</td><td>+</td><td class='diff-addedline'>as we can, by issuing a qdel -u all command at 4PM Tuesday</td></tr> <tr><td>-</td><td class='diff-deletedline'>February 27, 2009. Graceful shutdowns will allow checkpointed jobs,</td><td>+</td><td class='diff-addedline'>Novenmber 24, 2009. Graceful shutdowns will allow checkpointed jobs,</td></tr> <tr><td> </td><td class='diff-context'>such as G03 geometry optimizations, to be resubmitted after the upgrade.</td><td> </td><td class='diff-context'>such as G03 geometry optimizations, to be resubmitted after the upgrade.</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td>-</td><td class='diff-deletedline'>From now until next Friday, we recommend that users not run any</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>new non-checkpointed jobs (such as frequency jobs) that are likely to</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>terminate after Friday Feb 27th at 5PM--start jobs running longer than</td><td colspan="2"> </td></tr> <tr><td>-</td><td class='diff-deletedline'>one week after the upgrade.</td><td colspan="2"> </td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>The following servers will be inaccessible during the upgrade:</td><td> </td><td class='diff-context'>The following servers will be inaccessible during the upgrade:</td></tr> <tr><td colspan="2" align="left">Line 226:</td> <td colspan="2" align="left">Line 216:</td></tr> <tr><td> </td><td class='diff-context'>#monad.gc.cuny.edu</td><td> </td><td class='diff-context'>#monad.gc.cuny.edu</td></tr> <tr><td> </td><td class='diff-context'>#grid.gc.cuny.edu</td><td> </td><td class='diff-context'>#grid.gc.cuny.edu</td></tr> <tr><td>-</td><td class='diff-deletedline'> </td><td>+</td><td class='diff-addedline'>#cur.gc.cuny.edu</td></tr> <tr><td>-</td><td class='diff-deletedline'>We will send period reminders about the network upgrade. Other</td><td>+</td><td class='diff-addedline'></td></tr> <tr><td>-</td><td class='diff-deletedline'>upgrades are planned in the future.</td><td>+</td><td class='diff-addedline'></td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>=[[Inter-University Consortium for Political and Social Research]]=</td><td> </td><td class='diff-context'>=[[Inter-University Consortium for Political and Social Research]]=</td></tr> </table>

WikiSysop foobar Account policy 2009-11-16T03:01:04Z 2009-11-16T03:01:04 2009-11-16T03:01:04Z

/* Citing the facilities */ <table border='0' width='98%' cellpadding='0' cellspacing='4' class='diff'> <tr> <td colspan='2' width='50%' align='center' class='diff-otitle'>Revision as of 03:01, 16 Nov 2009</td> <td colspan='2' width='50%' align='center' class='diff-ntitle'>Current revision</td> </tr> <tr><td colspan="2" align="left">Line 4:</td> <td colspan="2" align="left">Line 4:</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>Cluster users must submit jobs to the cluster execution hosts. Cluster users should not run programs on the job submission hosts, monad.gc.cuny.edu or neptune.gc.edu.edu. Jobs found executing on the job submission hosts may be terminated. Cluster users are assigned to the Sun Grid Engine access control list (ACL) research or to the students ACL as appropriate. It is possible to be on both lists, but in that case care should be exercised so that jobs execute on the intended cluster.</td><td> </td><td class='diff-context'>Cluster users must submit jobs to the cluster execution hosts. Cluster users should not run programs on the job submission hosts, monad.gc.cuny.edu or neptune.gc.edu.edu. Jobs found executing on the job submission hosts may be terminated. Cluster users are assigned to the Sun Grid Engine access control list (ACL) research or to the students ACL as appropriate. It is possible to be on both lists, but in that case care should be exercised so that jobs execute on the intended cluster.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>==Citing the facilities==</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>GC Research Computing cluster and grid users are required to cite the use of these resources in announcing results and in publications. Our facilities should be cited in announcements, published research and technical reports as "the computational facility at the CUNY Graduate Center." Failure to cite the facilities may result in account termination.</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>== Research Cluster ==</td><td> </td><td class='diff-context'>== Research Cluster ==</td></tr> </table>

WikiSysop foobar Minutes of the RCC meeting of November 9, 2009 2009-11-16T02:56:23Z 2009-11-16T02:56:23 2009-11-16T02:56:23Z

/* Other business */ <table border='0' width='98%' cellpadding='0' cellspacing='4' class='diff'> <tr> <td colspan='2' width='50%' align='center' class='diff-otitle'>Revision as of 02:56, 16 Nov 2009</td> <td colspan='2' width='50%' align='center' class='diff-ntitle'>Current revision</td> </tr> <tr><td colspan="2" align="left">Line 53:</td> <td colspan="2" align="left">Line 53:</td></tr> <tr><td> </td><td class='diff-context'>* Steve Brier noted the need to be strategic with regard to the allocation of resources. Steve and Bob noted that funding at the GC for Research Computing has traditionally been fairly limited.</td><td> </td><td class='diff-context'>* Steve Brier noted the need to be strategic with regard to the allocation of resources. Steve and Bob noted that funding at the GC for Research Computing has traditionally been fairly limited.</td></tr> <tr><td> </td><td class='diff-context'>* Joe Dannenberg noted the need to assess the requests for services and resources against the production of research results and publications.</td><td> </td><td class='diff-context'>* Joe Dannenberg noted the need to assess the requests for services and resources against the production of research results and publications.</td></tr> <tr><td>-</td><td class='diff-deletedline'>* Bob Campbell sought the advice of the group regarding terms of use for Research Computing accounts that include a requirement to cite the use of these resources in announcing results and in publications. The group concurred.</td><td>+</td><td class='diff-addedline'>* Bob Campbell sought the advice of the group regarding terms of use for [http://research.gc.cuny.edu/wiki/index.php/Account_policy Research Computing accounts] that include a requirement to cite the use of these resources in announcing results and in publications. The group concurred.</td></tr> </table>

WikiSysop foobar How to apply for a computational cluster account 2009-11-16T02:55:37Z 2009-11-16T02:55:37 2009-11-16T02:55:37Z

<table border='0' width='98%' cellpadding='0' cellspacing='4' class='diff'> <tr> <td colspan='2' width='50%' align='center' class='diff-otitle'>Revision as of 02:55, 16 Nov 2009</td> <td colspan='2' width='50%' align='center' class='diff-ntitle'>Current revision</td> </tr> <tr><td colspan="2" align="left">Line 1:</td> <td colspan="2" align="left">Line 1:</td></tr> <tr><td> </td><td class='diff-context'>Research computing accounts one the computational cluster are available to CUNY doctoral faculty and their doctoral students. </td><td> </td><td class='diff-context'>Research computing accounts one the computational cluster are available to CUNY doctoral faculty and their doctoral students. </td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td>-</td><td class='diff-deletedline'>To apply for an account on the computational cluster, send an email to the [mailto:flengyel@gc.cuny.edu Assistant Director for Research Computing] with the following data: your email address, your CUNY Ph.D. program and your primary CUNY campus. Please include a brief statement of the research for which the account will be used; we use these statements to justify maintenance and development of the facilities. Requests for student accounts should include the name and email address of a faculty sponsor. Accounts are subject to [http://research.gc.cuny.edu/index.php/Account_policy research computing policies].</td><td>+</td><td class='diff-addedline'>To apply for an account on the computational cluster, send an email to the [mailto:flengyel@gc.cuny.edu Assistant Director for Research Computing] with the following data: your email address, your CUNY Ph.D. program and your primary CUNY campus. Please include a brief statement of the research for which the account will be used; we use these statements to justify maintenance and development of the facilities. Requests for student accounts should include the name and email address of a faculty sponsor. Accounts are subject to [http://research.gc.cuny.edu/wiki/index.php/Account_policy research computing policies].</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>Note: if the account application is approved, you will receive an email with your username and password and instructions on how to access the system. We will assign a username and a password that you may change using the ''passwd'' command. Please do not send us an initial password for us to assign. Our security subsystem may reject such passwords as insufficient and we prefer to minimize the transmission of passwords through insecure channels such as email. If you request a specific username, we will attempt to honor it, but depending on name collisions with other users, or whether we must adhere to campus-wide naming conventions, we might have to assign a username other than the requested one.</td><td> </td><td class='diff-context'>Note: if the account application is approved, you will receive an email with your username and password and instructions on how to access the system. We will assign a username and a password that you may change using the ''passwd'' command. Please do not send us an initial password for us to assign. Our security subsystem may reject such passwords as insufficient and we prefer to minimize the transmission of passwords through insecure channels such as email. If you request a specific username, we will attempt to honor it, but depending on name collisions with other users, or whether we must adhere to campus-wide naming conventions, we might have to assign a username other than the requested one.</td></tr> </table>

WikiSysop foobar Minutes of the RCC meeting of November 9, 2009 2009-11-16T02:27:29Z 2009-11-16T02:27:29 2009-11-16T02:27:29Z

/* Attendees */ New page<div>==Place and time== CUNY Graduate Center, room 8106, 2PM-3:30PM ==Chair== Ted Brown ==Attendees== K. Ayers, S. Brier, B. Campbell, J. Dannenberg, H. Fadaifard, Y. Feng, A. Greer, M. Marianski, M. Kress, F. Lengyel, J. Plumley, A. Rapp, M. Tsai, J. Weintrop ==Report on GC Research Computing== Florian Lengyel provided an update on the Graduate Center’s recent Research Computing activities; these include: * Procurement of a GPGPU system and additional storage capacity. * Pending procurement of an additional Infiniband switch, visualization workstation and 5450 processors. * Submission of a pending NSF grant proposal for funding for GPGPU systems and visualization workstations. * An additional RC Fellow position. Florian also urged researchers to cite the use of the GC Research Computing resources in their published work. Florian reported that one researcher has expressed an interest in taking advantage of resources available via the NYS Grid. ==Report on the CSI HPC facility== Mike Kress provided an update on the recent activities of the High Performance Computing (HPC) Center at the College of Staten Island; these include: * Major upgrades to the facilities, including electrical and HVAC systems. * The imminent availability of a new SGI system. * Acquisition of a GPU-based system. * New staff appointments and the current search for a Director for the Center. * The imminent availability of a 1 GB link to the HPC Center. Mike noted that Gaussian 09 may be a good candidate for focusing on at the GC. Florian noted that it may be appropriate to consider a division of labor between the GC and CSI with regard to supporting various software platforms. Mike concured with the notion of such a division of labor. ==Update on the CUNY Committee on Academic Technology== Steve Brier provided an update on the CUNY-wide Committee on Academic Technology. * The CAT is focused on the breath of academic technology issues; reps are selected by campus provosts. * The CAT founded the CUNY Academic Commons, which provides a forum for collaboration and communication. * Steve in particular promotes issues related to research computing and high performance computing. Florian noted his intent to migrate the RC wiki to the Academic Commons. ==Report from the Committee Chair== Ted Brown provided an update: * NSF has announced a new funding opportunity under the heading of cyber-infrastructure data nets. * A new cluster hire for CS at the GC will be a faculty member working in the area of visualization or simulation. * Ted noted the opportunity to do open source software development. ==Comments and questions of Prof. Joe Weintrop== Joe Weintrop made the following observations: * His perspective has always been that this committee is informational in nature. * He is interested in directing his students to pursue analyses with larger data sets; he noted that Baruch captures details on every NYSE trade made, so they have massive data available for analysis. * He wonders: ** Will the new GB connection to the HPC Center at CSI enable such work to be done via that resource? ** Is it the intent of the RC resources at the GC to support such activities? Bob Campbell noted that the GC has investigated resources to support massive data sets for activities in the social sciences, humanities, business, etc. but no funding source has yet been identified. ==Other business== * Steve Brier noted the need to be strategic with regard to the allocation of resources. Steve and Bob noted that funding at the GC for Research Computing has traditionally been fairly limited. * Joe Dannenberg noted the need to assess the requests for services and resources against the production of research results and publications. * Bob Campbell sought the advice of the group regarding terms of use for Research Computing accounts that include a requirement to cite the use of these resources in announcing results and in publications. The group concurred.</div>

WikiSysop foobar Research Computing Committee Meeting Minutes 2009-11-16T02:00:57Z 2009-11-16T02:00:57 2009-11-16T02:00:57Z

WikiSysop foobar Research Computing 2009-11-13T00:18:43Z 2009-11-13T00:18:43 2009-11-13T00:18:43Z

Step by step to create and submit a executable job to the cluster. <table border='0' width='98%' cellpadding='0' cellspacing='4' class='diff'> <tr> <td colspan='2' width='50%' align='center' class='diff-otitle'>Revision as of 00:18, 13 Nov 2009</td> <td colspan='2' width='50%' align='center' class='diff-ntitle'>Current revision</td> </tr> <tr><td colspan="2" align="left">Line 34:</td> <td colspan="2" align="left">Line 34:</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>==Getting Started==</td><td> </td><td class='diff-context'>==Getting Started==</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>The CUNY Research Computing cluster is composed of a few servers and a number of computing nodes. To start, we show an example to submit a binary executable job to the cluster. To use the cluster, you must have an account on the Graduate Center Research Computer Cluster. If you need one, please follow the application process [http://research.gc.cuny.edu/wiki/index.php/How_to_apply_for_a_computational_cluster_account here].</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>Now suppose you have the log in to the cluster,</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>1. From a secure shell terminal, e.g., [http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe putty] under windows or a Linux / Mac terminal</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> ssh -l <your username> neptune.gc.cuny.edu</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>2. Now you are on a linux shell environment. Make a new directory for your program and create your program.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> mkdir myfirstproject</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> cd myfirstproject</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> vim example.cpp</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>3. Finish writing your java program and save it to the cluster. Compile locally at the neptune server by</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> g++ example.cpp -o example</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>4. NOW, you are NOT suppose to run your executable ''example'' on the cluster, but rather, you need to submit it as a script job to the cluster. To do this, you need a job execution script.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> vim example.sh</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>and put the following lines in the shell script</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -S /bin/bash</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -q quad.q</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> </td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -m abe</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -M youremail@yourdomain.com</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> </td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -o $HOME/myfirstproject/example_result</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -e $HOME/myfirstproject/example_exception</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> $HOME/myfirstproject/example</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>The script tells the cluster a few things.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -S /bin/bash</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -q quad.q</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>The above lines tell the system the script should be executed using shell commands in /bin/bash, you could change this to other shell (e.g., C shell) by specifying the corresponding shell path. It also tells the job should be run on the quad.q queue of the cluster, which is a set of computing nodes, each composed of Intel Q9550 quad core CPU and 8GB of memory. The queue name can also be x86_64.q, which are of specification Intel Core 2 Duo 8400 CPU and 4GB of memory.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -m abe</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -M youremail@yourdomain.com</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>The two lines above tell you when your job begins to execute, ends, or aborts (due to some exception), the system should send you an email to youremail@yourdomain.com (change this to your own email).</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -o $HOME/myfirstproject/example_result</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> #$ -e $HOME/myfirstproject/example_exception</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>The above lines tell the system to redirect the system standard stream to file $HOME/myfirstproject/example_result, and system standard error stream to $HOME/myfirstproject/example_exception, where $HOME is your home directory on the cluster. You program is supposed to output the result in standard output stream (e.g., std::cout in C++, and system.out.print in java). If there was any exception, e.g., segmentation fault, you should check the standard error file ''example_exception''.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>The last line, which does not begin with a "#" sign, is the actual script to be executed by the cluster.</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> $HOME/myfirstproject/example</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>5. After finish the script file, you could now submit it to the cluster by</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'> qsub example.sh</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>to the cluster. Watch out your email when the job is done (or aborted) and check your result at "example_result" or reason for aborting at "example_exception".</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'></td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>Have fun.</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>==Research Computing Fellows==</td><td> </td><td class='diff-context'>==Research Computing Fellows==</td></tr> </table>

Yfeng foobar