CoSINe IT Services, Division of Arts and Sciences

If you are currently a Cosine customer you already have access to our cluster. To access the server, SSH into submit.hpc.cosine.oregonstate.edu using your cosine/science username and password.

 If connecting from Off campus, please refer to I can't connect to the cluster from off campus.

Join the Cosine HPC mailing list for notifications about software updates and maintenance. Visit http://lists.science.oregonstate.edu/mailman/listinfo/cosine-hpc

User home directories on the cluster are provided by a dedicated server with 26TB of disk space. For performance reasons, this space is not backed up.

1. Create a directory in your home for the z drive mount point
2. sshfs shell.cosine.oregonstate.edu: <your mount point>
3. Please copy files from zdrive to cluster home before execution

Example

mkdir ~/zdrive

sshfs shell.cosine.oregonstate.edu: ~/zdrive

The cluster uses the environments module to provide an easy way to switch between software revisions. These modules configure environmental variables such as PATH for each piece of software. 

To get a list of available modules to load, execute:

module avail

To get a list of what modules are currently loaded, execute:

module list 

To load the Matlab 2014b module, execute

module load matlab/R2014b 

To display modules set to load during login:

module initlist

To set a module to automatically load during login:

module initadd matlab/R2014b

To remove a module from loading during login:

module initrm matlab/R2014b

Jobs should be submitted using a special sh script which tells the scheduler how to handle the job.

An example with common options can be seen below: 

 submit.sh

#!/bin/sh

# Give the job a name
#$ -N example_job

# set the shell
#$ -S /bin/sh

# set working directory on all host to
# directory where the job was started
#$ -cwd

# send all process STDOUT (fd 2) to this file
#$ -o job_output.txt

# send all process STDERR (fd 3) to this file
#$ -e job_output.err

# email information
#$ -m e
 
# Just change the email address. You will be emailed when the job has finished.
#$ -M myusername@science.oregonstate.edu

# generic parallel environment with 2 cores requested
#$ -pe orte 2

# Load a module, if needed
module load sprng/5

# Commands
./my_program

If necessary, make my_program executable

chmod +x my_program

Submit the job

qsub submit.sh

qstat is used to check the status of jobs on the cluster. By itself it will show a brief overview

qstat

To show the status of all nodes and queued processes, execute

qstat -u '*'

The state codes that dare displayed in the last column of qstat are as follows:

If a job is currently running:

qstat -j <jobId>

After a job has been executed:

qacct -j <jobId>

The command:

qdel <job id of process>

Is used to remove a job from the queue. If the job is in a dr state the -f flag must be used to force the job to stop. The job ID is supplied as an argument to qdel.

Nodes in the all.q have mixed memory sizes. To ensure that a job lands on a node with enough memory, the mem_free resource can be used.

For example. to execute on nodes with at least 60GB of RAM available:

qsub -l mem_free=60G submit.sh

Create a Submission File

Place the following code into a .sh file (i.e. submit.sh):

#!/bin/sh
# Give the job a name
$ -N JOB_NAME
# set the shell
$ -S /bin/sh
# set working directory on all host to directory where the job was started
$ -cwd
# send all ERROR messages to this file
$ -e errors.txt
# Change the email address to YOUR email, and you will be emailed when the job has finished.
$ -m e
$ -M email_address@oregonstate.edu
# Ask for 1 core, as R can only use 1 core for processing
$ -pe orte 1
# Load the R Module
module load R
# Commands to run job
R inputFile.r > outputFile.out

Submit the Job to the Cluster

Type the following commands, replacing "submit.sh" with the name of your .sh file:

qsub submit.sh

R Examples

can be found on the cluster inside the /cm/shared/examples/R folder

You can download and install any R Libraries which you might need to run on the cluster into your home directory and simply use them from there. These instructions give you the steps to accomplish this.

 ssh to the cluster

1.  Load R module
   

   module load R

2. Launch R
   

   R
   

3. Type the command to install the desired package
   

   install.packages("package_name") 

If this is the first time you have run the install.packages() command, you will be asked if you want to create a personal library.  Answer 'y'.

 Follow the prompts to pick a mirror, etc.

R will download and install the library into the newly created personal library (in your home directory).

To use this library, use the library command (like any other installed library):

library(library_name)

Create a Submission File

Place the following code into a .sh file (i.e. submit.sh):

#!/bin/sh
# Give the job a name
$ -N JOB_NAME
# set the shell
$ -S /bin/sh
# set working directory on all host to directory where the job was started
$ -cwd
# send all ERROR messages to this file
$ -e errors.txt
# Change the email address to YOUR email, and you will be emailed when the job has finished.
$ -m e
$ -M email_address@oregonstate.edu
# Use 4 cores for processing
$ -pe orte 4
# Load the Gaussian Module
module load gaussian/g16
# Commands to run job
g16 < inputFile.com > outputFile.out

Submit the Job to the Cluster

Type the following commands, replacing "submit.sh" with the name of your .sh file:

qsub submit.sh

Gaussian Examples

Examples for Gaussian can be found on the cluster inside the /cm/shared/examples/g09 folder

The node cosine004 has two NVIDIA Tesla K40m GPU Computing Accelerators. Each card provides 2880 cores and 12GB of RAM. Each card has been set to exclusive mode, meaning only one process can access the gpu at a time.

The device names of these cards are /dev/nvidia0 and /dev/nvidia1. 

A dedicated queue, gpu.q has been created for these resources. 

For interactive use, use qlogin and specify the queue:

 qlogin -q gpu.q

For batch use, use qsub in the standard fashion, but specify the queue:

 qsub -q gpu.q submit.sh

CUDA 7.5 tools are installed, but must be loaded with the modules system, typically you will include the toolkit and the gdk

module load cuda75/toolkit/7.5.18
module load cuda75/gdk/352.79

NOTE: when comping software with nvcc, there is a module conflict with gcc/5.1.0, remove this module to use the system gcc 4.8.5 

module unload gcc/5.1.0

CUDA Example

A simple CUDA example can be found in the directory: /cm/shared/examples/cuda

The Cosine cluster is protected by a firewall which blocks access from off campus. Users need to establish a VPN connection or first ssh into a campus server before connecting to the submit node.

We recommend connecting using the VPN. However, if you cannot use the OSU VPN but are able to remote into a campus PC, you can SSH from the campus PC to access the cluster.
If you prefer to work locally when submitting jobs and accessing the cluster, you can also set up SSH multi-hopping.

If you run into an issue where:

[root@cluster-submit ~]# qmon
Warning: Cannot convert string "-adobe-helvetica-medium-r-*--14-*-*-*-p-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-helvetica-bold-r-*--14-*-*-*-p-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-helvetica-medium-r-*--20-*-*-*-p-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-helvetica-medium-r-*--12-*-*-*-p-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-helvetica-medium-r-*--24-*-*-*-p-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-courier-medium-r-*--14-*-*-*-m-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-courier-bold-r-*--14-*-*-*-m-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-courier-medium-r-*--12-*-*-*-m-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-helvetica-medium-r-*--10-*-*-*-p-*-*-*" to type FontStruct
X Error of failed request: BadName (named color or font does not exist)
Major opcode of failed request: 45 (X_OpenFont)
Serial number of failed request: 525
Current serial number in output stream: 536

Appears when attempting to run qmon, then you will need to install a package on your machine.

For machines running Ubuntu:

apt-get install xfstt
service xfstt start
apt-get install xfonts-75dpi
xset +fp /usr/share/fonts/X11/75dpi
xset fp rehash

- Where cores are allocated to more than one queue the investor queues take precedence during scheduling.

GPU Resources

cosine004: gpu.q 2 NVIDIA Tesla K40m GPUs 2880 cores and 12GB of RAM.

cosine004: hendrix-gpu.q 2 NVIDIA Tesla K40c GPUs 2880 cores and 12GB of RAM.

cosine001: sun-gpu.q 1 NVIDIA Tesla K40m GPUs 2880 cores and 12GB of RAM.

zuehlsdorff001-002: 8x GeForce RTX 2080 12GB RAM (each)
zuehlsdorff004-008:  4x GeForce RTX 3080 12GB RAM (each)

If this error occurs, you need to load an a newer version of gcc that has an updated libstdc++so.6 library. In your submit script, add the following lines to switch from gcc 5.1.0 to 9.2.0

module unload gcc/5.1.0
module load gcc/9.2.0