Farr Machine

HERC1 documentation

  • Overview / Access
  • Filesystem
  • Copying
  • Loading
  • Batch
  • Submitting
  • Checking
  • Deleting

0. Overview of the hardware

HERC1 (herc1.leeds.ac.uk) is an SGI UV2000 shared memory computer.

It has 256 cores and a total of 4Tb memory. This is configured as 32 x Intel E5-4650L (8 core, 2.6GHz, AVX capable) CPUs and 256 x 16Gb DIMMs (1600MHz, DDR3).

The cluster itself consists of 32 NUMA (Non-uniform memory architecture) nodes, each with 8 cores (a single CPU) and 128Gb memory. One of the 32 NUMA nodes acts as a login node and the other 31 nodes act as compute nodes and are accessed through a batch queue.

Users are not able to directly SSH into compute nodes.

The cluster runs the PBS scheduler, rather than SGE which runs on Polaris.

Disk storage is provide through a single array that provides:

  • A 4 Tb /home filesystem ($HOME directories and shared between all users)
  • A 40 Tb /scratch filesystem (again, shared between all users)

1. Accessing the cluster via SSH

HERC1 is accessible through SSH using an SSH client (such as MobaXTerm, PuTTY, Mac Terminal etc.) at herc1.leeds.ac.uk.

N8 users accessing from their home institutions will be able to SSH directly into the machine. Access from off-campus or from other institutions should be made through an N8 University VPN (or other remote access service) first.

Please contact your local N8 HPC helpdesk for more information on how to do this.

2. Navigating the Filesystem

Users have read and write access to two file storage locations:

  • Home directory ($HOME) where users are directed when they first log in. This is backed up and has an initial quota of 10GB

  • Scratch (/scratch). As on Polaris, the procedure is for users to create for themselves a directory within scratch with the same name as their username (so user mcraba would create a directory /scratch/mcraba).
There is no user quota on /scratch but data is not backed up and files will be expired (ie. removed) if they have not been used for 60 days.

This is less than the 90 day file expiry on Polaris.

Note that /scratch is intended as a short-term storage location. Important datasets and codes should be archived to a secure location off the cluster. By default, directories created on /scratch are world and group readable, so users should apply the correct permissions to this directory using chmod if the contents are sensitive or confidential.

3. Copying data to and from the cluster

Data can be transferred to and from the cluster via the usual methods, such as:

  • SSH File Transfer (SFTP) using the command line or a GUI client to herc1.leeds.ac.uk to port 22
  • rsync to synchronise files between the cluster and another location
  • scp secure copy
Version control with Git, Concurrent Versions System (cvs), Mercurial(hg) and Subversion(svn) is also supported.

4. Loading and unloading modules

On HERC1 the user environment and software applications are controlled through the modules system. This ensures that multiple versions of applications can co-exist and are only activated when required.

CommandDescriptionExample
module listshows the current set of modules loaded into the user environment
module showshows the set of modules installed and available on the system
module load <modulename>loads the <modulename> module into the user environmentmodule load R
module unload <modulename> removes the <modulename> module from the user environmentmodule unload R
module show <modulename>displays general information related to the module and PATHs, environment variables module show R

5. Using the batch scheduler

HERC1 uses the PBSPro scheduler. The syntax to request resources is a little different to the GridEngine syntax used on Polaris.

Standard PBSPro commands

There are a number of commands that can be used to test configuration of the cluster, submit jobs and confirm the status of jobs and queues.

CommandPurpose
pbsnodes -ashow the status of the nodes in the cluster
qsub <scriptname> submit a job to the scheduler
qdel <jobid>delete a job from the queue
qstatshow job, queue or server status
qstat -Qfshow detailed queue information
qstat -Qshow brief queue information
qstat -Bfshow detailed PBS server status
qstat -Bshow brief PBS server status
tracejobextract job information from log files
xpbsPBS graphical user interface
xpbsmonPBS graphical monitoring tool

PBS directives summary

Directives to the PBS scheduler are usually given as header lines in a batch submission script or as options directly on the qsub command line.

When included in a submission script, the PBS commands must come before any executable shell commands in the script.

The syntax is:

#PBS <option>

So for example, to request 10 hours of wallclock time (note that the l is the letter el and not the digit one):

#PBS -l walltime=10:00:00

Some of the most commonly used directives are given in the table below. A more complete list can be obtained through man qsub.

There is no direct equivalent to the -cwd SGE directive, so to change directory to the directory from which the job was submitted, there is a simple workaround:

cd $PBS_O_WORKDIR

The sample scripts show this in use.

DirectiveDescriptionExample
#PBS -l walltime=hh:mm:ssRequests wallclock time for a job#PBS -l walltime=5:00:00
#PBS -l mem=<amount>Requests the total amount of memory for a job across all the nodes allocated. Default unit is bytes although other units can be specified (b, kb, mb, or gb).#PBS -l mem=100gb
#PBS -l select=<n>Allocates an integer number <n> of NUMA nodes to a job#PBS -l select=2
#PBS -l place=exclRequests exclusive access to resources; used in conjunction with select to request dedicated access to resources.#PBS -l place=excl,select=2
#PBS –N <jobname>Sets the name of the jobs which is included in qstat listings and is the prefix to the job output and error files#PBS -N analysis_14
#PBS -VUses the environment (modules etc.) when the job was submitted
#PBS -m <options>Ends email on job progress. Options are (a)bort, (b)eginning, (e)nd#PBS –m bea will send email on abort, beginning and end of a job
#PBS –M <email>Defines the email address for use with #PBS -m. Mandatory. #PBS -M uiszl@leeds.ac.uk

PBS environment variables summary

Variable NameDescription
PBS_JOBIDJob ID number given to this job. This number is used by many of the job monitoring programs such as qstat.
PBS_JOBNAMEName of the job. This can be set using the -N option in the PBS script (or from the command line). The default job name is the name of the PBS script.
PBS_NODEFILEName of the file that contains a list of the HOSTS provided for the job.
PBS_ARRAY_INDEXSub-job ID numbers for jobs submitted with the -J flag. For example a job submitted with #PBS -J 1-5 will run a task array five times. The value of PBS_ARRAY_INDEX will be an integer between 1 and 5.
PBS_O_HOSTThe name of the host upon which the qsub command is running.
PBS_O_QUEUEThe name of the original queue to which the job was submitted.
PBS_O_WORKDIRThe absolute path of the current working directory of the qsubcommand. This can be used to redirect the job current working directory to the directory the job was originally submitted from: cd $PBS_O_WORKDIR (equivalent to the SGE pragma -cwd).

Sample scripts

a. Serial job on a shared node

Typically these will be small memory footprint jobs that will share a NUMA node with a number of other jobs.

For example, to submit a job requesting 8GB memory for 6 hours:

#Use the environment loaded at submission time
#PBS –V

#Request 6 hours wallclock time
#PBS -l walltime=6:00:00

#Request 8G memory
#PBS -l mem=8g

#Execute a code called myscript.x in the current directory
cd $PBS_O_WORKDIR
./mycode.x


The memory requested can be anything up to and including the total memory of the machine (4TB), requested as #PBS -l mem=4t.

b. Job requesting one or more dedicated nodes

This will typically be a (very) large memory job where the full memory allocated to one or more NUMA nodes needs to be allocated for use by a single-core or multi-core job.

Each NUMA node has 128GB memory so if a serial job (for example) requires 1TB memory this would need the memory of 8 nodes.

So to request exclusive access to 8 nodes (8 x 128GB = 1TB memory):

#PBS -l place=excl,select=8

A submission script requesting exclusive access to 8 nodes (1TB memory) for 4 hours:

#Use the environment loaded at submission time
#PBS –V

#Request 6 hours wallclock time
#PBS -l walltime=4:00:00

#Request exclusive access to 8 NUMA nodes
#PBS -l place=excl,select=8

#Execute a code called myscript.x in the current directory
cd $PBS_O_WORKDIR
./mycode.x


c. Array jobs

Array jobs are a feature of the scheduler which allow users to submit a single job script via qsub but to execute a series of sub jobs. A typical use-case is to batch process a large number of jobs, perhaps with the same code. but with different input and output files.

A job array (sometimes called a task array) is a single job with a list of sub-jobs.

The #PBS -J pragma allows users to define a range of sub-job indices.

The job array executes in a similar way to a loop (although each iteration might take place on a different node). When the job runs, the PBS environment variable $PBS_ARRAY_INDEX is used to refer to the job index.

ExampleDescription
#PBS -J 1-20sub-jobs indexed from 1 to 20 (inclusive)
#PBS -J 200-400sub-jobs indexed from 200 to 400
#PBS -J 100-200:2sub-jobs indexed from 100 to 200 in steps of 2 (ie. the steps are 100, 102, 104 etc.)

Typically, a job will use the value of $PBS_ARRAY_INDEX to define a file or a directory specific to the particular sub-task.

In this example, the script will execute the code mycode.x for ten iterations, each with a different input file and creating a different output file.

The input files take the form infile1.csv, infile2.csv etc. and the required output files are outfile1.csv, outfile2.csv etc.

Environment variable expansion can be used to substitute in values of the job index as the job runs.

#Use the environment loaded at submission time
#PBS -V

#Request 6 hours wallclock time
#PBS -l walltime=6:00:00

#Define a job array to run 10 times with script indices 1-10
#PBS -J 1-10

#Request 8G memory
#PBS -l mem=8g

#Execute a code called myscript.x in the current directory
cd $PBS_O_WORKDIR
./mycode.x < infile{$PBS_ARRAY_INDEX}.csv > outfile{$PBS_ARRAY_INDEX}.csv

6. Submitting jobs

Once a submission script has been written and saved with a suitable filename, jobs are submitted using the qsub command.

So to submit a job with a submission script named serial_shared.sh:

qsub serial_shared.sh

When the job is submitted, the job ID is reported:

[issuser@herc1 serial_shared]$ qsub serial_shared.sh
188.UV00000265-P000


The job ID is 188 in this case.

When the job completes, two files are written to the submission directory:

  • An output file containing data that would normally be written to STDOUT
  • An error file containing data that would normally be written to STDERR
By default, two separate files are created and these are named with both the submitting script and the job id. For the example above, this will produce:

An output file: serial_shared.s.o188

An error file: serial_shared.s.e188

These can be joined together so that only a single file is produced using: #PBS -j oe

The error and output files can also be renamed from the default:

  • To rename the error file: #PBS -e path\to\file
  • To rename the output file: #PBS -o path\to\file

7. Checking the status of user jobs

The status of a job can be checked with the qstat command.

Some useful options, of the form qstat <options>:

OptionDescription
-u <userid> displays all the jobs for the given <userid>
-aw <jobid> displays verbose information on the job with <jobid>
-f <jobid> displays full information on the job with the given <jobid>
-helpdisplays all possible qstat options

A worst-case-scenario estimated start time for a particular job can be obtained by: qstat -awT <jobid>

8. Deleting jobs

Deleting a running or queued job is a two stage process:

a. Get the job id

Use the qstat command to get the job ID:

[user@herc1 ~]$ qstat
Job idName   UserTime Use   S   Queue
183.UV00000265-P0  STDINuserid  00:00:00Ralloc_mem

Here, the job id listed in the first column is 183.UV00000265-P0

b. Delete the job

A running or queued job is deleted using the qdel command:

qdel <jobid>

Note that the <jobid> for use with the qdel command is the numeric part of the full job id returned by qstat, so to delete this job:

qdel 183

Login Form