File Management

An overview of the file system is presented here. The following tabs provide detailed information about copying files to and from the N8HPC facility. Further related information can also be found in the Running Jobs page.
  • Copying Files to the Facility
  • Copying Files from the Facility

Push

Pushing data is most easily achieved using scp/sftp/rsync command line clients from any location where ssh access is available (which should include any location in your home institution or any of the other N8 institutions). You may find it convenient to set up passwordless login for ssh to assist with this to avoid the need for password prompts, but be mindful of the security implications of protecting your private key and password strength before doing so. Note that scp, sftp, and rsync are available on Linux, Mac, and Windows (in some cases these are third party downloads), although in some cases the command names and options may vary slightly.

scp

Wikipedia entry

To copy a directory from a local source to your home directory, e.g:

scp -r mylocaldirectory myusername@polaris.leeds.ac.uk:

This will create a directory in your home directory called mylocaldiectory

sftp

Wikipedia entry

sftp myusername@polaris.leeds.ac.uk

This will then bring up an interactive prompt as typical for other command-line ftp clients

rsync

Wikipedia entry

Rsync allows you copy only those files that have changed from a local disk to the N8 system. An example command to copy any changed files from a local directory to the N8 might be:

rsync -vuarP mylocaldirectory myusername@polaris.leeds.ac.uk:

This will create a directory in your home directory called mylocaldirectory if it doesn't already exist, or synchronise the contents if it already does. There are many options for rsync, and there are some idiosyncracies with regard to what the command means when a directory name has a trailing slash, so please read your local installation's documentation before proceeding.

sshfs

On Linux clients it is possible to use the ssh user space filesystem (aka sshfs ), if installed, to mount your home file system onto your local machine.

sshfs myusername@polaris.leeds.ac.uk:/myhomedirectory /mountpoint

Where myhomedirectory is the location of your home directory on the N8 system, and /mountpoint is where you want to see it on your local machine. You can then use your standard GUI tools or command line tools, as if the directory was mounted locally to transfer data to (or from) this. However, be aware that the data will be mounted for a long period of time, and so take care of security in relation to it.

Graphical Clients

There are a number of graphical clients which allow connection over a secure connection. An example on Windows is WinSCP . In addition tools on Linux and Mac allow the mounting of remote directories in a way that is analogous to sshfs above. The details of this vary for versions of Linux and the desktop environment, so please check your distributions information. In addition there are a number of tools that allow graphical control of rsync, e.g. grsync and unison which may be useful.

Pull

Pull is when you are logged onto the N8 system and want to pull data onto it.

sftp, scp, rsync

Many of the tools from the push section above may be used but an example for scp would be:

scp -r myinstitutionalusername@myinstitutionhpc.institution.ac.uk:somedir .

Which would copy data from an institution to a directory called somedir on the N8.

An issue with this approach is that your local institution or the owner of the above resource myinstitutionhpc may not have its firewall enabled to allow traffic from the N8 system and so this will fail. In this instance, please use the push method. This may also be the case if you wish to pull data from another location to the N8 system.

Anonymous Connections

Many public datasets can be retrieved using anonymous authentication.

ftp

The classic ftp command line tool can be used from the N8 system to pull data onto it

wget

A tool which can cope with various protocols, including data accessible via http, is wget . For example if you know there is a useful dataset that is available at http://www.datasetsrus.com/usefuldataset.zip then you can grab this via:

wget http://www.datasetsrus.com/usefuldataset.zip

Which will create the file usefuldataset.zip in the directory where the command is run.

Caveats and Cautions

When potentially sensitive data or connections are used then secure methods are preferable (e.g. scp, sftp, or rsync over ssh). It is not advisable to run an insecure ftp site at your home institution to get round this, for example.

When data is not secure, e.g. public data sets you are pulling onto the system, then using the non-secure methods may allow you to get better throughput as there is no overhead in encryption.

Ensure you select an appropriate method for your use case.

Introduction

In many instances this is simply a case of reversing the sense of the commands used in copying data to the facility. You should also check out the caveats and cautions section on the above page.

Push

scp, sftp, rsync can be used, provided the firewall rules at your home institution's resource allow you to do this e.g., when logged onto the N8 system

scp somefile.txt myinstutitionalusername@myinstututionalhpc.myinstution.ac.uk:

would copy somefile.txt to your home directory on myinstitutionalhpc

Pull

This will generally be easy and the N8 system allows access from your institution by default.

scp myusername@polaris.leeds.ac.uk:somefile.txt .

Will copy the file somefile.txt in your home directory on the N8 to the current directory on your local system.

In addition the sshfs or other mounting methods will work bidirectionally, and allow you to pull data back.

Login Form