Pushing data is most easily achieved using scp/sftp/rsync command line clients from any location where ssh access is available (which should include any location in your home institution or any of the other N8 institutions). You may find it convenient to set up passwordless login for ssh to assist with this to avoid the need for password prompts, but be mindful of the security implications of protecting your private key and password strength before doing so. Note that scp, sftp, and rsync are available on Linux, Mac, and Windows (in some cases these are third party downloads), although in some cases the command names and options may vary slightly.
To copy a directory from a local source to your home directory, e.g:
scp -r mylocaldirectory email@example.com:
This will create a directory in your home directory called mylocaldiectory
This will then bring up an interactive prompt as typical for other command-line ftp clients
Rsync allows you copy only those files that have changed from a local disk to the N8 system. An example command to copy any changed files from a local directory to the N8 might be:
rsync -vuarP mylocaldirectory firstname.lastname@example.org:
This will create a directory in your home directory called mylocaldirectory if it doesn't already exist, or synchronise the contents if it already does. There are many options for rsync, and there are some idiosyncracies with regard to what the command means when a directory name has a trailing slash, so please read your local installation's documentation before proceeding.
On Linux clients it is possible to use the ssh user space filesystem (aka sshfs ), if installed, to mount your home file system onto your local machine.
sshfs email@example.com:/myhomedirectory /mountpoint
Where myhomedirectory is the location of your home directory on the N8 system, and /mountpoint is where you want to see it on your local machine. You can then use your standard GUI tools or command line tools, as if the directory was mounted locally to transfer data to (or from) this. However, be aware that the data will be mounted for a long period of time, and so take care of security in relation to it.
There are a number of graphical clients which allow connection over a secure connection. An example on Windows is WinSCP . In addition tools on Linux and Mac allow the mounting of remote directories in a way that is analogous to sshfs above. The details of this vary for versions of Linux and the desktop environment, so please check your distributions information. In addition there are a number of tools that allow graphical control of rsync, e.g. grsync and unison which may be useful.
Pull is when you are logged onto the N8 system and want to pull data onto it.
Many of the tools from the push section above may be used but an example for scp would be:
scp -r firstname.lastname@example.org:somedir .
Which would copy data from an institution to a directory called somedir on the N8.
An issue with this approach is that your local institution or the owner of the above resource myinstitutionhpc may not have its firewall enabled to allow traffic from the N8 system and so this will fail. In this instance, please use the push method. This may also be the case if you wish to pull data from another location to the N8 system.
Many public datasets can be retrieved using anonymous authentication.
The classic ftp command line tool can be used from the N8 system to pull data onto it
A tool which can cope with various protocols, including data accessible via http, is wget . For example if you know there is a useful dataset that is available at http://www.datasetsrus.com/usefuldataset.zip then you can grab this via:
Which will create the file usefuldataset.zip in the directory where the command is run.
When potentially sensitive data or connections are used then secure methods are preferable (e.g. scp, sftp, or rsync over ssh). It is not advisable to run an insecure ftp site at your home institution to get round this, for example.
When data is not secure, e.g. public data sets you are pulling onto the system, then using the non-secure methods may allow you to get better throughput as there is no overhead in encryption.
Ensure you select an appropriate method for your use case.
In many instances this is simply a case of reversing the sense of the commands used in copying data to the facility. You should also check out the caveats and cautions section on the above page.
scp, sftp, rsync can be used, provided the firewall rules at your home institution's resource allow you to do this e.g., when logged onto the N8 system
scp somefile.txt email@example.com:
would copy somefile.txt to your home directory on myinstitutionalhpc
This will generally be easy and the N8 system allows access from your institution by default.
scp firstname.lastname@example.org:somefile.txt .
Will copy the file somefile.txt in your home directory on the N8 to the current directory on your local system.
In addition the sshfs or other mounting methods will work bidirectionally, and allow you to pull data back.