SSH Directly to XSEDE Resources with GSISSH

Many know about the XSEDE Single Sign On Login Hub.  Many don't know that you can make your own version of this on your local systems.  To create the sign on hub, XSEDE uses the Globus Toolkit.

The steps include:

• Build Globus Toolkit with GSI enabled
• Download XSEDE Certificates

Gist with example

GridFTP Log Analysis with Logstash and Kibana / Elastic Search

As noted in my post about Lustre Stats with Graphite and Logstash we are huge fans of the ELK (Elastic Search, Logstash, Kibana) stack.  In that last example we didn't use the full ELK stack but in this example we are going to use ELK what it was meant for, log parsing and dash-boarding.

We run a GridFTP server using the Globus.org packages. GridFTP for those who don't know is a better performing way to transfer data around. If you want to setup GridFTP please use the globus.org Globus Connect Server, its much easier than setting up the certificate system, and it quickly becoming the standard auth and identity provider for national research systems.

GridFTP logs each transfer with your server.  What I want to know his where, who, and how much is going though the server.  I have been running this setup for a while now, but it could use some refining.  You can find my full logstash config as of this writing at Gist.

First the results:

Logstash has a number of filters that makes this easier.  We use the regular Grok filter to match the transfer stats lines from the GridFTP log.  You could modify this to capture the entire log in Elastic search for archive reasons.  Then the kv (key value) filter does a wonder on all of the log files key=value entries doing most of our work for us.

I have to use a few grok filters to get the IP of the remote server isolated, but once done logstash has a built in geoip filter that tags all the transfers with geolocation information which lets the maps be created.  Oh and in the dashboard those maps are interactive, so you can sort transfers just from another country by clicking on that country, or adding a direct filter for the country code, zipcode, etc.  Really handy.

Individual transfers are also mapped by what campus they are coming from if coming from a University address.  Our sub nets across the three campuses are known and published, so we use the cidr filter to add a tag for each campus, so we can look at traffic from a specific campus.  Again really handy, and would love to get contributions to see what traffic comes from internet2 / MiLR and the commodity internet.

A few warnings, the bandwidth calculation is commented out for a reason.  It works, but not all GridFTP log entries are complete to do the calculation, this makes ruby get angry and makes logstash hang.

So it was very easy to use logstash to understand the GridFTP log files, then the rest of the ELK stack let us quickly make dashboards for our file transfers. 

I was inspired to write this after thinking there must be an easier way to handle GridFTP logs after a presentation at XSEDE 14 where the classic, scripts, plus copy log files, system was employed.  The solution here is near real-time, and we found to be very durable.

File Transfer Tool Performance (Globus vs SCP)

The following is something I wrote up at the day job to satisfy my curiosity.  These measurements are not robust but encouraging.  I added some additional thoughts at the end.


On the HPC systems at ARC-TS we have two primary tools for transferring data, scp (secure copy), and Globus (GridFTP).  Other tools like rsync and sftp operate over scp and thus will have performance comparable to that tool.

So which tool performs the best?  We are going to test two cases each moving data to the XSEDE machine Gordon at SDSC. One test will be for moving a single large file, the second will be many small files.

Large file case.

For the large file we are moving a single 5GB file from Flux's scratch directory to the Gordon scratch directory.  Both filesystems can move data at GB/s rates so the network or tool will be the bottleneck.

scp / gsiscp

[brockp@flux-login2 stripe]$ gsiscp -c arcfour all-out.txt.bz2

                             gordon.sdsc.xsede.org:/oasis/scratch/brockp/temp_project

india-all-out.txt.bz2                     100% 5091MB  20.5MB/s  25.6MB/s   04:08   

Duration: 4m:08s

Globus 

Request Time            : 2015-04-26 22:41:04Z

Completion Time         : 2015-04-26 22:42:44Z

Effective MBits/sec     : 427.089

Duration: 1m:40s  2.5x faster than SCP

Many File Case

In this case the same 5GB file was split into 5093 1MB files.  Many may not know that every file has overhead, and that it is well known that moving many small files of the same size is much slower than moving one larger file of the same total size.  How much impact and can Globus help with this impact read below.

scp / gsiscp

[brockp@flux-login2 stripe]$ time gsiscp -r -c arcfour iobench

                             gordon.sdsc.xsede.org:/oasis/scratch/brockp/temp_project/

real 28m9.179s

Duration: 28m:09s

Globus

Request Time            : 2015-04-27 00:18:40Z

Completion Time         : 2015-04-27 00:25:30Z

Effective MBits/sec     : 104.423

Duration: 7m:50s  3.6x faster than SCP

Conclusion

Globus provides significant speedup both for single large files and many smaller files over scp.  The result is even more significant the smaller the files because of the overhead in scp doing one file at a time.

Additional Thoughts
Here are some additional thoughts for admins of Globus Servers.  Specifically those using Globus.org web client.  If you look at your endpoint settings in cli.globusonline.org   using endpoint-list -v there are settings for both concurrency and parallelism.  Both have settings for preferred and maximum settings. Further work would be for servers with large network connections and large parallel filesystems to increase these default values and see if better network utilization and performance could be realized.

Simple login and filetransfer using GSI to XSEDE

If you have ever used the XSEDE Single Sign on Service you have found the love of gsissh and gsiscp, TeraGrid provided the TeraGrid Client Toolkit, but no such package is provided anymore.  It turns out making your own for just login and scp is quite simple.

The steps are:

• Download the Globus Toolkit
• Build the data managment and myproxy bits
• Create a module/environment to set the default to point to the Xsede authority
• Use gsissh or gsiscp to login and move files

It should be easy at this point to also use globus-url-copy to do GridFTP transfers, but I highly recommenced you use GlobusOnline Transfer for this, but there are times when you can't such as you cannot install Globus Connect.

Appendix:

Blacklight: gsissh blacklight.psc.xsede.org
Darter: gsissh gsissh.darter.nics.xsede.org
Gordon Compute Cluster: gsissh gordon.sdsc.xsede.org
Gordon ION: gsissh gordon.sdsc.xsede.org
Keeneland: gsissh gsissh.keeneland.gatech.xsede.org
Lonestar: gsissh lonestar.tacc.xsede.org
Mason: gsissh mason.iu.xsede.org
Maverick: gsissh -p 2222 maverick.tacc.xsede.org
Nautilus: gsissh gsissh.nautilus.nics.xsede.org
Open Science Grid: gsissh submit-1.osg.xsede.org
Stampede: gsissh -p 2222 stampede.tacc.xsede.org
Trestles: gsissh trestles.sdsc.xsede.org

2013 GlobusWorld Talk

I am one of the speakers this year at GlobusWorld at Argonne April 16th-18th.

"Wishing on Wavelengths" -- The Dreams and Desires of a Centralized HPC Provider.

Space, Speed, Security:  the three S's for approaching centralized research storage.  Centralized resource providers at research institutions are faced with handling larger quantities of research data, forcing new approaches in management of petascale filesystems and backups.  These larger data requirements necessitate the creation/adoption/support of user-friendly transport methods that perform better than traditional tools, while maintaining higher levels of security (for authentication and client connections) to manage growing quantities of restricted data.