Parallel Compressor Performance for Science - pigz, lbzip2, xz

UPDATE: At requet of a friend I looked into zstd and wow it's a great option. As it becomes more ubiquitous it should likely replace most compressors. Compresson similar to xz and speed approaching llz4 for modest cpu increase.

Origonal Post

As data volumes grow and single core performance grows slower than core count, compressing large volumes of data quickly requires the use of compressors that are capable of utilizing multiple cores for keeping up with the data volumes and hardware investments.

Luckily there are several available that are compatible compressors out there, but how do they perform and compare to classic gzip?  Also how well do they work on scientific data?  Often scientific data has a few very large files that are often binary and thousands of small files that are compressible.

The Host

All tests were done on the Great Lakes login node.  The properties of this node are:

• 36 core 36 thread Intel Xeon 6154 
• 192 GB Memory
• 1.9PB GPFS File System 
• 100Gbps HDR Network

The Data

The data set has the following properties

• 6649 files
• 276 directories
• 221 GB total size

 Range                       Number
[   0.000  B -   0.000  B ) 1
[   0.000  B -   1.000 KB ) 560
[   1.000 KB -   1.000 MB ) 4935
[   1.000 MB -  10.000 MB ) 1175
[  10.000 MB - 100.000 MB ) 116
[ 100.000 MB -   1.000 GB ) 94
[   1.000 GB -  10.000 GB ) 43
[  10.000 GB - 100.000 GB ) 1
[ 100.000 GB -   1.000 TB ) 0
[   1.000 TB -        MAX ) 0

Results

This compares runtime and final archive size as compared to serial gzip.  This was accomplished with 

tar -I pigz -cf myarchive.tar.gz

Command	Compatible	Parallel Compress	Parallel Decom.	Speed vs. Gzip	Gzip Size 153 G
gzip	gzip	No	No	1x	153G
pigz	gzip	Yes	No	32x	153G
lbzip2	bzip2	Yes	Yes	23x	151G
mpibzip2	bzip2	Yes	Yes	*	151G
xz -T0	xz/lzma	Yes	No	5.5x	137G
pixz	xz/lzma	Yes	Yes	5.5x	137G
zstd -T0	zst	Yes	Yes	67x	155G
lz4	lz4	No	No	42.2x	171G

Notes

• pigz can only compress in parallel with very minimal speedup on decompression

• xz requires -T0 option to use all cores in the system or will default to 1

• xz cannot decompress files in parallel but pixz can

• lbzip2 and mpibzip2  can only decompress in parallel if the archive was compressed with a parallel aware compressor  

  lz4 is not parallel aware but is by far the fastest compressor of all, but with the least space savings

  zstd requires -T0 option to use all cores or will default to 1

 Conclusion

Overall using the drop in replacements for gzip and bzip2 are obvious improvements on modern multi-core systems.   While xz and lz4 are available on almost all modern systems they are still less portable than gzip and bz2 based compressors. 

lz4 is very interesting as it's so fast it uses almost no CPU.  If one was collecting data on a lower powered device using lz4 appears to be 'compression for free'.  While not as effective as the other compressors there is almost no performance impact during tar/untar when using lz4. 

One would hope over time the stock installs of gzip and bzip2 are replaced by the parallel versions. Xz is very stable but struggles to utilize very high core counts of modern systems, but still returns the best compression ratio.

Comparing the Cost of Public Cloud to On Prem for HPC

I was recently working on long term planning of a modestly large HPC resource (20,000+ cores).  The question proposed was why are we not doing this in the cloud?

Personally I love the cloud for a lot of use cases,  I would love to not worry about hardware, have the ability to burst to any scale, but after I did the work with one major cloud provider the economics were just not there.  Will I think they get there?  Probably, but not for at least 5-10 years for our shop without some discounting off list.  Below I'm laying out my formula that for another shop might change the calculation:

1. Data Center Reliability
   Cloud data centers aim to provide enterprise availability, probably Tier 3 or better.  In academic HPC that draws the most MW from the data center infrastructure we don't value this much, but it is expensive to provide that level of availability.
2. Offerings designed for web content, enterprise, and analytics
   Cloud offerings are almost all based on enterprise needs or web app delivery.  HPC does not map to these work flows.  Adding a few extra ms of network overhead is a small cost for human interaction on a website, but is awful in HPC MPI offerings.  Yes there are HPC specific providers out there, but most are not big enough to handle the scale we are looking at, and you sacrifice most of the flexibility of cloud.
   NOTE: As analytics becomes more important for academics we should pay very close attention here, and this might be the first option to large scale utilization of public cloud, as enterprise is ahead in this area currently. 
3. Scaling
   Cloud has way more scale in total cores than any HPC system out there, but as the adage goes, "there is no cloud only other people's computers" and as many in the community have pointed out, if you have a decent sized consistent need, even reserved and pre-paid instances rack up costs quickly compared to building your own if your north of 500KW of constant need of HPC.  Someone is paying for all that unused capacity to scale.  In this case the massive scaling works against you in your marginal cost for additional long term need. 
4. Staffing
   Refer to #2 because cloud never wants to let anyone down, they are staffing at very high levels to keep all services running all the time to meet enterprise needs.  This is great if you have a database that is key to your organization, it is very cheap compared to staffing in house for that one database, but for HPC again we don't value that as academics compared to the cost of doing that.

In general because there is no HPC specific cloud provider with scale providing a service that aims to provide "good enough" availability, if you have significant need public cloud economics won't work right now.   HPC is capital heavy and staffing light compared to enterprise.  Public cloud uses expensive capital for high availability (even if the lowest cost way to get it) that isn't valued by this community.

Now if I was an enterprise IT person in the hardware / data center line of work.  I would be worried and retool my skills for deploying and monitoring HA architecture across public clouds. The small and medium enterprise will be all cloud, it's just lower cost with greater flexibility.  Once your sunk cost of a data center goes away small scale operators cannot compete the the investment being made in cloud.  Your services should start shifting to running on cloud.

Lustre Stats with Graphite and Logstash

A while back Matthew Britt who handles most of our HPC Scheduler and Resource Manager services got fed up with the problem of debugging the distributed nature of jobs. We had logs on 1000+ nodes, server, scheduler, and accounting manager. To solve this he build a solution around the ELK stack, Elasticsearch, Logstash, and Kibana. His solution was one of the most useful user support and job debugging tools ever.  You can see Matt's talk form MoabCon on YouTube.

Once Matt showed me how easy this is, I instantly got an idea, Lustre, it's another distributed problem. In this case I didn't care about logs I cared about time series data, and I had two goals to solve.

1. What is our filesystem performance over time, in both bandwidth and open/close opps.
2. Find the users who open 999999 files/s in a single code.

To handle time series data rather than Elasticsearch I used Graphite a tool that has been around awhile. Think of it as RRD, but with a data collector for creating databases on the fly over the network, and with great performance.

All the config files used at the time of writing this are available on Github.

First some pictures:

So how did I easily get these data? Enter Logstash and the exec {} input. Because lustre stores all its summary stats in directories like /proc/fs/lustre/[mdt|obdfilter]/stats and /proc/fs/lustre/[mdt|obdfilter]//exports//stats I had to make my own handler.  Why couldn't I just cat the stats files? Logstash being designed for log files, sees each line as an event and parses them each on their own.  In our case I want the entire file.  The solution to this was to make a simple python script that turns the stats into json objects.

json-stats.wrapper.py

./json-stats-wrapper.py /proc/fs/lustre/mdt/scratch-MDT0000/md_stats  | python -mjson.tool
{
    "close": "241694077",
    "crossdir_rename": "300771",
    "getattr": "439797690",
    "getxattr": "3393359",
    "link": "117530",
    "mkdir": "1332774",
    "mknod": "1209",
    "open": "789470206",
    "rename": "522526",
    "rmdir": "1289414",
    "samedir_rename": "221755",
    "setattr": "12991707",
    "setxattr": "118798",
    "snapshot_time": "1414810134.237384",
    "source": "/proc/fs/lustre/mdt/scratch-MDT0000/md_stats",
    "statfs": "799026",
    "sync": "43951",
    "unlink": "25767242"
}

Logstash if told that the input data is json, will now treat this as an event. In general the logstash-lustre.conf and logstash-lustre-mds.conf configs parse each event, including the path to the stats and grabs each counter and builds a metric form it.

One wants to be smart about your groupings. Lucky for us the Lustre devs did things in a very logical way, and it almost falls in our laps.  You will use these groups/wildcards with graphite to quickly make lots of plots for the same metric over all OST's, MDT's, or clients, etc.

lustre....
Eg: lustre.scratch.MDT.0000.open
Eg:lustre.scratch.OST.*.10-255-1-100.read_bytes

I don't calculate any rates in logstash.  I chose to store the raw counter value from the stats files.  Graphite has built in lots of nice functions that let you calculate your rates like nonNegativeDerivative() which deals with counters that roll over such as reboot.  So keep your data quite raw.

Be careful with the number of metrics and your Graphite storage schema.  We keep our per OST/MDT data for a year (10s:7d,10m:30d,60m:180d,6h:1y) and per client stats for 30 days (2m:30d).  In our case we have over 1000 clients, 30 OSTs 1 MDT and we store 16 metrics/MDT, 16 metrics/MDT/client, 18 metrics/OST, 4 metrics/OST/client.  In call we store 53,300 metrics just for lustre. This is about 14GB of data right now, and because of the way Graphite works will not grow unless we add more metrics, or add more clients.

The more likely problem is you send metrics to fast for graphite, and the number of IO's your graphite server disk can provide won't be up to task. In our case we update OST/MDT summary stats every 10 seconds, and client OST/MDT stats every 2 minutes.  This is being handled with single 7,200 RPM SATA drive with some tweaking.

Future work:

• Use logstash to alert us for slow_attr and LBUG events.
• Use the DDN SFA SNMP MIB to get raw SFA counters into Graphite
• Alert if an OST is set deactivate or in recovery/out of recovery.

Blue Waters for GLCPC Members

I recently was asked to note that applications for the Blue Waters machine for Great Lakes Consortium for Petascale Computation (GLCPC) are open and due October 31st (Halloween Cycles!).

This is great option to get Blue Waters cycles for smaller users.  This is kinda tongue in cheek as "small" is 5 million core hours.  Given that normal Blue Waters allocations are much larger this is a small large scale user.

See the notice below:

I am writing to you as President of the Great Lakes Consortium for Petascale Computation to bring your attention to a call for proposals for GLCPC allocated computer hours on the new Blue Waters Machine.  As a member of the GLCPC, faculty at your institution are eligible to apply for an allocation of time reserved by NCSA for member institutions.  As XSEDE Campus Champions, I believe you may know faculty that are interested in this opportunity.

The call for proposals can be found at http://www.greatlakesconsortium.org/bluewaters.html.

I would appreciate it if you could forward this information to the faculty on your campus that might be interested in submitting proposals for this unique opportunity.

 If you are a University of Michigan reader and want more information contact me at CAEN Advanced Computing.

Communicating with Users, What are you selling?

HPC admins often must provide support for their systems. Not all admins like to do this, but most times the generic help desk at our institutions do not have the Linux experience or the HPC specific knowledge to be of much use. Thus HPC operators and admins find that they are often double duty, operating and supporting their systems as the only tier of usable support.

Understand what your goal is as an HPC provider. Is it to just run computers and make sure they work well enough to not get laid off?  Andrew Caird at U of M (and my director), put it best.

      "We don't like computers, we like computing. If we could do that without computers we would". 

We sell the end result. This requires inputs; hardware infrastructure, software infrastructure, code, input, and interpretation. Focus on the getting to the end result of an HPC workflow. We should focus our interactions with users, PI, administration towards getting to these end results.  Never only be a hardware/Linux/computer geek.  We should all hate computers, love the computing.

Looking For Linux Kernel Hackers

Jeff Squyres at the Cisco Performance Blog has a job for Linux Kernel Hackers in the High Performance Group.  Checkout his post for details.

Job Opening at The University of Michigan CAEN HPC Group

The CAEN HPC unit at the University of Michigan in Ann Arbor has a posting for an HPC software application professional.

Find the posting and apply at: Um Jobs posting 68757

In a less formal way.  You would be working with me (Brock Palen) to provide the user facing software environment on the campus resource.

I highly encourage anyone with software building experience to apply.  You do not need to be a rock star programmer. Just be familiar with HPC programing and build systems.

Keep File Counts Small

Recently I needed to move users from a Lustre parallel filesystem to a new Lustre filesystem. This task reminded me of a common problem.

Don't create many small files.

A common mistake made that can kill application performance is writing many small files. Most HPC systems use some sort of networked file system. NFS is probably the most common but more and more use Lustre. Let us quickly think about happens in the NFS case.

In NFS to create a file the client node must talk over the network to the NFS server ask for the file to be created, wait for a reply back and then write something to it. When the file is then closed, the file server will almost always wait for the data to hit disk before telling the client it has finished writing.

This process is slow, very slow, actually compared to using a local hard drive significantly slower.  It is not uncommon to find users applications spending more time in open() close() than in write() or read().

On the other hand if a user leaves files open and keep appending to them, the open() close() overhead is eliminated. This does introduce the risk of data loss because the file server may not have committed data to disk without close() or sync()/flush(), but if the application crashes we need to restart anyway. Obviously checkpoint files need to be close()'d to be useful.

Now let us look at Lustre.  Lustre meta-data (the existence of the file) lives on its' own server. There is only one for an entire file system. That huge 10's PB filesystem?  Only 1 meta-data server.  This can be a bottle neck.  To open a file, first the client talks to this MDS (meta-data server) which tells the client which OSS (storage server) to write data to.  Lustre will have many OSS's.  If the client keeps creating new files or opening and closing the same file, it keeps making that trip back to that single MDS.  If the client creates one file, doesn't close it, and keeps writing to it, the client never speaks to the MDS again!  Just to the, many, OSS nodes.

Obviously the client can avoid making the extra network trip over to the MDS and back multiple times, but it also avoids this single server bottle neck.

In the future we will look at performance of manipulating many small files vs fewer larger files.

The Changing Landscape of HPC Applications

When I first stared in HPC as a student in 2006, most applications were compiled from source and community developed.  This provided a few nice pathways for users and admins. Applications were linked against your preferred set of libraries; MPI, BLAS, Compilers. It also enabled applications to move across the different platforms; Cray, IA-64, Power, etc.

In 2012 the landscape is very much the same but also different. More ISV applications are targeting HPC use cases and are built with MPI. Most all of these applications are packaged removing the above flexibility. For the most part the vendors do a good job with compiler and math library choice. MPI on the other hand not so much.

For MPI to work the way we want there are a lot of system specifics, network support and resource manager integration being the two largest.  Every time I explore a new ISV application I wonder if it is binding correctly to the cpuset assigned by my resource manager and if it is correctly using my expensive network?

The worst is how they bury MPI start up under layers of fat. I have two examples:

app 3d -ssh -g -i inp -t4 -cnf=$PBS_NODEFILE
app job=jobname inp=1010101 inter CPUs=8 memory=6000mb

Both of these applications use MPI and when you start them up you can find an mpirun process. How would I modify any custom settings?  Queue Pair settings for IB?  What about core binding?

I would encourage ISV application providers to provide instructions and/or scripts that lets users unroll this fat.  So a user can invoke mpirun with all the usual options for that MPI library.  Even better, I like to point to LSTC, as an example of doing it right. They provide object files for their application and you link with your own build of one of their supported MPI libraries!  So I know my MPI library is ready to work with all my network specifics and I invoke the solver using mpirun.

The benefits are huge, most HPC sites that run many applications, I expect, are like my own. We have many codes we compile and invoke with mpirun, overlapping of documentation and user/admin experience is wonderful. Ambiguity of network options and resource manager integration would be reduced. Support and transitions to new products would be expedited.

A2 Data Dive Talk

On February 11th I gave a talk at the A2 Data Dive on Visit and mentions to GlobusOnline. The video is below:

Doing More With the Same Budget

Andrew Jones has an article up at HPCWire on my series of posts on HPC funding models.  I wish to move away from condo funding, that is researchers own the underlying hardware, to a service model where users effectively rent cores. This is exactly what we did at Michigan.

Note my comments on this blog are mine and do not reflect that of my major employer.

I don't actually see much disagreement between Andrew and myself. The free model, overheads in Andrews terms, is a wonderful example of The Tragedy of the Commons. Queue times get long and what is to stop anyone from running anything?  Under this wild west approach, which I don't think Andrew is advocating, users end up paying in time.  As with anything as you hold supply steady and increase demand, wait times go up.

I live and work in a world where nothing is free.  Nodes, storage, admins, consulting, power, facilities etc. All take real resources and their use needs to be moderated.  Many educational institutions live in the world where the support from central is normally admins, maybe power, and maybe some software tokens.

By default in this world the researcher must bring the hardware, in the form of the funds to purchase nodes.  Under most cases this hardware can only be used by that group.  I currently run user support for a cluster of 5600+ cores spread across 51 groups.  The utilization is 50-70% but the matter is because these groups cannot, and are not allowed to share, inside a given group is very different. Some are 100% and queued, some are idle.

If the assumption is that we will never be just given the gear to run, and that funding agencies are only going to spend $X on computational research a year what value do you want out of it?  I argue that just reorganizing how the capital funding is spent more value is realized, as Andrew says "Science and Business output".

I agree 100% with Andrew that users do not always like paying for high speed networking, consults etc and might not even realize until after the funding arrives that that is what they need.  I also agree that some of the most interesting stuff is the "just try this" jobs.  I think these are problems we can solve other ways out of the savings of driving utilization higher.  I don't expect that all resources should be billed by the unit.  Some resources will most efficiently be provided as a public good.

Under the system I propose I would expect to extract close to maximum value from the capital resources, while maintaining great service.  Thus far these resources had been the hardware and the facilities they consume. In my most recent post I point out the benefit for of a HAAS model to users:

For the user they gain flexibility in utilization.  Groups with small budgets can now utilize large chunks of HPC resources for short periods, opening HPC to an entire new class of user.  To illustrate the Michigan Flux Project, a group with a budget of $1000 could not even buy one node in a condo but can purchase 89 cores compute for 1 month, or 1 core for 89 months.  The options here are the biggest to be realized by moving to a HAAS model.

This flexibility of utilization of the hardware equates to flexibility in the most important resource which is the researcher behind the job.  I am not saying the user behind the desk does nothing while they are jailed to smaller core counts in their condo, that they must maintain for 5 years, but I do think they can gain huge outcomes from being able to use a large number of resources in a short period for the same cost or lower than that of a condo.

Lastly the biggest winner is HAAS being approachable to those with the smallest budgets. The bottom billion researchers.  If you don't get a bucket of money to provide HPC to everyone who comes knocking, what do you tell that experimentalist that wants to run one model to align their input?  That they have to pony up for all the cores they want to use for 5 years?  Under HAAS this large base of small total users can bring a large amount of resources to bear in a short period.  This is real science and business value created from the same set of resources.

Hardware will always cost $X for Y cores, and HPC admins will ran those Y cores for 5 years.  I don't think it is good to drag the users with it. $X is a constant, there is no reason users should be forced into Y cores for 5 years. They should be able to vary Y and the number of years (months or days) up and down until the total area under the curve is $X.

Reorganize how your resources are provided and for the same capital you will see more capability.

Why HPC Services are Smarter than Condos

In my continuing tirade against HPC condos I will now propose what I think the solution is: HPC As A Service or HAAS.

HAAS provides a number of advantages over condos, the main being flexibility in construction and utilization.

For the HPC operators leasing cores by the hour or month etc. provides the administering organization ownership over hardware. If users don't own the hardware this means operators can swap out underlying hardware as needs demand. Example would be wanting to swap out older hardware for newer more powerful hardware to free data center space for expatiation, cooling needs, etc.

With a condo this meant buyouts and negotiations that were long and drawn out, and had the overhead of negotiating with every hardware owner. With HAAS the service is maintained and can be moved from older equipment to new equipment without the involvement of users as long as the service sold before and after the hardware swap is the same or improved.

For funding agencies this HAAS model provides for better utilization of hardware.  The cost passed to the user, and thus the funding source, should be less than the condo due to recovered capital deprecation. This would be in the form of over subscription of resources, so their average utilization as a unit is higher than in the condo. Remember in the condo that groups own gear and if the group is not currently using their gear, no other group can.  In the HAAS model over subscription can reach 50% or higher of the available hardware driving capital utilization higher.  Thus more research for less buck.

For the user they gain flexibility in utilization.  Groups with small budgets can now utilize large chunks of HPC resources for short periods, opening HPC to an entire new class of user.  To illustrate the Michigan Flux Project, a group with a budget of $1000 could not even buy one node in a condo but can purchase 89 cores compute for 1 month, or 1 core for 89 months.  The options here are the biggest to be realized by moving to a HAAS model.

The existing groups with large funding and continuous needs also benefit from flexibility, again as they can now procure additional resources for short periods to augment their standard needs. This burst use in emergencies or other sporadic needs went unsatisfied under pure condo models.

I personally think more HPC providers should move to an HAAS model.  These models are already used in the commercial space with providers like Amazon, Penguin and IBM.  It is also heavily used in the academic space where funding does not change hands eg. XSEDE.  As soon as funding comes into play, the push is for condos because of funding requirements and this is unfortunate.

What is Out Of Core Processing

In my previous post on Swap and why to avoid it I mention out of core methods (OOC).  OOC used to be much more common than it is currently as memory sizes have increased and price per GB has fallen.  I still see traces of it in many engineering codes that have been with us a long time.  Many applications do OOC without realizing it.

OOC processing is the assumption that the working memory of the system, RAM, is less than that needed for the working set.  Thus the application is written such that data is read and written from disk to make up for the lacking space.  This is different from swap because the application does the read() and write() calls rather than the operating system doing this transparently for you.

While relying on the operating system is easy, the operating system is really making just educated guesses and tends to writes data in small blocks rather than large sequential reads and writes which are the best for performance.  I noted in my previous post that swapping hard drives maintain about 10-20MB/s, while they stream large reads and writes at 100MB+.   If you write your own OOC method to write all the data to disk,you will read later in a large block and then read the previously written large block in you will have much better performance than that provided by swapping.

A good example of this is iterating on a large array of values much larger than the RAM of the system.  The application would read a chuck of data into memory, do the calculation it could on that chuck, put it back down to disk and read the next chunk etc.  Obviously this is still much slower and much more complicated to code than fitting the entire application into RAM if available.

Thus OOC is not recommended if avoidable, but it is a better option if you know you will never have hardware with enough RAM to fit your data.  Lastly, the case of fitting all data into RAM is called in core.

Swap the Annoying Cousin of HPC

Swap, Paging, Virtual Memory, drunk relatives, whatever you call it, avoid it.  There has been a lot of confusion among users so I hope to dispel some of the myths and ideas around swap for HPC applications.

Virtual memory allows modern operating systems to present more memory to an application than is actually installed in the platform. There is a lot more going on here but for the sake of simplicity in our case what happens if my HPC job requires 4GB of ram and my node I am allocated only has 3GB free?

When the operating system needs memory, and there is not any more physical memory installed the system starts swapping. This is the process of taking some data out of memory and writing it to a swap/page file on disk.  This sounds like a great idea, I have talked to users who rely on swap space to run their application, not expecting any impact on their performance.

To start think of how a hard drive works, there is a physical spinning platter with a needle moving over it, while RAM in a system is charges stored in a capacitor. Which is faster: Flying electrons or 7200RPM record player?  When the operating system runs out of space and starts putting some of your data onto disk it tries to make a best guess about what was used last in hopes that you will not access that data again soon, in most HPC applications this is not the case. Most of our data is represented in a few large arrays that we walk up and down over and over again.

The result is the hard drive in the computer rushing back and forth, trying to write out some data from RAM to make space for the data just requested, then read that requested data back into memory to run your application on.  So what is the speed of a hard drive vs. ram?

In modern systems with 3 memory channels and two sockets 12 total cores stream provides memory bandwidth of 42,174MB/s.  Under the best of situations hard drives give 100MB/s, under the swapping case where chunks of data are both read and written at the same time this falls by 10X to about 10-20MB/s.

In these situations your application will crawl. If you expected to use your hard drive, for any type of data storage for the application, in addition to swap your performance will be even worse because of all the demands placed on the hard drive.

What about SSD drives?   For the cost/speed I would just buy extra ram, if you still use SSD use one of the PCIe cards not the ones that use the SATA bus.

Never would I ever, as my first recommendation, say 'use swap'.  I would in order say:

   * Buy more ram
   * Get an Xsede allocation on Blacklight
   * Write your application to use out of core methods with a pile of SSD
   * Use out of core methods without SSD
   * Partition your code to run in parallel on more nodes (more ram in total)
   * Build a system with ScaleMP/vSMP.
   * Fine use swap

That is the story, total memory on the system is ram+swap, the useable memory on the system is ram+0*swap=ram. Do not use swap!

Some Links

• Tutorial on PBS batch system
• Presentation/Tutorial on Globus Online (GlobusOnline.org)
• 2012 European-U.S Summer School on HPC

Some Links

• John Cook reviews The Linux Command Line.  I would encourage all new HPC users to look at acquiring these skills.
• Wat!  It is LOL worthy
• Jeff Squyres explains using MPI with STL objects
• Globus World 2012 I will be speaking on Wednesday at 3pm

Upcoming Speaking Events

I will be the University of Michigan Ann Arbor campus February 11th speaking at DataDrive about LLNL Visit. This is an Data Without Borders event is hosted by Open.Michigan.

I will be speaking at GlobusWORLD on Wednesday April 11th at 3:30pm.  I will be at the entire event if you want to chat. I will be speaking about setting up a campus wide GSI authentication service using Globus Connect Multi User.

2011 CI Days Recordings

You can find the recordings of my talks at 2011 Cyber Infrastructure Days at Michigan on the ORCI website.

• Facilitating Scientific Discovery with XSEDE (No Video)
• 3DLab and VisIt – First Stop: Exploring Data Visualization and Acquisition Services (Video)
  
• MPI & Parallel Compilers (Video)

I highly recommend that you check out all the videos from the events. I am sad that there are a few missing that I knew were recorded.  CI Days is a great event for idea swapping and general outreach between disciplines. 

Condo Clusters are Inefficent

As I noted in my post about HPC funding models at higher education the condo cluster is probably the most popular form of funding model. The condo, or a federation of many private clusters owned by faculty but managed as a single unit sharing common administration and infrastructure.  Purdue is probably the best known condo provider.

I see two large inefficiencies to condos and one lost strategic advantage:

• Capital utilization is low due to the private nature of each condo.
• Due to sunk cost of hardware purchased frivolous utilization is encouraged.
• The lost advantage is in flexibility of allocation for emergencies and competitive advantage.

We have a phenomenon we call cluster hugger. When you can attach a name and account to hardware faculty really think of the hardware as theirs and, rightfully, expect access to it in the same way they expect access to their desktop. It is there when they want it. I have experienced this when questions come in from users of our condo and they refer to their nodes as "Dr. X's cluster" our "our cluster".  Managing these expectations is difficult.

The result is almost always access to the nodes of the condo only being allowed to those who are members of it and access to those who are not is non-existent or severally limited. This leads to low utilization due to a lack of diversity in workload inside condos.  This consumes data center space network ports etc for machines that are on average 50% idle.

Because researchers pay the full cost upfront, and these nodes are not always utilized the marginal cost of running a job on them that is very low priority, or even frivolous, unwilling to be funded by any agency etc, are allowed to run.  If the hardware is there why not?

I actually think this is a good thing for condos. If marginal cost is low you might as well utilize the hardware. My proposed replacement puts the squeeze on these sort of marginal work.

Condos are slow changing and static. They also have a high start up costs, as researchers own the hardware the upfront cost of the hardware is large and one time. This pushes out small work that could benefit to limit access to HPC resources. If a group could rent just 100 cores for a week it would cost less than 1 machine and provide greater benefit.  Condos also do not allow for bursting, the idea that resources are needed quickly in an emergency situation.

In a later post I will put down my thoughts as to a solution to these problems.

Assorted Links

• Super Computer in a Church
• GlobusOnline Tutorial (Not Umich Only)
• (Umich) Stats Workshops at CSCAR
• iRODS The Technical Solution to Data Management Plans?
• Oh it is never a technical problem