The Changing Landscape of HPC Applications

When I first stared in HPC as a student in 2006, most applications were compiled from source and community developed.  This provided a few nice pathways for users and admins. Applications were linked against your preferred set of libraries; MPI, BLAS, Compilers. It also enabled applications to move across the different platforms; Cray, IA-64, Power, etc.

In 2012 the landscape is very much the same but also different. More ISV applications are targeting HPC use cases and are built with MPI. Most all of these applications are packaged removing the above flexibility. For the most part the vendors do a good job with compiler and math library choice. MPI on the other hand not so much.

For MPI to work the way we want there are a lot of system specifics, network support and resource manager integration being the two largest.  Every time I explore a new ISV application I wonder if it is binding correctly to the cpuset assigned by my resource manager and if it is correctly using my expensive network?

The worst is how they bury MPI start up under layers of fat. I have two examples:

app 3d -ssh -g -i inp -t4 -cnf=$PBS_NODEFILE
app job=jobname inp=1010101 inter CPUs=8 memory=6000mb

Both of these applications use MPI and when you start them up you can find an mpirun process. How would I modify any custom settings?  Queue Pair settings for IB?  What about core binding?

I would encourage ISV application providers to provide instructions and/or scripts that lets users unroll this fat.  So a user can invoke mpirun with all the usual options for that MPI library.  Even better, I like to point to LSTC, as an example of doing it right. They provide object files for their application and you link with your own build of one of their supported MPI libraries!  So I know my MPI library is ready to work with all my network specifics and I invoke the solver using mpirun.

The benefits are huge, most HPC sites that run many applications, I expect, are like my own. We have many codes we compile and invoke with mpirun, overlapping of documentation and user/admin experience is wonderful. Ambiguity of network options and resource manager integration would be reduced. Support and transitions to new products would be expedited.