Archivetar - A better tar for Big Data

 Challenge: Trade-offs of Cost/Bit vs Bits/File and Performance

In the options of Tape, HDDs, SSD, and NVMe, there are significant trade offs to expected performance for small files, at a higher cost per unit capacity.   In HPC we would love to deploy Peta Bytes of NVMe but most budgets cannot support it.

Tape and AWS Glacier have low costs, great bandwidth, but long seek times before the first file appears. Thus these technologies are often targeted at Archive use cases.  It is left to the user though to organize their data in a way that does not make recalling data painfully slow.

80/20 Rule of Project Folders

In a perfect world archived project folders would include data, source code, scripts to re-create the data etc.  This leads to a common 80/20 split, where 80% of the files have 20% of the data.  The total data volume drives the budget for storing the data, but the file count, which is only 20% of the data, drives management complexity.

Current Practices, One, Huge Tar

Currently most researchers, not having better options, will tar and entire project and upload to an archive. As projects get larger this introduces issues:

• Tars are larger than max object size
• Compression is limited to a single core
• To access subsets of data the entire archive must be retrieved and expanded. This requires 2x the storage space (Tar + Expanded Tar)
• Opportunities for parallelism, are lost when transferring data at the file level
• Large files, often binary, don't compress, dominate compressor time, for little benefit
• Low utilization of CPU, Storage IO, and Networking
  

Desired Outcome, Sort and Split

Preferably it would be better if files over a given size could be excluded. These will often be data files that are big enough to realize full archive performance.  Files under this threshold could be sorted into lists, and assigned to tars of a target size.  The end result being a folder of only large files and multiple tars of small files.  Subsets of data can be recalled without needing to expand all archives.

Archivetar - A better tar for Big Data

Archivetar aims to address exactly that workflow.

Archivetar benefits include:

• Utilized mpiFileUtils to quickly walk filesystems
• Creates multiple tars simultaneously for higher performance on network filesystems
• Auto detects many parallel compressors for multi-core systems
• Saves an index of files in each tar to find subsets of data without needing to recall and expand all archives
• Archives are still stand alone tars and can be expanded without archivetar installs
  

Example Archivetar

#example data file count
[brockp@gl-login1 box-copy]$ find . | wc -l
6925

# create tars of all files smaller than 10M
# tars should be 200M before compression
# save purge list
# compress with pigz if installed
archivetar --prefix my-archive --size 10M --tar-size 200M --save-purge-list --gzip

# delete small files and empty directories
archivepurge --purge-list my-archive-2020-09-17-22-35-20.under.cache

# File count after
[brockp@gl-login1 box-copy]$ find . | wc -l
379

# recreate
unarchivetar --prefix my-archive

[brockp@gl-login1 box-copy]$ find . | wc -l
6925

The Changing Landscape of HPC Applications

When I first stared in HPC as a student in 2006, most applications were compiled from source and community developed.  This provided a few nice pathways for users and admins. Applications were linked against your preferred set of libraries; MPI, BLAS, Compilers. It also enabled applications to move across the different platforms; Cray, IA-64, Power, etc.

In 2012 the landscape is very much the same but also different. More ISV applications are targeting HPC use cases and are built with MPI. Most all of these applications are packaged removing the above flexibility. For the most part the vendors do a good job with compiler and math library choice. MPI on the other hand not so much.

For MPI to work the way we want there are a lot of system specifics, network support and resource manager integration being the two largest.  Every time I explore a new ISV application I wonder if it is binding correctly to the cpuset assigned by my resource manager and if it is correctly using my expensive network?

The worst is how they bury MPI start up under layers of fat. I have two examples:

app 3d -ssh -g -i inp -t4 -cnf=$PBS_NODEFILE
app job=jobname inp=1010101 inter CPUs=8 memory=6000mb

Both of these applications use MPI and when you start them up you can find an mpirun process. How would I modify any custom settings?  Queue Pair settings for IB?  What about core binding?

I would encourage ISV application providers to provide instructions and/or scripts that lets users unroll this fat.  So a user can invoke mpirun with all the usual options for that MPI library.  Even better, I like to point to LSTC, as an example of doing it right. They provide object files for their application and you link with your own build of one of their supported MPI libraries!  So I know my MPI library is ready to work with all my network specifics and I invoke the solver using mpirun.

The benefits are huge, most HPC sites that run many applications, I expect, are like my own. We have many codes we compile and invoke with mpirun, overlapping of documentation and user/admin experience is wonderful. Ambiguity of network options and resource manager integration would be reduced. Support and transitions to new products would be expedited.