Wednesday, May 31, 2017

Opportunities for public cloud in research

It comes down to $/performance and government regulations / sponsor requirements. I covered these in others.  So what is my read of the tea leaves for cloud use in research computing.

  • Campus level / modest projects with with stock CUI / NIST 800-171 or similar regulations. 
    These require purpose built systems and heavy documentation.  FedRamp made this simpler, avoid doing all the work required for this and save yourself the time.
  • Logging/Administrative/Web/CI Systems/Disaster Recovery. 
    These systems are generally small and part of the administrative stack of just having a center.  These systems benefit the same way enterprise systems do with the flexibility of the cloud.  I personally love PAAS and docker here, yes I would like another elastic search cluster please,  no I do not want to worry about building it please.
  • High Availability Systems
    IOT / Sensor Nets ingest points.  Any system where you need higher availability than normal research.  Similar to the sensitive systems, if you have a 1MW HPC data center you don't put the entire thing on generators and have a second data center for 20KW of distributed message buses for sensor networks.  If you are not investing a lot of capital into the computer systems,  don't do it anywhere else, piggyback on clouds offering of multi site built in. 
  • High Throughput Computing / Grid ComputingNew lower cost price models via AWS Spot, Google Interruptible, and Azure Low Priority make the cost of actual cycles very close to what you can buy bulk systems for.  Every HPC center I know of is always running out of power and cooling, take these workloads that are insensitive to interruption / short run times and don't require high IO or tightly coupled parallelism and keep your scarce power for unique capability for HPC.
  •  Archive / HSM Replicas or the entire thingDepending on your use of your tape today, the cloud sites make great replicas at similar costs.  Some nitch providers like Oracle have costs that are hard to beat, with one catch.  As long as you never access your data. Cost predictability for faculty is a problem, and with cold storage costing as much as $10,000/retrieved PB in the cloud, if your HSM system is busy use the cloud only for a second copy for DR.  That is upload data (free generally), delete it (sometimes free) and never bring it back except on media error.  This should help you limit your capital spend on a second site as well as the second site to put the system in.
    If you are doing real archive, that is set it and forget it, ejected tape will forever be cheaper, but do you have a place to put it, and people to do the shuffle, there is a lot of value to maybe use the cloud for all of it.
This is my first (quick) set of thoughts, other systems like analytic systems should also be done in the cloud, they are already more mature than most research sites, and makes hosting things like notebooks, and splitting data across storage buckets for policy much more useful.

I'm sure many of you will disagree with me, feel free to tweet me at @brockpalen.

No comments:

Post a Comment