Staff Software Engineer, Observability Group, Atlassian

Hariharan Iyer  • Peerlist

Big data engines like Hadoop and Spark work well when running on homogeneous clusters. This allows the underlying resource manager to optimally place tasks on the nodes and also lets users tune their jobs as per the configuration of a single machine type. While this setup has performance benefits, in many use cases it can also lead to higher total cost of ownership (TCO) because of the inability to pick and choose instances (for a given cluster) based on the workload and the availability/cost of different instance types. Heterogeneous clusters automatically mix and match instance types based on the availability and cost, leading to significant cost and, in some cases, performance benefits.

Heterogeneous big data clusters in the cloud

Big data processing systems like Hive or Mapreduce work in multiple phases where the output of one phase is dumped to disk and then subsequently read by a later phase. When running in clouds like AWS, this may lead to two problems:
1. If using on-demand or guaranteed instances, you may have to hold on to an instance if it has some intermediate output, even if it's not running computer. Thus you're paying for (expensive) compute, though you only need (cheap) storage.
2. If using spot instances, you may lose the intermediate output before its consumer has run or read it. In this case the earlier task which wrote the ouput would have to be run again, leading to slowing down the job as a whole.

Offloading the output to a shared, (relatively) cheap storage like Lustre solves both of the above problems while not adding significantly to the cost or time for running a big data job; and in most cases saving on both.

Offloading intermediate output of big data systems to Lustre

Storage autoscaling automatically adds additional storage to nodes in a Hive or Spark cluster when it is running out of space. This prevents job failure when a task dumps an unexpectedly large amount of intermediate output. Additionally, it saves cost as one may start with a small amount of additional storage on the node and expand it only when required.