ApacheCon EU 2014

Apache Hadoop clusters are generally viewed as data analysis systems, running short-to medium life analysis applications —or installations of a single large application such as Apache HBase & Apache Accumulo.

There's no reason for this to be the case: you can deploy long lived services into a Hadoop cluster, gaining access to the HDFS filesystem, availability from a fault-resilient infrastructure, shared use driven by scheduling -and the ability to integrate with other services running in the YARN cluster.

In this talk I will look at the needs of long-lived services, where YARN is today with supporting them -and where we are going next. In particular, I will explore the JIRA issue YARN-896 is the focal point for evolving YARN's support of long-lived services, addressing needs such as security, logging and service discovery -demonstrating some of this in action.