First, create a Container Engine cluster with storage-full scopes. Isolation between workloads: Kubernetes' Quality of Service mechanism allows you to safely co-schedule batch workloads like Spark on the same nodes as latency-sensitive servers.įor this demo, we’ll be using Google Container Engine (GKE), but this should work anywhere you have installed a Kubernetes cluster.For example, you may have a streaming application running to feed a streaming Spark pipeline, or a nginx pod to serve web traffic - no need to statically partition nodes.
Increased server utilization: share nodes between Spark and cloud-native applications.A single, unified interface to your cluster: Kubernetes can manage a broad range of workloads no need to deal with YARN/HDFS for data processing and a separate container orchestrator for your other applications.So why would you run Spark on Kubernetes? You can run it on a Mesos cluster alongside other Mesos applications.You can run it on a YARN cluster, co-resident with Hadoop and HDFS.
You can run it using dedicated resources, in Standalone mode.There are many ways to run Spark outside of Kubernetes: Zeppelin allows the user to interact with the Spark cluster in a simple way, without having to deal with a command-line interpreter or a Scala compiler. As one of its backends, Zeppelin connects to Spark. Why Zeppelin?Īpache Zeppelin is a web-based notebook that enables interactive data analytics. Fortunately, with Kubernetes 1.2, you can now have a platform that runs Spark and Zeppelin, and your other applications side-by-side. With big data usage growing exponentially, many Kubernetes customers have expressed interest in running Apache Spark on their Kubernetes clusters to take advantage of the portability and flexibility of containers.
Install spark on windows using zeppelin series#
Editor's note: this is the fifth post in a series of in-depth posts on what's new in Kubernetes 1.2
0 kommentar(er)
