what is apache spark

Spark is an open source cluster computing system based on memory computing, which aims to make data analysis faster. Spark is very small and exquisite, and was developed by a small team led by Matei from the AMP Laboratory at the University of California, Berkeley. The language used is Scala, and the code for the core part of the project only has 63 Scala files, which is very short and concise.

Spark is an open source cluster computing environment similar to Hadoop, but there are some differences between the two. These useful differences make Spark superior in certain workloads. In other words, Spark enables in-memory distributed datasets that can optimize iterative workloads in addition to being able to provide interactive queries.

Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala are tightly integrated, with Scala making it possible to manipulate distributed data sets as easily as local collection objects.

Although Spark was created to support iterative jobs on distributed data sets, it is actually complementary to Hadoop and can run in parallel on the Hadoop file system. This behavior is supported through a third-party cluster framework called Mesos. Developed by the UC Berkeley AMP Lab (Algorithms, Machines, and People Lab), Spark can be used to build large-scale, low-latency data analysis applications.

Spark Cluster Computing Architecture
Although Spark has similarities with Hadoop, it provides a new cluster computing framework with useful differences. First, Spark is designed for a specific type of workload in cluster computing, namely those that reuse working data sets (such as machine learning algorithms) between parallel operations. To optimize these types of workloads, Spark introduces the concept of in-memory cluster computing, where data sets are cached in memory to reduce access latency.

Spark also introduces an abstraction called Resilient Distributed Dataset (RDD). An RDD is a collection of read-only objects distributed across a set of nodes. These collections are resilient and can be reconstructed if part of the data set is lost. The process of reconstructing a partial dataset relies on a fault-tolerant mechanism that maintains "lineage" (i.e., information that allows partial reconstruction of the dataset based on data derivation processes). An RDD is represented as a Scala object, which can be created from a file; a parallelized slice (spread across nodes); another transformed form of the RDD; and ultimately a complete change to the persistence of the existing RDD, such as requests Cached in memory.

Applications in Spark are called drivers, and these drivers implement operations that are performed on a single node or in parallel on a set of nodes. Like Hadoop, Spark supports single-node clusters or multi-node clusters. For multi-node operation, Spark relies on the Mesos cluster manager. Mesos provides an efficient platform for resource sharing and isolation for distributed applications. This setup allows Spark and Hadoop to coexist in a shared pool of nodes.

For more technical articles related to Apache, please visit the Apache Tutorial column to learn!

The above is the detailed content of what is apache spark. For more information, please follow other related articles on the PHP Chinese website!