Apache Spark

• Wikipedia:

Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance.

• IBM:

Apache Spark — which is also open source — is a data processing engine for big data sets. Like Hadoop, Spark splits up large tasks across different nodes. However, it tends to perform faster than Hadoop and it Uses RAM to cache and process data instead of a file system. This enables Spark to handle use cases that Hadoop cannot.

• Spark solves the limitation of the Hadoop, it also uses Map-Reduce paradigm. It provides a more expressive API than Hadoop’s MapReduce and supports a wider range of operations beyond just Map and Reduce, including transformations like map, flatMap, filter, reduceByKey, join, and more. Unlike Hadoop, Spark performs in-memory computations.
• Spark’s core abstraction is the RDD (Resilient Distributed Dataset).
  • An RDD is a distributed collection of objects that Spark can process in parallel across a cluster
  • By default, RDDs are transient. They’re recomputed whenever needed, unless you explicitly ask Spark to cache or persist them.
  • If you persist() an RDD, Spark can keep it in memory, spill it to local disk, or even replicate it — depending on the storage level you choose.
• Apache Spark itself does not come with a built-in persistent storage layer. It’s a distributed compute engine — it needs some storage system to read input data from and optionally write output to. Example of storage systems:
  • HDFS
  • Local file systems
  • Cloud storage

• Spark Core Concepts
• PySpark Reference
• Spark Data Structures
• Optimize Spark