UCX for Apache Spark
High-performance, scalable and efficient ShuffleManager plugin for Apache Spark
Apache Spark™ provides programmers with an API centered on a data structure called the Resilient Distributed Dataset (RDD), a read-only multiset of data items distributed over a cluster of machines that is maintained in a fault-tolerant way. It was developed in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflow structure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk. The Apache Spark RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory.
Apache Spark Accelerates MapReduce
The availability of RDDs facilitates the implementation of both iterative algorithms, that visit their dataset multiple times in a loop, and interactive/exploratory data analysis, i.e., the repeated database-style querying of data. The latency of such applications may be reduced by several orders of magnitude compared to a MapReduce implementation (as was common in Apache Hadoop® stacks). Among the class of iterative algorithms are the training algorithms for machine learning systems, which formed the initial impetus for developing Apache Spark™.
Accelerating Shuffle
Shuffling is the process of redistributing data across partitions (a.k.a. repartitioning) between stages of computation. Shuffle is a costly process that is avoided when possible. In Hadoop, shuffle writes intermediate files to disk. These files are pulled by the next step/stage. With Apache Spark Shuffle, RDDs are kept in-memory and allow data to be within reach, but when working with a cluster, network resources are required for fetching data blocks, thus adding overall execution time. Accelerating the network fetch for data blocks with RDMA (InfiniBand or RoCE) reduces CPU usage and overall execution time.
UCX for Apache Spark Plugin
UCX for Apache Spark is a high-performance, scalable and efficient ShuffleManager plugin for Apache Spark. It utilizes RDMA and other high-performance transports to reduce CPU cycles needed for Shuffle data transfers. It reduces memory usage by reusing memory for transfers instead of copying data multiple times down the traditional TCP stack. UCX for Apache Spark is in a code freeze for the foreseeable future. The plugin will continue to be available for download and usage.
Download Plugin