Diablo Technologies and Hortonworks are committed to delivering high performance Apache Spark solutions. As a part of HDP's ecosystem, Spark runs seamlessly with Apache Hadoop, YARN, and other common components, allowing it to leverage existing data sources and infrastructures. Diablo’s Memory1 is the first Storage Class Memory solution that expands byte-addressable memory to the multi-terabyte range. Benefits realized include more work achieved per server and less time needed to complete jobs.
Unlocking memory limitations on application data reduces costly fetches from disk, minimizes spillover to physical storage, and maximizes the utilization of processor cores. Several Spark workloads experience a 3 to 1 or even a 6 to 1 consolidation, significantly reducing the size of the overall cluster, and thereby allowing more applications to run concurrently. Combining Hortonworks with Diablo’s Memory1 solution provides substantial benefits for Machine Learning, Spark SQL, Streaming and Graph Analysis.