ScaleOut joined the Hortonworks Technology Partner Program and has recently achieved Hortonworks Certified status for ScaleOut hServer. ScaleOut Software is a pioneer in in-memory data grid software and the ScaleOut hServer can be installed directly on Hadoop nodes and runs in-memory. In this guest blog, William Bain, Founder and CEO, talks about certification and a use case.
Recently, ScaleOut Software announced technical certification of its ScaleOut hServer® product on Hortonworks Data Platform 2.1. With this integration Hortonworks users can run both batch and operational data analysis within the same Hadoop environment. Called “operational intelligence,” ScaleOut’s technology enables Hortonworks users to perform continuous MapReduce in real time on live, in-memory data within their Hadoop deployments and integrate it with their ongoing business intelligence. This creates significant new opportunities for extending data analytics to handle fast-changing data in diverse sectors including retail, financial services, media, manufacturing, transportation, and e-commerce. It also offers cost savings by leveraging Hadoop skill sets for both business and operational intelligence.
A great example of the value that ScaleOut hServer’s integration with HDP 2.1 creates can be found in a previous post. Sofia Parfenovich, a data scientist at Altoros Systems, guest-posted about building stock trading strategies in Hadoop and describes a customer who moved off their old system to the Hortonworks Data Platform. The goal was to build trading strategies faster and with more parameters. HDP delivered a 20% decrease in latency, increased revenue by 12%, and created an additional revenue channel. While HDP greatly improved this customer’s batch-based business intelligence, the opportunity still exists to obtain operational intelligence using the HDP platform by running MapReduce in real time. Running ScaleOut hServer on their HDP cluster could provide this capability while also consolidating their batch and operational environments.
ScaleOut hServer integrates a scalable, highly available, in-memory data store with its own integrated Hadoop MapReduce execution engine. In the above example, this means that hServer can hold batch-discovered trading strategies in memory, update each strategy with changing market data, and continuously run standard Hadoop MapReduce over that changing data to produce immediate feedback. So if this customer wanted to adjust their strategies in real-time relative to changing market conditions, they could do that. They also could run the strategies in memory and ETL the changing market data back into HDFS for batch processing.
Unlike Apache Spark and Spark Streaming, ScaleOut hServer executes Java MapReduce programs unchanged; no code changes are required to take advantage of hServer’s in-memory execution engine. Also, hServer allows updates to individual data items instead of just offering data-parallel operators. This is important in operational environments, which often require updates to individual items held in memory. Equally important for live applications, hServer provides high availability using data replication (in contrast to Spark’s lineage mechanism) so that it can immediately recover from server failures.
Combining operational and business intelligence within a single, Hadoop-based platform creates new business opportunities and lower costs. Hadoop 2 with Apache Hadoop YARN as its architectural core enables multiple, “fit-for-purpose” processing engines to plug into an Enterprise Data Lake. With ScaleOut hServer, users now can combine the MapReduce design pattern with a scalable, in-memory execution engine to offer compelling operational intelligence for live systems managing fast-changing data. Together, these technologies deliver a low-cost, flexible platform for both operational and business intelligence, and this Hortonworks/ScaleOut integration ensures that installing and running both technologies are simple and intuitive.
To view the partner page for ScaleOut Software, click here. To see a demo of ScaleOut hServer evaluating trading strategies relative to changing market data, check out this video: