Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics, offering information and knowledge of the Big Data.

cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

closeClose button
March 17, 2016
prev slideNext slide

Hortonworks HDP and SAS Event Stream Processing together, using YARN

The Hortonworks Data Platform (HDP)[1] compliments core Apache Hadoop with Enterprise level governance, security, and operations capabilities. At HDP’s architectural center is YARN, providing resource management and a pluggable architecture that enables batch, interactive and real-time workloads to run effectively together, in the same cluster.

Screen Shot 2016-03-17 at 11.29.50 AM

SAS® Event Stream Processing[1] is embeddable software that analyzes streaming data while it’s still in motion. Embedding SAS Event Stream Processing into gateways, routers, edge devices or other event transmission sources has the benefit of assessing patterns of interest in a high throughput data (millions of events per second) with low latency response (millisecond and microsecond response times).

SAS’s pattern of interest detection for single and multi-event source environments makes decisions on data before its stored, to help ensure downstream processing focuses on right and relevant data – decreasing irrelevant noise. Decisions defined in patterns of interest can include pre-built data quality routines, aggregations, correlations, business rules, contextual assessment of text data, and advanced analytics.

Pre-built connectors and adapters source streaming information, the event stream patterns of interest are designed from a visual, drag-and-drop environment with options for custom adapters via documented API and with design coding options in both XML and C++. One open environment for streaming data source ingestion, correction, normalization and analysis – for full traceability and governed adaptability to new conditions.

SAS Event Stream Processing supports the integration of YARN, adding the power of live streaming analytics allowing for greater scalability within a distributed environment. By using YARN, which was designed as a generic resource negotiator platform for a distributed system like SAS Event Stream Processing, the same underlying daemons and APIs within the common ecosystem of other Hadoop applications. The SAS Event Stream Processing nodes run directly within the fully managed YARN environment thereby leveraging the power of the YARN resource manager (as illustrated in Figure 2). SAS Event Stream Processing delegates resource management to YARN while integrating through dedicated YARN clients that can supply the necessary resource requests for YARN to fulfill.

 

Image

By leveraging YARN, SAS Event Stream Processing can be deployed and managed within an existing Hortonworks HDP Hadoop cluster. YARN can now allocate a SAS Event Stream Processing Application Master within a sites existing Hadoop cluster. This means that SAS can be governed by the YARN Resource Manager, so organizations can maximize their Hadoop infrastructure investments while minimizing total operational and administrative costs.

In a Hortonworks environment, the SAS Event Stream Processing runs seamlessly with other applications running in an environment managed by the YARN cluster manager.

Using the SAS Event Stream Processing YARN client, the application dictates the number of nodes and cores to use for execution, thereby providing maximum flexibility at run time.

When problems arise, YARN stays at the center to manage any errant processes.

Together, SAS Event Stream Processing derives the benefits of data cleansing, aggregation and analysis before streaming events land in Hadoop – providing initial assessments to identify patterns of interest and anomalies warranting further investigation. Execution of event streams to the HDP using the YARN cluster manager, delegates control for resources to YARN for scheduling start up, tear down, and prioritization of jobs. As SAS Event Stream Processing runs in this environment, all of the same YARN features are available with no additional coding or management – providing a single management approach for processing event streams that is already applied to other applications running in YARN, including monitoring metadata using the Consul.

This integration is delivered through open APIs to provide an additional level of information about the health of the environment specific to SAS Event Stream Processing including data about ports, nodes, and the health of the specific nodes. This information is critical to maintaining the highly available environments required by streaming analytics applications.

Learn More

[1] For more information on Hortonworks HDP see: https://hortonworks.com/hdp/whats-new/

[2] For more information regarding SAS Event Stream Processing details, please refer to the product fact sheet: http://www.sas.com/content/dam/SAS/en_us/doc/factsheet/sas-event-stream-processing-106151.pdf

Tags:

Leave a Reply

Your email address will not be published. Required fields are marked *