With HDP for Windows, you can deploy Apache Hadoop and the HDP stack of components natively on a Windows Server cluster. The HDP for Windows download includes an MSI and remote installation scripts. With these artifacts, you can setup a multi-node Hadoop cluster in either a Workgroup or Active Directory Domain networking configuration. This enables HDP for Windows to be deployed for production use in Windows Data centers.

The best way to get started and evaluate HDP is to set up a single node cluster. We’ve written a quick start guide that walks you through all the pre-requisites and install steps needed to get going. With a single node cluster, you can experience the full functionality of the product – load data into HDFS, execute Hive, Pig and MapReduce jobs, schedule processing workflows through Oozie.

HDP enables seamless integration with the Microsoft BI tool ecosystem. You can explore data in HDFS through the Data Explorer  in Excel. You can query and analyze Hive data in Excel by using the ODBC driver to connect to Hive Server 2. You can import/export data from and to SQL Server through Apache Sqoop.

These integrations enable HDP to become an integral part of your Enterprise Data Architecture, and allow you to utilize the same tools that you are familiar with to interact with HDP.

Learn More. Please take a look at the Hortonworks Documentation to learn more about installing and using HDP 1.1 for Windows.

The post Get Started with Hadoop on Hortonworks Data Platform 1.1 for Windows appeared first on Hortonworks.

What does this mean for the Mobile Telecom industry?  For one of the top five mobile phone service providers in the world, responsible for developing and managing advanced data services for European countries with data services including mobile internet access for various devices, mobile email, instant messaging, news, weather updates and traffic reports ─ it means as mobile data services grow in revenue, so does the need to monitor that contribution easily and accurately. While that sounds obvious, the mobile telecom growth rate has expanded so rapidly, the company’s existing systems could not keep up. And once the business leaders had the data – they wouldn’t trust it. Making accurate business decisions at the right time would be essential for their success and growth.

Big Data Challenge

The customer – a Mobile Telecom giant – had an existing method for determining business performance was ad hoc and decentralized. There was no single system to extract the information in a reliable and consistent manner. “We had a mix of systems and information which needed lots of cross-checking – if indeed this was even possible. Getting access to data took a long time and, even then, the business users in marketing had no real confidence in the information they were getting.” This in turn compromised their ability to develop and manage these services.

In order to gain market share and stay competitive, the customer had to be able to:

• Leverage the data from mobile usage to get accurate information about real customer activity to provide improved levels of customer satisfaction.
• Spot upcoming trends in mobile use to drive intelligent marketing.
• Improve the information on customer usage, which drives the changes needed to their service offerings, such as the ability to offer the latest mobile phone technologies.
• Handle large volumes of data, be easily configurable by in-house business users, and provide graphical representations of the results

Hadoop Solution

The strategy included harnessing Hadoop to handle the large volumes of data – 36 terabytes- that had to be consolidated into a single environment. Our Mobile Telecom customer decided to use Actuate – a Hortonworks partner in open source based Business Intelligence and Reporting Tools (BIRT) technology that connects analytics capabilities directly to Hadoop. Actuate’s ability to report directly against the Hadoop big data source, meanwhile, allows business users to generate on-demand analytics and reports consisting of thousands of pages in a matter of seconds through an easy-to-use web portal, with negligible training.

The Mobile Telecom giant now has a single source of clean data they can stand behind with absolute confidence in making the right decisions to stay competitive, and keep customer satisfaction levels high.  In addition, the consumer data services division is now in a position where it can replace several of its older systems, dropping extra licenses and hardware, because of the ability to do all of its business analytics in one place.

A Business Intelligence Analyst at the company stated; “It’s all automatic. Before, business users would be sending emails and calls to chase the data. Anyone across the whole business can have access to the information they need, and find it on their own. I particularly like the ability to drill down into the figures. You can now see at a glance what’s happening right across our activities.”

Customers’ want accurate and fast analytics reporting without a lot of training so a partnership between Hortonworks and Actuate, just makes big data sense.

Thank you to our partner Actuate for this Hadoop use case. Find more partners here.

 Actuate founded and co-leads the BIRT (Business Intelligence and Reporting Tools) open source project with the Eclipse Foundation, the home of the open source Eclipse Development Framework, the leading IDE worldwide. The BIRT project’s goal was to bring the web design metaphor to creating visualizations of data. 

The post Mobile Telco Dials In and Harnesses Big Data with Hadoop appeared first on Hortonworks.

One of the key benefits of the partnership is that it enables organizations to easily take advantage of the massive scale out storage and processing capabilities of Apache Hadoop with Splunk Enterprise via Splunk Hadoop Connect, which easily and reliably moves data between Splunk Enterprise and Hadoop.

This capability means the enterprise can easily use Splunk Enterprise to collect machine data from across the enterprise and deliver it to Hadoop for batch analytics. Likewise, the output of Hadoop jobs can be imported into Splunk Enterprise for rapid analysis and visualization.

Visit the Splunk website to learn more about Splunk Enterprise and Splunk Hadoop Connect.

Find out more about how Hadoop and the Hortonworks Data Platform enables next-generation data architecture.

The post Boosting Big Data and the Hadoop Ecosystem with Splunk Alliance appeared first on Hortonworks.

We have been blown away with the number and size of organizations who have downloaded the beta bits of this 100% open source, and native to Windows distribution of Hadoop and engaged Hortonworks and Microsoft around evolving their data architecture to respond to the challenges of enterprise big data.

With this key milestone HDP for Windows offers the millions of customers running their business on Microsoft technologies an ecosystem-friendly Hadoop-based solution that is built for the enterprise and purpose built for Windows. This release cements Apache Hadoop’s role as a key component of the next generation enterprise data architecture, across the broadest set of datacenter configurations as HDP becomes the first production-ready Apache Hadoop distribution to run on both Windows and Linux.

Additionally, customers now also have complete portability of their Hadoop applications between on-premise and cloud deployments via HDP for Windows and Microsofts’s HDInsight Service.

Enterprise Hadoop Momentum

Since its beta availability, we’ve been working with customers across a wide range of industries including automotive, manufacturing, financial services, retail and government. Here are just a few examples of the tremendous opportunity those customers are seeing:

• Automotive – a major automotive company wants to use HDP on Windows to create a centralized repository for all of the sensor data collected from their cars. The refinement and exploration of the data trends and patterns found through driving habits, maintenance and repair data and myriad other signals will be used to further improve the quality of their cars.
• Healthcare – a major healthcare applications provider is looking to build the next generation of healthcare apps that integrate patient health record data with clinical study and FDA data so that the customer experience is enriched and provides a higher level of health care services at a lower cost.
• Financial services – multiple major financial services organizations are looking to create centralized repositories across different divisions enabling them to explore and gain deeper insight into customer risk patterns.
• Manufacturing – a major manufacturer of electronics will create a centralized repository of machine generated data coming from the production lines and compare and analyze that data with part failure and return data enabling them to identify and predict problems in production and increasing the quality of their products.

This is just a small sample of the emerging use cases for HDP on Windows. You can explore how Hadoop fits into your data architecture here.

Availability & Training

Hortonworks Data Platform for Windows is now available for download at: http://hortonworks.com/download/.

We also have training specifically designed for HDP on Windows, you can get more information here: http://hortonworks.com/hadoop-training/hadoop-on-windows-for-developers/

The post Hadoop, Hadoop, Hurrah! HDP for Windows is Now GA! appeared first on Hortonworks.

The report indicated that enterprises could improve their business analytics deployment by facilitating communication between data scientists and the C-suite executives who make operational decisions. Specifically, executives should be aware of how these tools work and how to best utilize them to maximize their effectiveness.

An IT executive recently presented several steps companies can take to avoid common analytics pitfalls and optimize their business intelligence initiatives, including:

• Enterprises should broaden the focus of a business analytics program to the entire enterprise to find new connections and relationships. This includes expanding data collection efforts and taking a holistic view of analytics projects.
• Companies should look inward to find their data analytics leader. Many executives may be tempted to find a high profile hire who will jump-start operations, but an existing employee will already be familiar enough with the company's needs and culture to foster a successful data-driven culture.

Enterprises can attain significant benefits from their Hadoop analytics and business intelligence programs. However, getting the most out of these processes requires a broad vision, internal communication and a strong business culture dedicated to the pursuit of data analytics.

The post Getting the most out of business analytics appeared first on Hortonworks.

The newest trend in big data insight is analysis directly targeted at people, reported Bloomberg Businessweek. In particular, 'People Analytics' is an approach through which organizations turn the focus inward and use big data comprised of external research and intercompany observations to make operational improvements and interact better with employees.

"Because most communication and collaboration happens face to face, the data are critical for people analytics to take that next leap forward and become a transformative organizational tool," wrote Businessweek contributor Ben Waber. "By combining precise data from both real and virtual worlds, we can understand behavior at a previously unimaginable scale."

According to Businessweek, examples of company solutions that resulted from such a combination of data and observation include changing workday structure to improve morale and reduce attrition, as well as reducing the number of coffee stations to facilitate impromptu interactions among personnel.

Hyperpersonal analytics focus Hadoop big data inward
As Hadoop data becomes increasingly critical for facilitating measured objectivity to personal pursuits , more will start to apply it to their own activities. Harvard Business Review correspondent H. James Wilson wrote about 'auto-analytics,' a semantic category for the process of applying big data insights and visualization to daily actions.For example, according to a study by the Pew Researcher Center, 69 percent of people use a self-tracking mechanism when they exercise. Twenty-one percent of these people use smartphone apps or other data-based functions to make self-tracking more objective and precise.

Wilson wrote that people are seizing upon data-based technological resources to make their own lives and practices more efficient and informative. Technology consultants ABI Research projected that by 2018, more than 485 million wearable devices, from smart watches to various articles of smart clothing, will have been shipped worldwide. The trend toward data-based self-analysis parallels one of the important advances in Apache Hadoop, which offers its users a highly customizable, personalized approach to big data filtration and analysis.

The post Hadoop big data gets personal appeared first on Hortonworks.

SQL-In-Hadoop simply can’t be SQL without SQL compatibility

Today, HiveQL provides a fairly good set of SQL data types and semantics and while this (or a subset thereof) may be good enough for some of the “on” Hadoop solutions, we feel there needs to be more, especially if Hadoop and Hive are to meet the stringent requirements of enterprise class business analytics. To this end, we have set a goal of compatibility with most of SQL-92 and beyond with some SQL-2003 extensions.

The release of Apache Hive 0.11 pushes us further towards SQL-compatibility with the decimal data type becoming more usable (JIRA HIVE-4271) and the addition of analytic functions for windowing and aggregates.  It also vastly improves joins and all the while improves performance.  Awesome.

What else?

There is a lot more work to be done however and well work with the community to get it done.  Hive 0.11 had contributions from over 50 community members to close over 380 Jira tickets.  That is astounding and a huge proof point of the open community and its unrivaled capability to innovate faster than any proprietary solution.

We will reach our goal soon.  Here is what’s left to be done:

We look forward to providing updates to Hive all summer long!

The post Hive 0.11, Stinger and SQL-Compatibility appeared first on Hortonworks.

Delivering on Stinger: Phase 1. Just this week, Hive 0.11 has been released. Owen (@owen_omalley) brought us the news that 55 – yes, fifty-five – developers from across the community have addressed 386 JIRA tickets and have delivered significant improvements to Hive along with an awesome demonstration of the power of community open-source development. Thanks to everyone! This release of Hive means that we’ve delivered on the first phase of the Stinger Initiative too – aiming to deliver 100x performance increases to Hive.

Taking Hadoop Beyond Batch with YARN. All of which means we step closer to delivering SQL-in-Hadoop and respond to the needs of enterprises for multi-application operating systems for their big data. Arun (@arunmurthy) gives a terrific update on Hadoop 2.0 and YARN and how that development will move Hadoop Beyond Batch. Stay tuned!

Delivering Enterprise Hadoop through MTTR for HBase and NFS access to HDFS. Meanwhile, Nicolas Liochon (@nkeywal) and Devaraj Das (@ddraj) provide an introduction on how HBase availability is being improved through work on Mean Time To Recover (MTTR) capabilities. And then Brandon Li (@brandonli11) and Suresh Srinivas (@suresh_m_s) updated us on progress to simplify data management through NFS access to HDFS. All critical stuff for the enterprise, and all driven through the community.

Microsoft love for .NET Hadoop fans. If you’re a .NET developer and have been missing out on a little Hadoop fun, then Microsoft has started pushing out SDKs and tutorials for its Hadoop-in-the-Cloud service – HDInsight – so you can fire up Visual Studio and get rocking on that big data.

Hadoop Summit Meetups. We only announced them this week, and they’re nearly full already. Still time to try and squeeze into one of the Meetups: Hive, Pig, HBase, YARN, Accumulo, Ambari, Oozie, Data Science and Architecture or maybe attend Big Data Camp or Machine Learning Evening on 25th June as part of Hadoop Summit.

Now it’s time to go play. Have a great weekend.

The post Week in Review: SQL IN Hadoop and Hive, Beyond Batch with YARN, NFS access to HDFS and HBase MTTR appeared first on Hortonworks.

However, these organizations need access to accurate customer data to gain the full benefits of business analytics solutions. According to a recent Experian QAS survey, many financial institutions have struggled to ensure the accuracy of the data they gather, Credit Union Journal reported. Ninety-one percent of the organizations that participated in the survey suspected that the information they collected was inaccurate in some fashion. While respondents reported that as much as 18 percent of their data could not be ensured for accuracy on average, 27 percent of the total number of participating enterprises could not say how much of their information was compromised.

There are several steps that financial institutions can take to increase the accuracy of their data:

• Establish regular database maintenance tasks to manage files
• Integrate automated verification tools to ensure that client and prospect data is up to date
• Create a full data workflow to prioritize high-volume entry points

Information services expert Thomas Schutz noted in Bank Systems & Technology that banks and other financial institutions could improve the accuracy of their collected information by condensing the number of databases they maintained. This will prevent duplicate entries from being entered into multiple systems. One of the problems with operating multiple databases is that updated information may not be spread to each system, leaving some with inaccurate consumer data. In addition, Schutz recommended that banks place more emphasis on training personnel to enter and access data in a streamlined and uniform process. This will eliminate entry errors and inconsistencies, maintaining the accuracy of gathered data across the enterprise and ensuring that it maximizes the effectiveness of their Hadoop initiatives.

The post The importance of data accuracy for Hadoop banking analytics appeared first on Hortonworks.

Introducing Apache Hive 0.11 – 386 JIRA tickets closed

As representatives of this open, community led effort we are very proud to announce the first release of the new and improved Apache Hive, version 0.11.  This substantial release embodies the work of a wide group of people from Microsoft, Facebook , Yahoo, SAP and others.  Together we have addressed 386 JIRA tickets, of which there were 28 new features and 276 bug fixes. There were FIFTY-FIVE developers involved in this and I would like to thank every one of them.  See below for a full list.

Delivering on the promise of Stinger Phase 1

As promised we have delivered phase 1 of the Stinger Initiative in late spring.  This release is another proof point that that the open community can innovate at a rate unequaled by any proprietary vendor.  As part of phase 1 we promised windowing, new data types, the optimized RC (ORC) file and base optimizations to the Hive Query engine and the community has delivered these key features.

Key features in Hive 0.11

• ORCFile.  It’s Optimized.
  The ORC File (Optimized RC File) presents key new features that speed access of data Apache Hive as it adds meta information at the file and block data level so that queries can be more intelligent and use meta data to optimize access.  Further, with the ORC file, only the bytes from the required columns are read from HDFS which minimizes I/O and speeds the query chain.  These are major advances for improved performance in Hive.
• Improved Data Types
  As Apache Hive marches towards full SQL-compatibility, an update to the decimal data type was made more usable.
• Analytic Functions
  Hive 0.11 introduces windowing functions for RANK, LEAD/LAG, ROW_NUMBER, FIRST_VALUE, LAST_VALUE and more. It also introduces aggregate OVER functions with PARTITION BY and ORDER BY
• Joins improved in Hive 0.11
  Both the broadcast join and the SMB join were improved considerably in Hive 0.11.  Both joins work without user hints, so that the Hive optimizer now picks the correct join rather than depending on the user to do so. More broadcast joins are now packed into a single MapReduce job, making star join queries much more efficient.

Towards YARN and the Power of SQL-IN-Hadoop

Hadoop 2.0 and explicitly YARN turns Hadoop from a single application system to a multi-application operating system.  The next generation of Apache Hive, built on YARN, becomes part of the platform itself and can be managed by YARN to ensure that multiple use cases can be addressed beyond interactive query.  It is the delivery of a multi-application data system.  In this new world, Hive is a first class citizen along with a variety of workloads within a cluster and resources can be managed more discreetly.

Ultimately, this leads to further performance enhancements for Hive and with the inclusion of Tez, we will be able to demonstrate even more significant improvements as service startup times are removed a newly optimized execution chain within core Hadoop is delivered.  The near future is exciting!

Apache Hive is empowering an ecosystem of SQL Based Applications

This release represents significant enhancements to Hive that will improve direct SQL interaction with Hive and light up the hundreds of applications that already rely on Hive as the defacto SQL interface for Hadoop.  If you are one of the hundreds of software companies using Hive already, we hope you test out this new release and are happy with the results.  We look forward to supporting it in HDP 1.3 in the very near future. 

Thank You to the Community

Thanks to 55 developers who contributed time and effort on this release: Alan Gates, Amareshwari Sriramadasu, Andrew Chalfant, Arup Malakar, Ashish Singh, Ashish Vaidya, Ashutosh Chauhan, Bennie Schut, Bhushan Mandhani, Billie Rinaldi, Brock Noland, Carl Steinbach, Chen Chun, Chris Drome, Dilip Joseph, Edward Capriolo, Gang Tim Liu, Gopal V, Gunther Hagleitner, Harish Butani, Ivan Gorbachev, Jarek Jarcec Cecho, Jean Xu, Jingwei Lu, Johnny Zhang, Jonathan Chang, Kevin Wilfong, Lars Francke, Li Yang, Mark Grover, Mayank Garg, Mikhail Bautin, Namit Jain, Navis, Nick Collins, Owen O’Malley, Pamela Vagata, Prajakta Kalmegh, Prasad Mujumdar, Roshan Naik, Sam Tunnicliffe, Samuel Yuan, Sean Busbey, Shreepadma Venugopalan, Sushanth Sowmyan, Teddy Choi, Thejas M Nair, Thiruvel Thirumoolan, Travis Crawford, Vikram Dixit K, Vinod Kumar Vavilapalli, Wonho Kim, Xiao Jiang, Zhenxiao Luo

The post Apache Hive 0.11: Stinger Phase 1 Delivered appeared first on Hortonworks.