The Hortonworks Blog

The Spark lit in Azure HDInsight last week was just the beginning to this week’s energy and momentum we saw around the partnership at Microsoft’s Worldwide Partner’s Conference.  During the conference keynote, Microsoft CEO Satya Nadella affirmed, “Together with our partners we are transforming the business world for the next era”.

This year’s big announcement of Apache Spark being brought to HDInsight was the latest Big Data announcement delivered by Microsoft and Hortonworks.…

In version 1.2.0, Apache Spark introduced a Data Source API (SPARK-3247) to enable deep platform integration with a larger number of data sources and sinks. We are proud to announce that support for the Apache Optimized Row Columnar (ORC) file format is included in Spark 1.4 as a new data source. This support was added through a collaboration between Hortonworks and Databricks, tracked by SPARK-2883.

The Apache ORC file format and associated libraries recently became a top level project at the Apache Software Foundation.…

Microsoft’s Worldwide Partner Conference is well underway. The conference is a great opportunity for Microsoft’s partner community to understand key marketing and business strategies and engage with the Microsoft community. At this year’s conference, Microsoft’s National Sales Organization announced its 2015 Partner of the Year awards, and Hortonworks has been recognized with the Cloud Partner of the Year Award for both the West and Central U.S. Regions.

The Hortonworks team is honored to receive the Enterprise Partner Group Cloud Partner of the Year Award – an honor that we received thanks to our longstanding relationship with Microsoft.…

The world is now estimated to be churning out quintillions of bytes of data every day, which goes to show why the industry is struggling to keep up with the ever increasing demand for data scientists to pinpoint those insights that can increase company revenues, or better manage their resources.

The McKinsey Global Institute predicts the industry will need at least 190,000 more workers with analytics expertise and 1.5 million more data-savvy managers by 2018 in the US alone.…

Apache Spark has garnered a lot of developer attention and is often the top of agenda in my customer interactions. Since we announced support for Spark in HDP, we have seen broad customer adoption of our Spark offering. Our customers love Spark for the simplicity of its API, speed of development and the runtime performance. Spark is also democratizing Machine Learning and making it easier and approachable to more developers.

Today Microsoft announced support for Spark in HDInsight – this is a big step towards driving customer adoption for Spark workloads on Hadoop clusters in Azure.…

If security and compliance were already keeping you up at night, wait until you hear these statistics. Based on Vormetric 2015 Insider Threat Report, 44% of organizations in the United States have suffered a breach or failed a compliance audit in 2015 (48% in ASEAN and 40% in the U.K.).

Register Now

By leveraging Big Data, you accept that your data comes from a variety of sources, including CRM systems, files, spreadsheets, video, social media, payment data and health records and more.…

It was a frenetic month of June with Hadoop Summit in San Jose and the GDS International Oil and Gas CIO Summit the following week in San Antonio. Both events were well attended and very positively received by Hortonworks customers and O&G companies beginning to evaluate Hadoop for new advanced analytics applications.

Hadoop Summit was well attended by a large contingent across all segments of the industry and we saw a 500% growth in attendees from oil and gas customers over last year.…

Drink from Elephant’s Well Of Knowledge

Developer success starts with open and reusable code, and a community that allows for both consumption of code and contribution of updates to the code base. This success engenders a thriving and evolving community.

To that end, today we are announcing the Hortonworks Gallery for developers. Located on GitHub, the Gallery brings together the Hortonworks’ Apache Hadoop code, Apache Ambari Views and extensions, as well as related resources into a single view for developers to use within the familiar context of Git and open source software.…

Early this year, ApacheTM FalconTM became a Top Level Project (TLP) in the Apache Software Foundation.

The project continues to mature as a framework for simplifying and orchestrating data lifecycle management in Hadoop by offering out-of-the-box data management policies. The Apache Falcon 0.6.1 release builds on this foundation by providing simplified mirroring functionality and a new user interface (UI).

The community worked very diligently to offer more than 150 product enhancements, and over 30 new features and improvements.…

Hortonworks is always pleased to see new contributions come into the open-source community. We worked with our customer, Hotels.com, to help them develop libraries and utilities around Apache Hive, the Apache ORC format and Cascading. It’s great to see the results released for the community. In this guest blog, Adrian Woodhead, Big Data Engineering Team Lead at Hotels.com, discusses the CORC project.

Hotels.com is pleased to announce the open source release of Corc, a library for reading and writing files in the Apache ORC file format using Cascading.…

The Apache Lucene/Solr community is continuing its rapid release cycles to meet community and customer requirements. In this guest blog, we have invited Sarath Jarugula from Lucidworks to share with us the many improvements in the Apache Solr 5.2 release.

The Apache Solr community has announced its Solr 5.2 release. Solr 5.2 is a follow-up release to Solr 5.0, a significant major release in February 2015. The community has delivered 25 new features, 5 optimizations, and 38 bug fixes in this release.…

As YARN drives Hadoop’s emergence as a business-critical data platform, the enterprise requires more stringent data security capabilities. The Apache Ranger delivers a comprehensive approach to security for a Hadoop cluster. It provides a platform for centralized security policy administration across the core enterprise security requirements of authorization, audit and data protection.

On June 10th, the community announced the release of Apache Ranger 0.5.0. With this release, the community took major steps to extend security coverage for Hadoop platform and deepen its existing security capabilities.…

Earlier this month, Hortonworks had the pleasure of joining Yahoo! in hosting the 8th Annual Hadoop Summit, the leading conference for the Apache Hadoop community. Summit is always an important and exciting experience, bringing together thought leaders, technologists, and data specialists from throughout the community to explore and advance the art and science of Big Data.

This year’s event came at a pivotal time for Hadoop and Hortonworks, with news about Open Enterprise Hadoop and the launch of the newest version of Hortonworks Data Platform (HDP 2.3™) poised to transform the way large organizations in every industry process data.…

In his blog, Tim Hall wrote, “Enterprises are embracing Apache Hadoop to enable their modern data architectures and power new analytic applications. The freedom to choose the on-premises or cloud environments for Hadoop that best meets the business needs is a critical requirement.”

One of the choices in deploying Hadoop in the cloud environment is with Microsoft Azure using Cloudbreak. Other choices include Google Cloud Platform, Openstack, and AWS.

But in this blog, I’ll show how you can deploy Hadoop in Azure with few clicks by running HDP multimode cluster in Azure’s Linux VM using Cloudbreak.…

Mayank Bansal, of EBay, is a guest contributing author of this collaborative blog.

This is the 4th post in a series that explores the theme of enabling diverse workloads in YARN. See the introductory post to understand the context around all the new features for diverse workloads as part of Apache Hadoop YARN in HDP.

Background

 In Hadoop YARN’s Capacity Scheduler, resources are shared by setting capacities on a hierarchy of queues.…