The Hortonworks Blog

Is a Lake Big Enough to House Your Ocean of Data?

Contrary to popular belief, Hadoop was not the elephant-in-the-china-shop that marauded and disrupted the data center. The real culprit is data and how it has exploded in volume. The past two or three years have seen a rise in the number of successful Hadoop projects in enterprises to tackle this explosion of big data. These large volumes of data, the emergence of the Hadoop technology and the need to store all the siloed data in one place have prompted the phenomenon called the Data Lake among enterprises.…

Our guest blogger today is Rob Rosen, Senior Director Partner Solutions at Platfora, describes how to help customers achieve strategic advantage through data discovery.

While many people have heard the notion of “known unknowns” and “unknown unknowns,” it may surprise you to discover that the concept was first popularized by a NASA scientist. In a presentation given at TEDx GeorgeMasonU, Dr. Kirk Borne described how he used the concept of “known unknowns” (things that we knew might exist, but hadn’t seen evidence of) and “unknown unknowns” (things that we could discover and knew nothing about, but would truly surprise us), and how they relate to the concept of Big Data.…

The advent of connected manufacturing has ushered in an era where low-cost machine sensors take thousands of measurements per second at many points across the manufacturing process. This stream of sensor data enables manufacturers to quickly detect emerging anomalies and solve issues before they impact yield and quality.

Big Data insights enable predictive analytics for those rapid, proactive process adjustments. Manufacturers can capitalize on this opportunity by following an approach that combines the power of Teradata with Hortonworks Data Platform’s storage and compute efficiencies at extreme scale.…

I recently had the pleasure of visiting with Arvind Battula, Sr. Data Scientist at Schlumberger. We discussed his background as a chemical and mechanical engineer and his move onto the Data and Analytics team as a data scientist. The following is a transcript of my conversation with Arvind. We discussed his background, his interesting focus areas for data science in oil and gas, and technologies that he believes will help transform the industry.…

Metro Transit of St. Louis (MTL) operates the public transportation system for the St. Louis metropolitan region. The organization’s mission is “Meeting the region’s transit needs by providing safe, reliable, accessible, customer-focused service in a fiscally responsible manner.”

Meeting the Challenge to Provide Safe, Reliable Public Transport

To ensure the safety of passengers and the proper use of public funds, MTL has always performed regular maintenance on its bus fleet. But lacking detailed data on how bus components were actually performing, the agency maintained vehicles retroactively.…

The Personalized Medicine Initiative (PMI), based out of the Life Sciences Institute of the University of BC, has deployed HDP and PHEMI Central Big Data Warehouse to collect, store and manage genomic and clinical data for Molecular You (MY). 

PHEMI is a Hortonworks Technology partner and in this blog, Richard Proctor, General Manager, Global Healthcare at Hortonworks interviews PHEMI’s Roy Wilds, Dir. of Product Management, along with PMI’s Chief Operating Officer and Co-founder of Molecular You, Rob Fraser, to discuss this groundbreaking work.  …

The journey to data driven business transformation can be confusing and challenging. At Hortonworks, we understand this, and are offering a number of tools that will help companies map out their journey to fully utilize the value of their Big Data.

The journey begins with understanding the opportunities unique to your business, and understanding how the maturity of your organization enables or inhibits your ability to strategically pursue Big Data programs aligned to your business goals.…

There’s excitement in the air as one of Benelux’s largest Big Data conferences “Big Data Expo”, comes to Utrecht in The Netherlands.

We’re sponsoring and you’ll find our experts Chris Harris and Jhon Masschelein presenting such topics as “5 Steps for Effective use of Apache Spark in Hortonworks Data Platform 2.3” and “Lessons Learned: 5 Common Hadoop Use Cases”. You can register here.

As Hortonworks continues to extended its footprint in Europe, we’re seeing  some exciting use cases and an increasing momentum of enterprise adoption of Hadoop.…

Today Microsoft has announced the Generally Availability of Azure HDInsight, with Apache Hadoop 2.6, available on Ubuntu Linux clusters. Azure HDInsight is a Hadoop managed service in the cloud and uses the Hortonworks Data Platform (HDP).

This release is a direct result of the commitment that Microsoft has to Open Source. Microsoft has worked along with Hortonworks® in the community to contribute towards Apache Hadoop and related projects, including Apache Ambari.…

Today, I’m excited to share that we have released the GA version of Hortonworks DataFlow (HDF), a new offering that directly addresses the unique big data needs of the Internet of Anything (IoAT). Hortonworks DataFlow is powered by Apache Nifi a top-level open source project made available through the NSA Technology Transfer Program.

By making this technology a commercial offering, we now provide our customers the ability to connect, collect and curate data from a broad spectrum of connected yet disparate data sources – sensors, machines, geo-location devices, social feeds, connected cars, web clicks, server logs and more.…

Since the partnership between Hortonworks and SAS we have created some awesome assets (i.e., SAS Data Loader sandbox tutorial, educational webinars and array of blogs) that have enabled Hadoop and Big Data enthusiasts’ hands-on training with Apache Hadoop and SAS’ powerful analytics solutions. You can find more details around our partnership and resources here: http://hortonworks.com/partner/sas

To continue the momentum, we have Paul Kent, Vice President of Big Data at SAS, share his insights on the value of  YARN and the benefits it brings to SAS and its users- this time around SAS Grid and YARN. …

Big Data, the Internet of Anything (IoAT) and the Connected Car have created a new Information Superhighway that fundamentally changes the relationship between automakers and car buyers.

Previously, automakers had an incomplete feedback loop after they sold a vehicle. They learned of negative customer sentiment through slumping sales, increasing warranty expenses or when they needed to recall their vehicles. Positive signals of driver happiness were similarly sparse.

Read the White Paper

The connected car has changed all that.…

In a world that creates 2.5 quintillion bytes of data every year, how can organizations take advantage of unprecedented amounts of data? Is data becoming the largest untapped asset? What architectures do companies need to put in place to deliver new business insights while reducing storage and maintenance costs?

Cisco and Hortonworks have been partnering since 2013 to offer operational flexibility, innovation and simplicity when it comes to deploying and managing Hadoop clusters.…

Yahoo! JAPAN needed a data platform that could scale to generate 100,000 reports per day as well as having the ability to process large amounts of data. It needed to keep the last 13 months’ worth of data, which is approximately 500 billion rows, organized and easily accessible. Relational Database Management Systems (RDBMS) cannot scale to these levels from a cost and processing power perspective. Yahoo! JAPAN explored Hadoop to achieve this and evaluated two platforms based on our requirements; Hortonworks Hive and Tez on YARN and Cloudera Impala.…

Hortonworks is a huge supporter of the Apache Software Foundation (ASF) and fully embrace the processes and procedures through the only 100% open source Hadoop platform HDP. As Forrester VP Mike Gualtieri said in the Forrester Wave “Hortonworks lives and loves open source.” And that will be fully on display at the inaugural at Apache: Big Data Europe 2015 event this year in Budapest, Hungary.

The event will be held 28-30 September at the Corinthia Hotel and Hortonworks will be contributing in a big way as a Diamond Sponsor.…