From the Dev Team
Follow the latest developments from our technical team
Two months ago, we announced the acquisition of XA Secure. and at that time we stated that the software would be generally available by the end of June. We are happy to announce that we have delivered as promised and the solution is available for download for everyone today. Also, if you are an HDP Enterprise Plus Subscription customer, additional support for these new functions is now provided.
HDP Advanced Security expands on the solid security features already found in HDP to provide central administration and coordinated enforcement of enterprise security policy for a Hadoop cluster.…
We recently hosted the sixth of our seven Discover HDP 2.1 webinars, entitled Apache Storm for Stream Data Processing in Hadoop. Over 200 people attended the webinar and joined in the conversation.
Thanks to our presenters Justin Sears (Hortonworks’ Product Marketing Manager), Himanshu Bari (Hortonworks’ Senior Product Manager for Storm), and Taylor Goetz (Hortonworks’ Software Engineer and Apache Storm Committer) who presented the webinar. The speakers covered:
- Why use Apache Storm?
It has been an exciting few weeks for the XA Secure team. We formally joined Hortonworks on 5/15 and have received a warm welcome from our new peers. Even more exciting are the numerous discussions we have had with current customers and prospects on how we can bring together a comprehensive and holistic security capabilities to HDP. We now begin the journey to incubate our XA Secure functionality as a completely open source project governed by the Apache Software Foundation.…
We recently hosted the fifth of our seven Discover HDP 2.1 webinars, entitled Apache Solr for Hadoop Search. Over 200 people attended the webinar, prompting an informative discourse.
The speakers outlined the Apache Solr overview and features, followed by a practical demo of how to process, index, search, and visualize server log data.
Thanks to our presenters Justin Sears (Hortonworks’ Product Marketing Manager), Rohit Bakhshi (Hortonworks’ senior product manager), and Paul Codding (Hortonworks’ Solution Engineer) who presented the webinar.…
Enterprises are using Apache Hadoop powered by YARN as a Data Operating System to run multiple workloads and use cases instead of using it just as a single purpose cluster.
A multi-purpose enterprise wide data platform often referred to as a data lake gives rise to the need for a comprehensive approach to security across the Hadoop platform and the workloads. Few weeks back Hortonworks acquired XA Secure to further execute on our vision to bring a holistic security framework to the Hadoop community irrespective of the workload.…
Join us June 19 at 6 pm at the Hilton Fort Worth, Texas for an educational workshop hosted by Hortonworks and Sendero Business Services. The topic is “The Key To Success is Consistently Making Good Decisions & The Key To Good Decisions is Good Information.” The speaker is Don Hilborn, Solutions Engineer at Hortonworks.
Don will introduce the paradigm of
- Efficiency – double processing in Hadoop on the same hardware while providing predictable performance and quality of service; and
- Resource sharing – providing a stable common set of shared resources across multiple, coordinated workloads in Hadoop.
This is the second in the series of blogs exploring how to write data-driven applications in Java using the Cascading SDK. The series are:
Historically, programming languages and software frameworks have evolved in a singular direction, with a singular purpose: to achieve simplicity, hide complexity, improve developer productivity, and make coding easier. And in the process, foster innovation to the degree we have seen today—and benefited from.
Anyone among you is “young” enough to admit writing code in microcode and assembly language?…
With the release of Apache Hadoop YARN in October of last year, organizations are moving from single-application Hadoop clusters to a versatile, integrated Hadoop 2 data platform hosting multiple applications — eliminating silos, maximizing resources and bringing true multi-workload capabilities to Hadoop. Many enterprises have adopted YARN as the architectural center of a set of integrated technologies and capabilities that form the blueprint for enterprise Hadoop.YARN Enabling the Ecosystem Technologies
Hortonworks is making it easier to develop YARN applications through a number of technologies. …
Introduced in 2008, Apache Hive has been the de-facto SQL solution in Hadoop. By 2012, SQL had become a key battleground for Hadoop and many vendors started to publish benchmarks showing massive performance advantages their solutions had over Hive. Each of these vendors predicted that Hive would eventually be supplanted by the proprietary solution they were pushing.
The concerns about Hive’s performance were real. Hadoop in 2012 was a purely batch platform and no work had ever been done within Hive to address low-latency or interactive workloads.…
A significant reason for the increased adoption of the Hortonworks Data Platform by customers and partners has been Apache Hadoop YARN. This major advance, introduced last October in HDP2, allows Hadoop to move from many single-purpose clusters to a versatile, integrated data platform that hosts multiple business applications.
YARN has become the architectural center of Hadoop. We intend to make it easier for applications to work in a YARN environment, and benefit from the integrated capabilities and technologies that form the blueprint for enterprise Hadoop.…
Apache Ambari has always provided an operator the ability to provision an Apache Hadoop cluster using an intuitive Cluster Install Wizard web interface, guiding the user through a series of steps:
- confirming the list of hosts
- assigning master, slave, and client components to configuring services, and
- installing, starting and testing the cluster.
With Ambari Blueprints, system administrators and dev-ops engineers can expedite the process of provisioning a cluster. Once defined, Blueprints can be re-used, which facilitates easy configuration and automation for each successive cluster creation.…
We recently hosted the fourth of our seven Discover HDP 2.1 webinars, entitled Apache Hadoop 2.4.0, HDFS and YARN. It was very well attended and a very informative discourse. The speakers outlined the new features in YARN and HDFS in HDP 2.1 including:
- HDFS Extended ACLs
- HTTPs support for WebHDFS and for the Hadoop web UIs
- HDFS Coordinated DataNode Caching
- YARN Resource Manager High Availability
- Application Monitoring through the YARN Timeline Server
- Capacity Scheduler Preemption
Many thanks to our presenters, Rohit Bakhshi (Hortonworks’ senior product manager), Vinod Kumar Vavilapalli (co-author of the YARN Book, PMC, Hadoop YARN Project Lead at Apache and Hortonworks), and Justin Sears (Hortonworks’ Product Marketing Manager).…
Traditionally, HDFS, Hadoop’s storage subsystem, has focused on one kind of storage medium, namely spindle-based disks. However, a Hadoop cluster can contain significant amounts of memory and with the continued drop in memory prices, customers are willing to add memory targeted at caching storage to speed up processing.
Recently HDFS generalized its architecture to include other kinds of storage media including SDDs and memory . We also added support for caching hot files in memory .…
Julian Hyde will present the following talks at the Hadoop Summit:
What to do with all that memory in a Hadoop cluster? The question is frequently heard. Should we load all of our data into memory to process it? Unfortunately the answer isn’t quite that simple.
The goal should be to put memory into its right place in the storage hierarchy, alongside disk and solid-state drives (SSD).…