The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

HDFS Forum

WRF with Hadoop

  • #54423

    Is it possible build Weather Research and Forecasting (WRF) model in hadoop? Currently we are using computer cluster where there is one head node and 10 compute node; when a job submitted in the system through the head node, other 10 compute node perform the same submitted job. the total system act like a super-computer where total combined processing power used to perform the same job. is this possible with HDP?


  • Author
  • #54424
    Sheetal Dolas

    Absolutely! Hadoop cluster in a sense is a supercomputer where multiple nodes of cluster work together to achieve a common goal. That is how the MapReduce, Tez works and that’s how machine learning algorithms are executed using libraries like Mahout and R on hadoop.
    Quite a few weather researchers are already using Hadoop for this purpose.


    Thanks for reply. I am new in hadoop and have very little understanding, based on that knowledge when a job submitted in hadoop master node distribute the job to lots of data node and each data node perform its own portion of work and send the result back when data node complete its own portion. In this approach there is no communication between the data nodes each other. for example sometime it may require to communicate node to node which is called node to node message passing layer. what i need is I need to combine the total processing power to perform one singe job, the job cannot be distributed but only computation to complete the job can be distributed. please help me to understand how this works in hadoop.

The forum ‘HDFS’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.