Home Forums HDFS Communicating with HDFS from outside

This topic contains 4 replies, has 2 voices, and was last updated by  Robert Molina 9 months ago.

  • Creator
    Topic
  • #47174

    Aysan Rasooli
    Participant

    Hi,

    I have Hortonworks installed in my server, and I have few Hive scripts doing some process and generating the results back into the HDFS. Now, I want to develop a portal to get some configuration variables from user, and present the query results back in my portal. I was wondering how can I transfer the results from HDFS into my portal. Is there any component for this? Or shall I have a Sql DB in middle and have the HDFS put the results in Sql using sqoop, and portal reading from the Sql. I would appreciate your help on this.

    Best,
    Aysan

Viewing 4 replies - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #47282

    Robert Molina
    Moderator

    Hi Aysan,
    WebHdfs should be installed on the sandbox. As long as the property within hdfs-site.xml is set to true.

    dfs.webhdfs.enabled
    true

    Webhdfs is default on 50070 I believe.

    Hope that helps.

    Regards,
    Robert

    Collapse
    #47218

    Aysan Rasooli
    Participant

    Hi Robert,

    Another quick question: Is WebHDFS installed on HDP sandbox? Is it required to do any settings to make it work on the sandbox? Could you please let me know which port shall I use in webhdfs rest call to connect to the HDFS?

    I really appreciate your help!
    Best,
    Aysan

    Collapse
    #47190

    Aysan Rasooli
    Participant

    Hi Robert,
    Thank you very much for your reply. In my portal I want users to change Hive variables whenever they want, and upon any changes in the variables, I need to restart the Hive queries with new variables. To do this entire process, do I need to have a business layer between my portal and the HDP? So, that it detects the variable changes and starts new queries? How can I manage this in a cluster which only has HDP installed on it?

    Collapse
    #47184

    Robert Molina
    Moderator

    Hi Aysan,
    HDFS has the webhdfs feature where one can upload or download the file from the cluster. So that would be one way. Your other way of using sqoop to export the data to a sql db, then have the client query that db would also work as well. Placing it in the DB probably allows the client to possible consume it faster if there are extra queries being done on the data already outputted by hive.

    Regards,
    Robert

    Collapse
Viewing 4 replies - 1 through 4 (of 4 total)