Hortonworks Sandbox Forum

What is Templeton configuration? (connecting via Talend Studio)

  • #25580
    Kai Waehner
    Member

    HDP seems to work fine. I can run PIG, HIVE, etc. I use Talend Studio.

    I just cannot run HCatalog examples. I think my Templeton configuration is wrong. My host IP is correct (IP of sandbox). My port is 50111. Maybe this is wrong? I cannot access http://“IP-of-HDP”:50111 via webbrowser.

    What is the correct configuration of Templeton? Can I find out this information from HDP vm or within web application?

    Here is my error when running your Talend example job. It already dies when trying to create a HCatalog database:

    [statistics] connecting to socket on port 3386
    [statistics] connected
    2013-05-15 16:09:53.314 java[5945:120b] Unable to load realm info from SCDynamicStore
    Exception in component tHCatalogOperation_1
    2013-05-15 16:10:08|CGgn8K|APRIL_2013_LOCAL_PROJECT|hcatalog_load|java||Failed|Job execution error|Error 500 (Internal Server Error): We received an unexpected result.
    —>Failure calling hcat: create database talend location ‘/user/hdp/test/usersdb';
    [statistics] disconnected
    java.lang.Exception: Error 500 (Internal Server Error): We received an unexpected result.
    —>Failure calling hcat: create database talend location ‘/user/hdp/test/usersdb';
    at april_2013_local_project.hcatalog_load_0_1.hcatalog_load.tHCatalogOperation_1Process(hcatalog_load.java:1696)
    at april_2013_local_project.hcatalog_load_0_1.hcatalog_load.tHDFSDelete_1Process(hcatalog_load.java:1533)
    at april_2013_local_project.hcatalog_load_0_1.hcatalog_load.runJobInTOS(hcatalog_load.java:6402)
    at april_2013_local_project.hcatalog_load_0_1.hcatalog_load.main(hcatalog_load.java:6236)
    job failed
    java.lang.Exception: Error 500 (Internal Server Error): We received an unexpected result.
    —>Failure calling hcat: create database talend location ‘/user/hdp/test/usersdb';
    at april_2013_local_project.hcatalog_load_0_1.hcatalog_load.tHCatalogOperation_1Process(hcatalog_load.java:1696)
    at april_2013_local_project.hcatalog_load_0_1.hcatalog_load.tHDFSDelete_1Process(hcatalog_load.java:1533)
    at april_2013_local_project.hcatalog_load_0_1.hcatalog_load.runJobInTOS(hcatalog_load.java:6402)
    at april_2013_local_project.hcatalog_load_0_1.hcatalog_load.main(hcatalog_load.java:6236)

    Thanks for help.
    Kai

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #25646
    Seth Lyubich
    Moderator

    Hi Kai,

    Tempeton/WebHcat should run on port 50111. It does not have a web ui, but you can test if Templeton responds with the command below using you browser:

    http://ip-of-sandbox:50111/templeton/v1/status .

    If it responds you should see something like below:

    {“status”:”ok”,”version”:”v1″}

    Can you please test this? If you still have an issue can you please let us know which example you are using and we will try to reproduce.

    Hope this helps,
    Thanks,
    Seth

    #25678
    Kai Waehner
    Member

    Thanks for quick answer.

    My Templeton status is ok. I am trying to run your example Talend job “hcatalog_load”. Maybe I have done some misconfiguration here.

    tHDFSDelete works, then tHCatalogOperation fails with the above error message, I cannot create the database. So connection via HCatalog does not work.

    Do you see some misconfiguration?

    Distro: Hortonworks V1
    Templeton hostname: “IP of VM”
    Templeton port: 50111
    Operation on “Database”: “Create”
    username: sandbox
    database location: /user/hdp/test/usersdb
    database description: “”
    die on error: true

    #25929
    Seth Lyubich
    Moderator

    Hi Kai,

    I have not tried Talend with Sandbox and I am not sure if this is supported but I think default database location is /apps/hive/warehouse.

    Can you please try to change database location to ‘/apps/hive/warehouse’ and try again?

    Thanks,
    Seth

    #25933
    Seth Lyubich
    Moderator

    Hi Kai,

    I just tried and I was able to create database using Talend when I chose database location as /apps/hive/warehouse.

    Hope this helps.

    Thanks
    Seth

    #26331
    Kai Waehner
    Member

    Thanks, “/apps/hive/warehouse” seems to work.

    Though, now the second tHCatalogOperation (for creating the Table) does not work.

    Operation: Table (Drop if exists and create)
    Database: “talend” (which I created before)
    Table: “users”
    Username: “sandbox” (same as before)

    Can you maybe send me your working Talend job via email? I can then adjust it with my configuration. That would be easier to solve the problem…
    Thanks.

    ######

    Here is the error (connection to database not working) ?!

    Starting job hcatalog_load at 08:03 27/05/2013.

    [statistics] connecting to socket on port 3706
    [statistics] connected
    Exception in component connectionStatsLogs
    com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:

    ** BEGIN NESTED EXCEPTION **

    java.net.SocketException
    MESSAGE: java.net.ConnectException: Connection refused

    STACKTRACE:

    java.net.SocketException: java.net.ConnectException: Connection refused
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:156)
    at com.mysql.jdbc.MysqlIO.(MysqlIO.java:284)
    at com.mysql.jdbc.Connection.createNewIO(Connection.java:2569)
    at com.mysql.jdbc.Connection.(Connection.java:1485)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
    at java.sql.DriverManager.getConnection(DriverManager.java:579)
    at java.sql.DriverManager.getConnection(DriverManager.java:221)
    at routines.system.SharedDBConnection.getConnection(SharedDBConnection.java:63)
    at routines.system.SharedDBConnection.getDBConnection(SharedDBConnection.java:131)
    at demo_project_5_2_1.hcatalog_load_0_1.hcatalog_load.connectionStatsLogsProcess(hcatalog_load.java:6698)
    2013-05-27 08:03:45|pCCmWu|DEMO_PROJECT_5_2_1|hcatalog_load|java||Failed|Job execution error|Communications link failure due to underlying exception:

    ** BEGIN NESTED EXCEPTION **

    java.net.SocketException
    MESSAGE: java.net.ConnectException: Connection refused

    #26411
    Kai Waehner
    Member

    Ok guys. Thanks for your help. Problem solved. The last error was some “strange” status of my environment. I used a new Talend Studio and restarted everything. Now, it works.

    Just one last (newbie) question: Why is database location /apps/hive/warehouse? in HDFS, the data is here: / user/ hdp/ test/ data (as I defined it this way in Talend’s components? Can you please explain me why I nee an additional database location? What is it? Probably, I have to read some getting started articles about Hive, right ?!

    #26436
    Larry Liu
    Moderator

    Hi, Kai

    The database location is configured in the hive-site.xml.

    hive.metastore.warehouse.dir
    /apps/hive/warehouse

    Larry

    #54502
    Mo Lin
    Participant

    Seth,

    I run into the same issue with Kai’s, the error looks like

    [statistics] connecting to socket on port 3877
    [statistics] connected
    [INFO ]: org.apache.hadoop.conf.Configuration.deprecation – fs.default.name is deprecated. Instead, use fs.defaultFS
    Exception in component tHCatalogOperation_1
    java.lang.Exception: Error 500 (Internal Server Error): We received an unexpected result.
    —>Failure calling hcat: create database talend location ‘/apps/hive/warehouse';
    at webloganalysis.test_0_1.test.tHCatalogOperation_1Process(test.java:628)
    at webloganalysis.test_0_1.test.tHDFSDelete_1Process(test.java:470)
    at webloganalysis.test_0_1.test.runJobInTOS(test.java:924)
    at webloganalysis.test_0_1.test.main(test.java:755)
    [statistics] disconnected
    Job test ended at 15:57 27/05/2014. [exit code=1]

    I tried http://ip-of-sandbox:50111/templeton/v1/status and it responds with correct answer like below

    {“status”:”ok”,”version”:”v1″}

    please advise

    thanks,
    Nemo

    #54726
    Mo Lin
    Participant

    Looks like the issue only happens on HDP for Windows (2.0 & 2.1). I tested against HDP Linux and worked fine. Any advice?

    thanks,
    Nemo

    #54734
    Mo Lin
    Participant

    I checked templeton.log and found below invalid file path “\hcatalog/bin”. looks like a bug for HDP for Windows. Can anyone help confirm this?

    Running: C:\Python27\python.exe c:\hdp\hive-0.13.0.2.1.1.0-1621\hcatalog/bin/hcat.py

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.