Home Forums Hue Hue questions

Tagged: ,

This topic contains 18 replies, has 5 voices, and was last updated by  Vikas Parashar 1 month, 1 week ago.

  • Creator
    Topic
  • #45506

    lou82
    Participant

    Hi – I am having some issues with setting up HUE 2.3 on an existing HPD 2.0 cluster. I am getting errors with Beeswax with my query editor tab as well as for Databases, and Tables. Any thoughts?

    IndexError at /beeswax/execute/
    list index out of range
    Request Method: GET
    Request URL: http://hue:8000/beeswax/execute/
    Django Version: 1.2.3
    Exception Type: IndexError
    Exception Value:
    list index out of range
    Exception Location: /usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py in _get_col_position, line 190

Viewing 18 replies - 1 through 18 (of 18 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #58453

    Vikas Parashar
    Participant

    Hi folks,

    Got it, that was my silly mistake..

    Actually no one have the permission to write on hdfs apart from HDFS user.

    Collapse
    #58451

    Vikas Parashar
    Participant

    Hello,

    Today i get stuck in a strange behavior of hive. I have sql query that perfectly works when i try to run with hue. But in case of hive shell, it promting null value error.

    My query is–>

    insert into table atmtxn_orc PARTITION (settlementdate) select InternalID ,OldID ,RecordInternalID ,ProcessorID ,Terminal ,TxnTypeID ,ResponseCodeID ,RejectCodeID ,BankID ,NetworkID ,TerminalSequenceNumber ,SettlementTime ,ActivityDate ,ActivityTime ,Amount ,Fee ,Surcharge ,Interchange ,Txn ,OurBank ,EBTTransaction ,LogID ,P_NetworkCode ,P_TxnCode ,P_ResponseCode ,P_RejectCode ,P_Field1 ,P_Field2 ,ATMInternalID ,EP ,InterchangeCalc ,International ,IchgRateID ,Branded ,DenyTxnType ,CardID ,DCCAmount ,DCCTotal ,DCCTXn ,Deny711 ,CountryCode ,HP4 ,HP5 ,HP6 ,HP7 ,HP8 ,HP9 ,HP10 ,HP11 ,HP12 ,HP13, HP14, HP15, HPCATM ,PAN,from_unixtime(unix_timestamp(concat(substr(settlementdate,1,10),’ ‘,substr(settlementdate,12,12)),’yyyy-MM-dd HH:mm:ss.SSS’)) from atmtxn_staging where settlementdate = ’2014-01-02 05:00:00′;

    ======================>
    and o/p from hive sheel is
    ==>
    hive> insert into table atmtxn_orc PARTITION (settlementdate) select InternalID ,OldID ,RecordInternalID ,ProcessorID ,Terminal ,TxnTypeID ,ResponseCodeID ,RejectCodeID ,BankID ,NetworkID ,TerminalSequenceNumber ,SettlementTime ,ActivityDate ,ActivityTime ,Amount ,Fee ,Surcharge ,Interchange ,Txn ,OurBank ,EBTTransaction ,LogID ,P_NetworkCode ,P_TxnCode ,P_ResponseCode ,P_RejectCode ,P_Field1 ,P_Field2 ,ATMInternalID ,EP ,InterchangeCalc ,International ,IchgRateID ,Branded ,DenyTxnType ,CardID ,DCCAmount ,DCCTotal ,DCCTXn ,Deny711 ,CountryCode ,HP4 ,HP5 ,HP6 ,HP7 ,HP8 ,HP9 ,HP10 ,HP11 ,HP12 ,HP13, HP14, HP15, HPCATM ,PAN,from_unixtime(unix_timestamp(concat(substr(settlementdate,1,10),’ ‘,substr(settlementdate,12,12)),’yyyy-MM-dd HH:mm:ss.SSS’)) from atmtxn_staging where settlementdate = ’2014-01-02 05:00:00′;
    Query ID = root_20140807162222_670485e4-a554-42f9-b129-7a67afae4af9
    Total jobs = 1
    In order to set a constant number of reducers:
    Starting Job = job_1407391608252_0070, Tracking URL = http://agent2.poc.com:8088/proxy/application_1407391608252_0070/
    Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1407391608252_0070
    Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
    2014-08-07 16:23:07,796 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.37 sec
    2014-08-07 16:23:15,127 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.69 sec
    MapReduce Total cumulative CPU time: 3 seconds 690 msec
    Ended Job = job_1407391608252_0070
    Loading data to table test.atmtxn_orc partition (settlementdate=null)
    Failed with exception copyFiles: error while checking/creating destination directory!!!
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
    MapReduce Jobs Launched:
    Job 0: Map: 1 Reduce: 1 Cumulative CPU: 3.69 sec HDFS Read: 2620 HDFS Write: 6167 SUCCESS
    Total MapReduce CPU Time Spent: 3 seconds 690 msec
    ============================

    Collapse
    #53013

    Very Test
    Participant

    Hi Dave

    Thanks you ! You save my life (again).

    Best regards

    Collapse
    #52989

    Dave
    Moderator

    Hi Very,

    Our version of hue does not support hiveserver2 – which is why you get the above error.
    server_interface=hiveserver2

    Please change this to beeswax.

    Thanks

    Dave

    Collapse
    #52619

    Very Test
    Participant

    Hello

    I am facing similar issue I think. I configured hue.ini
    beeswax_server_host=localhost
    beeswax_server_port=19800
    server_interface=hiveserver2
    The hiveserver2 is up and running to port 19800 and configured to MySQL. The metastore runs on 19805 port. The following commands ran successfully:
    /usr/lib/hive/bin/schematool -dbType mysql -initSchema
    /usr/lib/hive/bin/hive -e “show tables”
    /usr/lib/hive/bin/metatool -listFSRoot

    When I navigate to Hue, everything is Ok except the following URLs:
    /hcatalog
    /beeswax/tables

    The exception in runcpserver.log is:

    [30/Apr/2014 10:01:18 +0000] access       INFO     192.168.18.1 laya-hadoop - "GET /hcatalog/ HTTP/1.0"
    [30/Apr/2014 10:01:18 +0000] views        DEBUG    Getting database name from cookies
    [30/Apr/2014 10:01:18 +0000] thrift_util  DEBUG    Thrift call: <class 'TCLIService.TCLIService.Client'>.GetSchemas(args=(TGetSchemasReq(schemaName=None, sessionHandle=TSessionHandle(sessionId=THandleIdentifier(secret='\xe2\x987\xedL\xcd@\xfe\x9al\xa0c\xc37:\x11', guid='F\x13\xc5\x05\x0b.E\x08\xbabP\x14\xf9R\xbb\xa
    ...
    [30/Apr/2014 10:01:18 +0000] middleware   INFO     Processing exception: list index out of range: Traceback (most recent call last):
      File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/handlers/base.py", line 100, in get_response
        response = callback(request, *callback_args, **callback_kwargs)
      File "/usr/lib/hue/apps/hcatalog/src/hcatalog/views.py", line 53, in index
        return show_tables(request, database=database)
      File "/usr/lib/hue/apps/hcatalog/src/hcatalog/views.py", line 92, in show_tables
        databases = db.get_databases()
      File "/usr/lib/hue/apps/beeswax/src/beeswax/server/dbms.py", line 92, in get_databases
        return self.client.get_databases()
      File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 630, in get_databases
        return [table[col] for table in self._client.get_databases()]
      File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 362, in get_databases
        return HiveServerTRowSet(results.results, schema.schema).cols((col,))
      File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 126, in cols
        cols[col_name] = row.col(col_name)
      File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 186, in col
        pos = self._get_col_position(colName)
      File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 190, in _get_col_position
        return filter(lambda (i, col): col.columnName == column_name, enumerate(self.schema.columns))[0][0]
    IndexError: list index out of range
    

    Any clues ?

    Collapse
    #46914

    Carlos Diaz
    Participant

    Thanks Dave. I got it working now!

    At first, i tried changing server_interface to hiveserver2, but when I did that I could no longer use the beeswax interface. After looking around on the internet, I noticed that someone else was having a similar issue but their issue was that hue could not access the hive-site.xml file. I know that i did have a hive-site.xml file in /etc/hive/conf/hive-site.xml, but what I never realized is that since my hive server runs on a different host, /etc/hive/conf/hive-site.xml on this server was set to default values. I simply copied over the /etc/hive/conf/hive-site.xml from the hive server and it started working now.

    Thanks again!

    Collapse
    #46777

    Dave
    Moderator

    Hi Carlos,

    In your hue.ini are you pointing to hiveserver2?
    server_interface=hiveserver2

    This looks like it is caused by Hue to talk with hiveserver2 requires hive 10, we use hive 12 which appears to be the problem.

    Thanks

    Dave

    Collapse
    #46775

    Carlos Diaz
    Participant

    Please let us know if/how you were able to solve this. I’m also experiencing the exact same issue and I have not been able to figure this out.

    Collapse
    #46773

    Dave
    Moderator

    Hi Lou,

    I have sent you an email to your registered email address.

    Thanks

    Dave

    Collapse
    #46731

    lou82
    Participant

    Oh My apologies! Ok since the space is limited in posts is there another way I can post it or send it to you?

    Collapse
    #46727

    Dave
    Moderator

    Hi Lou,

    I was waiting for you to post your hue.ini so I could have a look at it.

    Thanks

    Dave

    Collapse
    #46685

    lou82
    Participant

    Anyone have any information on this? i still do not have HUE set up correctly. Any information is appreciated!

    Collapse
    #46187

    Carlos Diaz
    Participant

    Was anyone able to figure this out? I’m having the exact same issue. I installed a 10 node cluster using Abari. Out of those, I have 6 slaves that are running DataNode, HBase Region Server and NodeManager. Then I’m using the other nodes to run some of the master services as well as the mysql database.

    While going through the HUE install procedures listed here:

    http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap-hue.html

    and then go to start hue, I get the following error in the /var/log/hue/beeswax_server.out file:

    “FATAL beeswax.Server: hive.metastore.local is set to false. The Beeswax internal metastore is not supposed to run.”

    From looking at the Apache documentation it almost sounds like we are supposed to remove the ‘hive.metastore.uris’ key/value from /etc/hive/conf/hive-site.xml for a local metastore. However, when I try to clear out that field from Ambari, I get an error that this field cannot be empty. Yet in the following document, it says that the hive.metastore.uris should be empty if you are using a local metastore, which this is! What gives?

    https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-LocalMetastore

    Can anyone help?

    Collapse
    #46157

    Dave
    Moderator

    Hi Lou,

    Posting your entire hue.ini file will be helpful.
    Do you have the hive client installed on your HUE server?

    Bear in mind that you should have a fully functioning cluster and then you point HUE at the various components. These components should not point to HUE itself – think of it more as a web console to interact with the deployed cluster – it is not like Ambari where it configures & runs your cluster.

    Thanks

    Dave

    Collapse
    #46122

    lou82
    Participant

    so my cluster is 14 data nodes and then I have my name nodes. I set up HUE on my primary name node but my master services like Hive, etc are on other data nodes. So should I have things pointing to the Hive server or the HUE server? I am confused on that maybe. Some things in HUE are working , however HIve Query editor, Databases, and Tables tab all provide the error above. I also get a similar error with HCatalog and then my job browser also errors that there was an error connecting to the server and just spins. Pig Shell also seems to be present in HUE.

    Any thoughts? Should post my whole hue.ini?

    Thanks very much for any help!

    Collapse
    #45900

    Dave
    Moderator

    Hi Lou,

    Are you running everything on a single host?
    As this is from my hue.ini [note everything is commented out to accept default values]

    [beeswax]

    # Host where Beeswax server Thrift daemon is running.
    # If Kerberos security is enabled, the fully-qualified domain name (FQDN) is
    # required, even if the Thrift daemon is running on the same host as Hue.
    ## beeswax_server_host=

    # Port where Beeswax Thrift server runs on.
    ## beeswax_server_port=8002

    # Host where internal metastore Thrift daemon is running.
    ## beeswax_meta_server_host=localhost

    # Configure the port the internal metastore daemon runs on.
    # Used only if hive.metastore.local is true.
    ## beeswax_meta_server_port=8003

    # Hive home directory
    ## hive_home_dir=/usr/lib/hive

    # Hive configuration directory, where hive-site.xml is located
    ## hive_conf_dir=/etc/hive/conf

    # Timeout in seconds for thrift calls to beeswax service
    ## beeswax_server_conn_timeout=120

    # Timeout in seconds for thrift calls to the hive metastore
    ## metastore_conn_timeout=10

    # Maximum Java heapsize (in megabytes) used by Beeswax Server.
    # Note that the setting of HADOOP_HEAPSIZE in $HADOOP_CONF_DIR/hadoop-env.sh
    # may override this setting.
    ## beeswax_server_heapsize=1000

    # Share saved queries with all users. If set to false, saved queries are
    # visible only to the owner and administrators.
    ## share_saved_queries=true

    # The backend to contact for queries/metadata requests
    # Choices are ‘beeswax’ (default), ‘hiveserver2′.
    ## server_interface=beeswax

    It looks like you have enabled everything here which you do not require – note the error in the log:

    FATAL beeswax.Server: hive.metastore.local is set to false. The Beeswax internal metastore is not supposed to run.

    Please try again using the settings from my hue.ini and let me know if it works.

    Thanks

    Dave

    Collapse
    #45538

    lou82
    Participant

    Yes – Have looked in the logs and it basically says

    [13/Dec/2013 14:09:30 +0000] beeswax_server DEBUG Hue is not configured to use Beeswaxd, not starting it.
    ^C
    [root@lhn00 ~]# tail -f /var/log/hue/beeswax_server.
    beeswax_server.log beeswax_server.out
    [root@lhn00 ~]# tail -f /var/log/hue/beeswax_server.
    beeswax_server.log beeswax_server.out
    [root@lhn00 ~]# tail -f /var/log/hue/beeswax_server.out
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1679)
    … 12 more
    13/12/13 13:49:16 INFO beeswax.Server: Starting metastore at port 8003
    13/12/13 13:49:16 INFO beeswax.Server: Starting beeswaxd at port 10000
    13/12/13 13:49:16 INFO beeswax.Server: Parsed core-default.xml sucessfully. Learned 134 descriptions.
    13/12/13 13:49:16 INFO beeswax.Server: Parsed hdfs-default.xml sucessfully. Learned 137 descriptions.
    13/12/13 13:49:16 INFO beeswax.Server: Parsed mapred-default.xml sucessfully. Learned 169 descriptions.
    13/12/13 13:49:16 WARN beeswax.Server: Could not parse or find: hive-default.xml. Learned 0 description, this is not a problem.
    13/12/13 13:49:16 INFO beeswax.Server: Starting beeswax server on port 10000, talking back to Desktop at 127.0.0.1:8000
    13/12/13 13:49:16 FATAL beeswax.Server: hive.metastore.local is set to false. The Beeswax internal metastore is not supposed to run.

    this is my hue.ini section on hive – do you see any issues? Thanks
    [beeswax]

    # Host where Beeswax server Thrift daemon is running.
    # If Kerberos security is enabled, the fully-qualified domain name (FQDN) is
    # required, even if the Thrift daemon is running on the same host as Hue.
    beeswax_server_host=server1.mycompany.local

    # Port where Beeswax Thrift server runs on.
    beeswax_server_port=10000

    # Host where internal metastore Thrift daemon is running.
    beeswax_meta_server_host=server1.mycompany.local

    # Configure the port the internal metastore daemon runs on.
    # Used only if hive.metastore.local is true.
    beeswax_meta_server_port=10000

    # Hive home directory
    hive_home_dir=/usr/lib/hive

    # Hive configuration directory, where hive-site.xml is located
    hive_conf_dir=/etc/hive/conf

    # Timeout in seconds for thrift calls to beeswax service
    beeswax_server_conn_timeout=120

    # Timeout in seconds for thrift calls to the hive metastore
    metastore_conn_timeout=10

    # Maximum Java heapsize (in megabytes) used by Beeswax Server.
    # Note that the setting of HADOOP_HEAPSIZE in $HADOOP_CONF_DIR/hadoop-env.sh
    # may override this setting.
    ## beeswax_server_heapsize=1000

    # Share saved queries with all users. If set to false, saved queries are
    # visible only to the owner and administrators.
    ## share_saved_queries=true

    # The backend to contact for queries/metadata requests
    # Choices are ‘beeswax’ (default), ‘hiveserver2′.
    server_interface=hiveserver2

    Collapse
    #45533

    Dave
    Moderator

    Hi Lou,

    You should look at the location /var/log/hue/beeswax_server.out & .log to see what errors you can see in there.

    Thanks

    Dave

    Collapse
Viewing 18 replies - 1 through 18 (of 18 total)