The Hortonworks Community Connection is now live. A completely rebuilt Q&A forum, Knowledge Base, Code Hub and more, backed by the experts in the industry.

You will be redirected here in 10 seconds. If your are not redirected, click here to visit the new site.

The legacy Hortonworks Forum is now closed. You can view a read-only version of the former site by clicking here. The site will be taken offline on January 31,2016

Hive / HCatalog Forum

Hive: unable to create thread – HDP2.1

  • #53552

    After upgrading to HDP2.1, I’m now seeing the hiveserver2 thread count continually increase over time as we use it (a periodic cron job that is loading data; it’s running & then exiting.)

    We ended up hitting the default(?) centos6.5 limit of 1024 processes/user for hive, at which point hive started throwing:
    java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(
    at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(
    at org.apache.hive.service.cli.session.HiveSessionProxy$
    at Method)

    The same cron process was running the same way against the HDP2.0 cluster, and the thread count in hiveserver2 never budged. (We’ve got graphs!)

    This appears to be a HDP2.1 regression. Anyone have any tips on troubleshooting (what are these threads doing?)?

  • Author
  • #53672
    Vaibhav Gumashta

    Hi Steve,

    In HiveServer2, we use the config hive.server2.thrift.max.worker.threads (default = 500) to set the max number of handler threads and hive.server2.async.exec.threads (default = 100), to set the number of async (background) threads. What are your config values for these two parameters? Can you also take a jstack for the HiveServer2 process when the issue occurs and attach the results here?



    Thanks, that helped (at least get to the next thing).

    Both values are at their defaults:
    hive.server2.async.exec.threads = 100
    hive.server2.thrift.max.worker.threads = 500

    Jstack, at this point (thread count has been climbing steadily over past ~48 hours) shows 546 threads of “ATS Logger 0”. (All other named threads are only running one copy).

    Example entry (they all basically look the same):
    “ATS Logger 0” daemon prio=10 tid=0x00007f6e11a65800 nid=0x35a4 waiting on condition [0x00007f6dd994b000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    – parking to wait for <0x00000000ee0fcac8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
    at java.util.concurrent.LinkedBlockingQueue.take(
    at java.util.concurrent.ThreadPoolExecutor.getTask(
    at java.util.concurrent.ThreadPoolExecutor$

    Also, I do see entries like the following in the hiveserver2.log:
    2014-05-14 02:10:23,699 INFO [ATS Logger 0]: hooks.ATSHook ( – Failed to submit plan to ATS: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.hooks.ATSHook$
    at java.util.concurrent.Executors$
    at java.util.concurrent.FutureTask$Sync.innerRun(
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
    at java.util.concurrent.ThreadPoolExecutor$

    Also …

    2014-05-14 00:10:25,367 INFO [pool-5-thread-5]: impl.TimelineClientImpl ( – Timeline service address: http://{resourcemanager}:8188/ws/v1/timeline/

    However – nothing is listening on resourcemanager:8188

    This config comes from the “Custom yarn-site.xml” specified here:

    The cluster is managed via Ambari (v1.5.1.110), and according to the Ambari console everything is running – but apparently it’s not starting the ATS/history(?) server.

    Is Ambari expected to start the history server?

    Thejas Nair

    Looks like an issue with ATS system integration for reporting hive query progress.
    You can disable that by setting following configurations to empty

    Thanks for reporting this. We are looking into the issue.


    Thanks – that workaround fixed it. If you can, let me know when it’s fixed and/or a bug to track.

    As a side note, Ambari wouldn’t actually let me set this to empty (complained that “it must be set” – this may be an ambari bug) so I set it to a comma (,) instead (because the comma is a split character between the jar names, and nothing’s on either side of the comma, it seems to be treating it as equivalent to ’empty).

    Yang Yang

    we are also seeing the ATS problem.

    2014-07-08 12:40:46,144 INFO hooks.ATSHook ( – Failed to submit plan to ATS: com.sun.jersey.api.client.ClientHandlerException:
    xception: Connection refused
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
    at com.sun.jersey.api.client.Client.handle(
    at com.sun.jersey.api.client.WebResource.handle(
    at com.sun.jersey.api.client.WebResource.access$200(
    at com.sun.jersey.api.client.WebResource$
    at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPostingEntities(
    at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(
    at org.apache.hadoop.hive.ql.hooks.ATSHook.fireAndForget(
    at org.apache.hadoop.hive.ql.hooks.ATSHook$
    at java.util.concurrent.Executors$
    at java.util.concurrent.ThreadPoolExecutor.runWorker(
    at java.util.concurrent.ThreadPoolExecutor$
    Caused by: Connection refused
    at Method)

    Yang Yang

    we added

    -hiveconf hive.exec.pre.hooks=” \
    -hiveconf” \
    -hiveconf hive.exec.failure.hooks=” \

    to hive command line, and added

    set hive.exec.pre.hooks=”;
    set hive.exec.failure.hooks=”;

    to the front of the hive script, but still got the same problems

    Noam Cohen

    I encountered this problem as well. Just wondering – do you have a bug opened for it? Is it solved in HDP2.1.3?

The forum ‘Hive / HCatalog’ is closed to new topics and replies.

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.