Home Forums Hive / HCatalog Nested IF is not working with Hadoop 1.3

This topic contains 1 reply, has 2 voices, and was last updated by  Carter Shanklin 1 year, 2 months ago.

  • Creator
    Topic
  • #31429

    Anil Langote
    Member

    Hi

    We are using Hadoop 1.3 and have hive queries which is using IF statement but we are getting below exception when we remove the IF it works fine.

    Below is the pseudo code which wont work
    SUM(IF(Condition))

    If i remove IF condition it works
    SUM()

    Below is the error we get

    2013-07-15 07:53:30 Starting to launch local task to process map join; maximum memory = 1065484288
    2013-07-15 07:53:31 Processing rows: 703 Hashtable size: 703 Memory usage: 7864352 rate: 0.007
    2013-07-15 07:53:31 Dump the hashtable into file: file:/tmp/web01/hive_2013-07-15_07-53-21_483_1057643048422702299/-local-10003/HashTable-Stage-7/MapJoin-mapfile01–.hashtable
    2013-07-15 07:53:32 Upload 1 File to: file:/tmp/web01/hive_2013-07-15_07-53-21_483_1057643048422702299/-local-10003/HashTable-Stage-7/MapJoin-mapfile01–.hashtable File size: 53692
    2013-07-15 07:53:37 Processing rows: 200000 Hashtable size: 199999 Memory usage: 64081384 rate: 0.06
    2013-07-15 07:53:40 Processing rows: 300000 Hashtable size: 299999 Memory usage: 91730808 rate: 0.086
    2013-07-15 07:53:41 Processing rows: 400000 Hashtable size: 399999 Memory usage: 124973928 rate: 0.117
    2013-07-15 07:53:43 Processing rows: 500000 Hashtable size: 499999 Memory usage: 153908584 rate: 0.144
    2013-07-15 07:53:45 Processing rows: 600000 Hashtable size: 599999 Memory usage: 182857136 rate: 0.172
    2013-07-15 07:53:48 Processing rows: 700000 Hashtable size: 699999 Memory usage: 209202736 rate: 0.196
    2013-07-15 07:53:50 Processing rows: 800000 Hashtable size: 799999 Memory usage: 246410592 rate: 0.231
    2013-07-15 07:53:51 Processing rows: 900000 Hashtable size: 899999 Memory usage: 275452544 rate: 0.259
    2013-07-15 07:53:54 Processing rows: 1000000 Hashtable size: 999999 Memory usage: 304423912 rate: 0.286
    2013-07-15 07:53:55 Processing rows: 1100000 Hashtable size: 1099999 Memory usage: 333256736 rate: 0.313
    2013-07-15 07:53:59 Processing rows: 1200000 Hashtable size: 1199999 Memory usage: 362320352 rate: 0.34
    2013-07-15 07:53:59 Processing rows: 1300000 Hashtable size: 1299999 Memory usage: 390814408 rate: 0.367
    2013-07-15 07:54:00 Processing rows: 1400000 Hashtable size: 1399999 Memory usage: 419855056 rate: 0.394
    2013-07-15 07:54:05 Processing rows: 1500000 Hashtable size: 1499999 Memory usage: 443703056 rate: 0.416
    2013-07-15 07:54:06 Processing rows: 1600000 Hashtable size: 1599999 Memory usage: 489415672 rate: 0.459
    2013-07-15 07:54:06 Processing rows: 1700000 Hashtable size: 1699999 Memory usage: 518033120 rate: 0.486
    2013-07-15 07:54:11 Processing rows: 1800000 Hashtable size: 1799999 Memory usage: 546992064

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #32575

    Carter Shanklin
    Participant

    What you posted is not an error. Hive might only build that hashtable based on what is in the if statement but I can’t tell without some more detail on what you’re doing, can you post a more complete example?

    Collapse
Viewing 1 replies (of 1 total)