Home Forums HDP on Linux – Installation Hdfs command goes to cluster, Pig command goes to local filesystem ??

Tagged: 

This topic contains 1 reply, has 1 voice, and was last updated by  Niels Basjes 3 months, 2 weeks ago.

  • Creator
    Topic
  • #53725

    Niels Basjes
    Participant

    Hi,

    I have just installed HDP 2.1.2 on my systems and setup the environment almost the same as what I had with CDH 5.0.0.
    I.e. environment setting like $HADOOP_CONF_DIR and $HADOOP_MAPRED_HOME have been setup right (as far as I can tell)
    So if I do:
    hdfs dfs -ls -R /
    I see the files that are on my HDFS.

    I do this:
    $ pig
    2014-05-15 10:41:55,810 [main] INFO org.apache.pig.Main – Apache Pig version 0.12.1.2.1.2.0-402 (rexported) compiled Apr 27 2014, 16:49:54

    2014-05-15 10:41:56,284 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to hadoop file system at: file:///

    When I do this

    $ pig -printCmdDebug
    Find hadoop at /usr/bin/hadoop
    dry run:
    HADOOP_CLASSPATH: /home/nbasjes/hadoop-environment-configs/kluster-hdp21:/usr/lib/pig/bin/../conf:/usr/java/default/lib/tools.jar:/etc/hadoop/conf:/usr/lib/pig/bin/../lib/avro-1.7.4.jar:/usr/lib/pig/bin/../lib/avro-ipc-1.7.4-tests.jar:/usr/lib/pig/bin/../lib/avro-mapred-1.7.4.jar:/usr/lib/pig/bin/../lib/jruby-complete-1.6.7.jar:/usr/lib/pig/bin/../lib/json-simple-1.1.jar:/usr/lib/pig/bin/../lib/jython-standalone-2.5.3.jar:/usr/lib/pig/bin/../lib/piggybank.jar:/usr/lib/pig/bin/../pig-0.12.1.2.1.2.0-402-withouthadoop.jar:
    HADOOP_CLIENT_OPTS: -Xmx1000m -Dpig.log.dir=/usr/lib/pig/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/usr/lib/pig/bin/..
    /usr/bin/hadoop jar /usr/lib/pig/bin/../pig-0.12.1.2.1.2.0-402-withouthadoop.jar

    Ok, so I manually set these two and run the command at the bottom.
    $ /usr/bin/hadoop jar /usr/lib/pig/bin/../pig-0.12.1.2.1.2.0-402-withouthadoop.jar

    2014-05-15 10:36:00,329 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine – Connecting to hadoop file system at: hdfs://node1.kluster.basjes.lan/

    What am I doing wrong???
    Why doesn’t pig go to my HDFS when running the normal script??

Viewing 1 replies (of 1 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #53729

    Niels Basjes
    Participant

    The best guess I have right now is that BOTH my HADOOP_CONF_DIR and /etc/hadoop/conf are in the classpath.
    After I removed all files from /etc/hadoop/conf is suddenly worked.

    Collapse
Viewing 1 replies (of 1 total)