Home Forums HDP on Windows – Installation DistributedCache problem

This topic contains 0 replies, has 1 voice, and was last updated by  Subroto Sanyal 1 year, 6 months ago.

  • Creator
    Topic
  • #27377

    Subroto Sanyal
    Participant

    Hi,
    I have one node cluster(Windows) on which hdp-1.1.0-GA is installed. The Services of NameNode, DataNode, JobTracker and TaskTracker are running fine.
    I have a JobClient running on my MacBook with user “subroto”.
    The jobs fail because of missing Classes specified by Distributed Cache
    When a Job is submitted to the cluster; the distributed cache jars are downloaded on the Node at:
    c:/hdp/data/hdfs/mapred/local/taskTracker/distcache/5532737485201049015_1549685754_1016958893/WIN-AUJQ7LN3IAO/user/subroto/********
    I can see the jar files also at these locations.
    When I open the taskjvm.cmd I can see:

    set HADOOP_CLIENT_OPTS=”-Dhadoop.tasklog.taskid=attempt_201306130935_0012_m_000002_0 -Dhadoop.tasklog.iscleanup=false -Dhadoop.tasklog.totalLogFileSize=0″
    set SHELL=”cmd”
    set HADOOP_WORK_DIR=”c:\hdp\data\hdfs\mapred\local\taskTracker\subroto\jobcache\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\work”
    set HOME=”C:\Users\hadoop”
    set LOGNAME=”subroto”
    set HADOOP_TOKEN_FILE_LOCATION=”c:/hdp/data/hdfs/mapred/local/taskTracker/subroto/jobcache/job_201306130935_0012/jobToken”
    set HADOOP_ROOT_LOGGER=”INFO,TLA”
    set HADOOP_HOME=”C:\hdp\hadoop\hadoop-1.1.0-SNAPSHOT”
    set LD_LIBRARY_PATH=”c:\hdp\data\hdfs\mapred\local\taskTracker\subroto\jobcache\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\work”
    set USER=”subroto”
    C:\Java\jdk1.6.0_45\jre\bin\java “-classpath” “C:\hdp\data\hdfs\mapred\local\taskTracker\subroto\jobcache\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\classpath-430686823459265078.jar” “-Xmx200m” “-Djava.net.preferIPv4Stack=true” “-Dhadoop.metrics.log.level=WARN” “-Djava.library.path=;C:\hdp\hadoop\hadoop-1.1.0-SNAPSHOT\lib\native\Windows_NT-amd64-64;c:\hdp\data\hdfs\mapred\local\taskTracker\subroto\jobcache\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\work” “-Djava.io.tmpdir=c:/hdp/data/hdfs/tmp” “-Dhadoop.log.dir=c:\hadoop\logs\hadoop” “-Dhadoop.root.logger=INFO,TLA” “-Dhadoop.tasklog.taskid=attempt_201306130935_0012_m_000002_0″ “-Dhadoop.tasklog.iscleanup=false” “-Dhadoop.tasklog.totalLogFileSize=0″ “org.apache.hadoop.mapred.Child” “127.0.0.1” “49891” “attempt_201306130935_0012_m_000002_0″ “c:\hadoop\logs\hadoop\userlogs\job_201306130935_0012\attempt_201306130935_0012_m_000002_0″ “2092027230” > c:\hadoop\logs\hadoop\userlogs\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\stdout 2>> c:\hadoop\logs\hadoop\userlogs\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\stderr

    I think the distributed cache files are getting cached to wrong folder location.

    Please let me know if this is expected behavior?

You must be logged in to reply to this topic.