DistributedCache problem

to create new topics or reply. | New User Registration

This topic contains 0 replies, has 1 voice, and was last updated by  Subroto Sanyal 1 year, 11 months ago.

  • Creator
    Topic
  • #27377

    Subroto Sanyal
    Participant

    Hi,
    I have one node cluster(Windows) on which hdp-1.1.0-GA is installed. The Services of NameNode, DataNode, JobTracker and TaskTracker are running fine.
    I have a JobClient running on my MacBook with user “subroto”.
    The jobs fail because of missing Classes specified by Distributed Cache
    When a Job is submitted to the cluster; the distributed cache jars are downloaded on the Node at:
    c:/hdp/data/hdfs/mapred/local/taskTracker/distcache/5532737485201049015_1549685754_1016958893/WIN-AUJQ7LN3IAO/user/subroto/********
    I can see the jar files also at these locations.
    When I open the taskjvm.cmd I can see:

    set HADOOP_CLIENT_OPTS=”-Dhadoop.tasklog.taskid=attempt_201306130935_0012_m_000002_0 -Dhadoop.tasklog.iscleanup=false -Dhadoop.tasklog.totalLogFileSize=0″
    set SHELL=”cmd”
    set HADOOP_WORK_DIR=”c:\hdp\data\hdfs\mapred\local\taskTracker\subroto\jobcache\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\work”
    set HOME=”C:\Users\hadoop”
    set LOGNAME=”subroto”
    set HADOOP_TOKEN_FILE_LOCATION=”c:/hdp/data/hdfs/mapred/local/taskTracker/subroto/jobcache/job_201306130935_0012/jobToken”
    set HADOOP_ROOT_LOGGER=”INFO,TLA”
    set HADOOP_HOME=”C:\hdp\hadoop\hadoop-1.1.0-SNAPSHOT”
    set LD_LIBRARY_PATH=”c:\hdp\data\hdfs\mapred\local\taskTracker\subroto\jobcache\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\work”
    set USER=”subroto”
    C:\Java\jdk1.6.0_45\jre\bin\java “-classpath” “C:\hdp\data\hdfs\mapred\local\taskTracker\subroto\jobcache\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\classpath-430686823459265078.jar” “-Xmx200m” “-Djava.net.preferIPv4Stack=true” “-Dhadoop.metrics.log.level=WARN” “-Djava.library.path=;C:\hdp\hadoop\hadoop-1.1.0-SNAPSHOT\lib\native\Windows_NT-amd64-64;c:\hdp\data\hdfs\mapred\local\taskTracker\subroto\jobcache\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\work” “-Djava.io.tmpdir=c:/hdp/data/hdfs/tmp” “-Dhadoop.log.dir=c:\hadoop\logs\hadoop” “-Dhadoop.root.logger=INFO,TLA” “-Dhadoop.tasklog.taskid=attempt_201306130935_0012_m_000002_0″ “-Dhadoop.tasklog.iscleanup=false” “-Dhadoop.tasklog.totalLogFileSize=0″ “org.apache.hadoop.mapred.Child” “127.0.0.1” “49891” “attempt_201306130935_0012_m_000002_0″ “c:\hadoop\logs\hadoop\userlogs\job_201306130935_0012\attempt_201306130935_0012_m_000002_0″ “2092027230” > c:\hadoop\logs\hadoop\userlogs\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\stdout 2>> c:\hadoop\logs\hadoop\userlogs\job_201306130935_0012\attempt_201306130935_0012_m_000002_0\stderr

    I think the distributed cache files are getting cached to wrong folder location.

    Please let me know if this is expected behavior?

You must be to reply to this topic. | Create Account

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.