Home Forums MapReduce Cannot see MR logs in Hadoop 2.x

This topic contains 3 replies, has 3 voices, and was last updated by  Anand M 5 months ago.

  • Creator
    Topic
  • #46485

    Kyle Lin
    Member

    My Environment: HDP 2 (Hadoop 2.2.0).

    When using Hadoop 1.x, I write log imformation in my MapReduce code, then see the log message in TaskTracker log. But In YARN, I cannot find the log message in any log files(historyserver, nodemanager, resourcemanager), even using System.out.println.

    I also try to see app log through the address of yarn.resourcemanager.webapp.address. Still cannot see my log messages.

    Where can I find the log files with my log message?

Viewing 3 replies - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #54415

    Anand M
    Participant

    A MR job has a unique application id in the format – “application_<application id>_<job sequence no>”

    After the completion of your job, execute yarn command as below to get your logs.

    yarn logs “application_<application id>_<job sequence no>” > /store_it_somewhere_in_your_local_logs_path

    Please read this link for details.

    http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/

    Thanks
    -Anand

    Collapse
    #49574

    D Blair Elzinga
    Participant

    You *should* be able to go the history server web page and view the jobs: <historyserver>:19888/jobhistory.

    Or from the resource manager <resource manager host>:8088/cluster to view all applications, then click on the ID link, then on “logs” link.

    Collapse
    #46486

    Kyle Lin
    Member

    Sorry for my rough asking. Refine my question.

    When using hadoop 1.x, I can see the MR job logs under “userlogs” folder in local file system. But I cannot find the MR log in hadoop 2.x environment. There is no “userlogs” folder.

    Can anybody tell me where to find the log files?

    Collapse
Viewing 3 replies - 1 through 3 (of 3 total)