MapReduce Forum

Cannot see MR logs in Hadoop 2.x

  • #46485
    Kyle Lin

    My Environment: HDP 2 (Hadoop 2.2.0).

    When using Hadoop 1.x, I write log imformation in my MapReduce code, then see the log message in TaskTracker log. But In YARN, I cannot find the log message in any log files(historyserver, nodemanager, resourcemanager), even using System.out.println.

    I also try to see app log through the address of yarn.resourcemanager.webapp.address. Still cannot see my log messages.

    Where can I find the log files with my log message?

to create new topics or reply. | New User Registration

  • Author
  • #46486
    Kyle Lin

    Sorry for my rough asking. Refine my question.

    When using hadoop 1.x, I can see the MR job logs under “userlogs” folder in local file system. But I cannot find the MR log in hadoop 2.x environment. There is no “userlogs” folder.

    Can anybody tell me where to find the log files?

    D Blair Elzinga

    You *should* be able to go the history server web page and view the jobs: <historyserver>:19888/jobhistory.

    Or from the resource manager <resource manager host>:8088/cluster to view all applications, then click on the ID link, then on “logs” link.

    Anand M

    A MR job has a unique application id in the format – “application_<application id>_<job sequence no>”

    After the completion of your job, execute yarn command as below to get your logs.

    yarn logs “application_<application id>_<job sequence no>” > /store_it_somewhere_in_your_local_logs_path

    Please read this link for details.


You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.