HDP on Linux – Installation Forum

Errors while trying to install HDP on IBM PowerLinux (ppc64)

  • #17218
    Naga Raja
    Member

    Hi,
    I have this instance on a IBM powerlinux running Redhat 6. I am trying to install HDP 1.2. I am running into bunch of
    problems.
    Note: Because I am using PowerLinux, I could install only the IBM java, the details as follows.

    java version “1.6.0”
    Java(TM) SE Runtime Environment (build pxp6460sr13ifix-20130303_02(SR13+IV37419))
    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux ppc64-64 jvmxp6460sr13-20130114_134867 (JIT enabled, AOT enabled)
    J9VM – 20130114_134867
    JIT – r9_20130108_31100
    GC – 20121212_AA)
    JCL – 20130303_02

    ===========
    The problems that I am have are

    1) When I look at the jobtracker logs, I see this

    54310 failed on connection exception: java.net.ConnectException: Connection refused
    2013-03-12 17:09:31,476 INFO org.apache.hadoop.mapred.JobTracker: Problem connecting to HDFS Namenode… re-trying
    java.net.ConnectException: Call to 172-29-171-181.dal-ebis.ihost.com/172.29.171.181:54310 failed on connection exception: java.net.ConnectException: Connection refused
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:1136)
    at org.apache.hadoop.ipc.Client.call(Client.java:1112)

    2) On the tasktracker logs file, I have see this error

    2013-03-12 17:11:28,907 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory…
    2013-03-12 17:11:30,415 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory…
    2013-03-12 17:11:33,811 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.mapred.$Proxy2.getSystemDir(Unknown Source)
    at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1775)

to create new topics or reply. | New User Registration

  • Author
    Replies
  • #17240
    tedr
    Member

    Hi Naga,

    Thanks for trying Hortonworks Data Platform.
    Have you turned off the firewall (iptables) and disabled SELinux?

    Thanks,
    Ted.

    #17253
    Naga Raja
    Member

    Hi Ted,
    Thanks for responding, yes, I have turned off iptables and SELinux.

    [Host ~]$ cat /etc/selinux/config

    # This file controls the state of SELinux on the system.
    # SELINUX= can take one of these three values:
    # enforcing – SELinux security policy is enforced.
    # permissive – SELinux prints warnings instead of enforcing.
    # disabled – No SELinux policy is loaded.
    SELINUX=disabled
    # SELINUXTYPE= can take one of these two values:
    # targeted – Targeted processes are protected,
    # mls – Multi Level Security protection.
    SELINUXTYPE=targeted

    =======

    [u0012225@172-29-171-181 ~]$ sudo /etc/init.d/iptables status
    Table: filter
    Chain INPUT (policy ACCEPT)
    num target prot opt source destination

    Chain FORWARD (policy ACCEPT)
    num target prot opt source destination

    Chain OUTPUT (policy ACCEPT)
    num target prot opt source destination

    #17269
    Naga Raja
    Member

    Hi Ted,
    Just now, I formatted the filesystem and started, I see different errors now, atleast I do not see any connection refused. Seems to me like more of filesystem issue now.

    2013-03-13 12:35:58,140 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2013-03-13 12:35:58,149 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
    2013-03-13 12:35:58,163 ERROR org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Error getting localhost name. Using ‘localhost’…
    java.net.UnknownHostException: 172-29-171-181.dal-ebis.ihost.com: 172-29-171-181.dal-ebis.ihost.com
    at java.net.InetAddress.getLocalHost(InetAddress.java:1532)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.getHostname(MetricsSystemImpl.java:463)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSystem(MetricsSystemImpl.java:394)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:390)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:152)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:133)
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:311)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:306)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:301)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4815)
    2013-03-13 12:35:58,165 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
    2013-03-13 12:35:58,165 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
    2013-03-13 12:35:58,244 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source QueueMetrics,q=default registered.
    2013-03-13 12:35:58,708 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
    2013-03-13 12:35:58,709 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
    2013-03-13 12:35:58,710 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
    2013-03-13 12:35:58,711 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
    2013-03-13 12:35:58,711 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1
    , -1)
    2013-03-13 12:35:58,712 INFO org.apache.hadoop.util.HostsFileReader: Refreshing

    #17270
    Naga Raja
    Member

    From Jobtracker logs (part II)
    2013-03-13 12:35:58,721 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as hdfs
    2013-03-13 12:35:58,744 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
    2013-03-13 12:35:58,746 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort54311 registered.
    2013-03-13 12:35:58,747 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort54311 registered.
    2013-03-13 12:35:58,794 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2013-03-13 12:35:58,829 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2013-03-13 12:35:58,832 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
    2013-03-13 12:35:58,834 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
    2013-03-13 12:35:58,834 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
    2013-03-13 12:35:58,834 INFO org.mortbay.log: jetty-6.1.26
    2013-03-13 12:35:59,128 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
    2013-03-13 12:35:59,134 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
    2013-03-13 12:35:59,135 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered.
    2013-03-13 12:35:59,145 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 54311
    2013-03-13 12:35:59,145 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
    2013-03-13 12:35:59,157 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54311: starting
    2013-03-13 12:35:59,157 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
    2013-03-13 12:35:59,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 54311: starting
    2013-03-13 12:35:59,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 54311: starting
    2013-03-13 12:35:59,159 INFO

    #17272
    Naga Raja
    Member

    from jobtracker logs (part III)
    2013-03-13 12:35:59,159 INFO org.apache.hadoop.mapred.JobTracker: Setting safe mode to true. Requested by : hdfs
    2013-03-13 12:35:59,301 INFO org.apache.hadoop.mapred.JobTracker: Setting safe mode to false. Requested by : hdfs
    2013-03-13 12:35:59,339 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory
    2013-03-13 12:35:59,366 INFO org.apache.hadoop.mapred.JobTracker: History server being initialized in embedded mode
    2013-03-13 12:35:59,370 INFO org.apache.hadoop.mapred.JobHistoryServer: Started job history server at: localhost:50030
    2013-03-13 12:35:59,370 INFO org.apache.hadoop.mapred.JobTracker: Job History Server web address: localhost:50030
    2013-03-13 12:35:59,372 INFO org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is inactive
    2013-03-13 12:35:59,407 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hdfs/hadoop/tmp/mapred/system/jobtracker.info
    could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1637)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:757)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(AccessController.java:310)
    at javax.security.auth.Subject.doAs(Subject.java:573)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

    at org.apache.hadoop.ipc.Client.call(Client.java:1107)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
    at com.sun.proxy.$Proxy4.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
    at com.sun.proxy.$Proxy4.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)

    #17273
    Naga Raja
    Member

    From jobtracker logs (part IV)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)

    2013-03-13 12:35:59,407 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null
    2013-03-13 12:35:59,407 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file “/home/hdfs/hadoop/tmp/mapred/system/jobtracker.info” – Aborting…
    2013-03-13 12:35:59,407 WARN org.apache.hadoop.mapred.JobTracker: Writing to file hdfs://172.29.171.181:54310/home/hdfs/hadoop/tmp/mapred/system/jobtracker.info failed!
    2013-03-13 12:35:59,408 WARN org.apache.hadoop.mapred.JobTracker: FileSystem is not ready yet!
    2013-03-13 12:35:59,414 WARN org.apache.hadoop.mapred.JobTracker: Failed to initialize recovery manager.
    org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hdfs/hadoop/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1637)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:757)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(AccessController.java:310)
    at javax.security.auth.Subject.doAs(Subject.java:573)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

    #17277
    Naga Raja
    Member

    Filesystem formatting logs
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = java.net.UnknownHostException: 172-29-171-181.dal-ebis.ihost.com: 172-29-171-181.dal-ebis.ihost.com
    STARTUP_MSG: args = [-format]
    STARTUP_MSG: version = 1.1.2.21
    STARTUP_MSG: build = git://localhost.localdomain/ on branch bimota-1.1.2 -r d729878637b5a39cadb8527841c102b5b8feced2; compiled by ‘jenkins’ on Thu Jan 10 02:38:03 PST 2013
    ************************************************************/
    Re-format filesystem in /home/hdfs/hadoop/tmp/dfs/name ? (Y or N) Y
    13/03/13 12:34:42 INFO util.GSet: VM type = 64-bit
    13/03/13 12:34:42 INFO util.GSet: 2% max memory = 20.0 MB
    13/03/13 12:34:42 INFO util.GSet: capacity = 2^21 = 2097152 entries
    13/03/13 12:34:42 INFO util.GSet: recommended=2097152, actual=2097152
    13/03/13 12:34:42 INFO namenode.FSNamesystem: fsOwner=hdfs
    13/03/13 12:34:42 INFO namenode.FSNamesystem: supergroup=supergroup
    13/03/13 12:34:42 INFO namenode.FSNamesystem: isPermissionEnabled=true
    13/03/13 12:34:42 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
    13/03/13 12:34:42 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
    13/03/13 12:34:42 INFO namenode.NameNode: Caching file names occuring more than 10 times
    13/03/13 12:34:42 INFO common.Storage: Image file of size 110 saved in 0 seconds.
    13/03/13 12:34:43 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/hdfs/hadoop/tmp/dfs/name/current/edits
    13/03/13 12:34:43 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/hdfs/hadoop/tmp/dfs/name/current/edits
    13/03/13 12:34:43 INFO common.Storage: Storage directory /home/hdfs/hadoop/tmp/dfs/name has been successfully formatted.
    13/03/13 12:34:43 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: 172-29-171-181.dal-ebis.ihost.com: 172-29-171-181.dal-ebis.ihost.com
    ************************************************************/

    #17280
    tedr
    Member

    Hi Naga

    The error in the JobTracker log indicates the the datanode is not running? Look in the datanode logs to see what could be causing it to die.

    Thanks,
    Ted.

    #17281
    Naga Raja
    Member

    Hi Ted,
    Thanks much, I looked at the datanode logs, I see the error as show below.
    so in the mapred-site.xml and core-site.xml, if I set the parameters to “localhost” instead of ip address, would it work?.

    013-03-13 13:20:46,878 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2013-03-13 13:20:46,888 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
    2013-03-13 13:20:46,903 ERROR org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Error getting localhost name. Using ‘localhost’…
    java.net.UnknownHostException: 172-29-171-181.dal-ebis.ihost.com: 172-29-171-181.dal-ebis.ihost.com
    at java.net.InetAddress.getLocalHost(InetAddress.java:1532)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.getHostname(MetricsSystemImpl.java:463)

    #17282
    tedr
    Member

    Hi Naga,

    No, you need to add a line in your /etc/hosts file linking the hostname of you box with the ip of your box, and then enter that hostname in the configuration files. Also this name in the configuration files need to be the hostname returned by “hostname -f”

    Thanks,
    Ted.

    #17329
    Naga Raja
    Member

    Hi Ted,
    Thanks for all your help, I did edit the /etc/hosts and everything started working like a charm. Thanks so much for your help.

    #17451
    Larry Liu
    Moderator

    HI, Naga

    Thanks for letting us know.

    Larry

You must be to reply to this topic. | Create Account

Support from the Experts

A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.

Enterprise Support »

Become HDP Certified

Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world

Training »

Hortonworks Data Platform
The Hortonworks Data Platform is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor.
Get started with Sandbox
Hortonworks Sandbox is a self-contained virtual machine with Apache Hadoop pre-configured alongside a set of hands-on, step-by-step Hadoop tutorials.
Modern Data Architecture
Tackle the challenges of big data. Hadoop integrates with existing EDW, RDBMS and MPP systems to deliver lower cost, higher capacity infrastructure.