Home Forums HDP on Linux – Installation Errors while trying to install HDP on IBM PowerLinux (ppc64)

This topic contains 12 replies, has 3 voices, and was last updated by  Larry Liu 1 year, 6 months ago.

  • Creator
    Topic
  • #17218

    Naga Raja
    Member

    Hi,
    I have this instance on a IBM powerlinux running Redhat 6. I am trying to install HDP 1.2. I am running into bunch of
    problems.
    Note: Because I am using PowerLinux, I could install only the IBM java, the details as follows.

    java version “1.6.0″
    Java(TM) SE Runtime Environment (build pxp6460sr13ifix-20130303_02(SR13+IV37419))
    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux ppc64-64 jvmxp6460sr13-20130114_134867 (JIT enabled, AOT enabled)
    J9VM – 20130114_134867
    JIT – r9_20130108_31100
    GC – 20121212_AA)
    JCL – 20130303_02

    ===========
    The problems that I am have are

    1) When I look at the jobtracker logs, I see this

    54310 failed on connection exception: java.net.ConnectException: Connection refused
    2013-03-12 17:09:31,476 INFO org.apache.hadoop.mapred.JobTracker: Problem connecting to HDFS Namenode… re-trying
    java.net.ConnectException: Call to 172-29-171-181.dal-ebis.ihost.com/172.29.171.181:54310 failed on connection exception: java.net.ConnectException: Connection refused
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:1136)
    at org.apache.hadoop.ipc.Client.call(Client.java:1112)

    2) On the tasktracker logs file, I have see this error

    2013-03-12 17:11:28,907 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory…
    2013-03-12 17:11:30,415 INFO org.apache.hadoop.mapred.TaskTracker: Failed to get system directory…
    2013-03-12 17:11:33,811 ERROR org.apache.hadoop.mapred.TaskTracker: Caught exception: java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.mapred.$Proxy2.getSystemDir(Unknown Source)
    at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1775)

Viewing 12 replies - 1 through 12 (of 12 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #17451

    Larry Liu
    Moderator

    HI, Naga

    Thanks for letting us know.

    Larry

    Collapse
    #17329

    Naga Raja
    Member

    Hi Ted,
    Thanks for all your help, I did edit the /etc/hosts and everything started working like a charm. Thanks so much for your help.

    Collapse
    #17282

    tedr
    Member

    Hi Naga,

    No, you need to add a line in your /etc/hosts file linking the hostname of you box with the ip of your box, and then enter that hostname in the configuration files. Also this name in the configuration files need to be the hostname returned by “hostname -f”

    Thanks,
    Ted.

    Collapse
    #17281

    Naga Raja
    Member

    Hi Ted,
    Thanks much, I looked at the datanode logs, I see the error as show below.
    so in the mapred-site.xml and core-site.xml, if I set the parameters to “localhost” instead of ip address, would it work?.

    013-03-13 13:20:46,878 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2013-03-13 13:20:46,888 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
    2013-03-13 13:20:46,903 ERROR org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Error getting localhost name. Using ‘localhost’…
    java.net.UnknownHostException: 172-29-171-181.dal-ebis.ihost.com: 172-29-171-181.dal-ebis.ihost.com
    at java.net.InetAddress.getLocalHost(InetAddress.java:1532)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.getHostname(MetricsSystemImpl.java:463)

    Collapse
    #17280

    tedr
    Member

    Hi Naga

    The error in the JobTracker log indicates the the datanode is not running? Look in the datanode logs to see what could be causing it to die.

    Thanks,
    Ted.

    Collapse
    #17277

    Naga Raja
    Member

    Filesystem formatting logs
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = java.net.UnknownHostException: 172-29-171-181.dal-ebis.ihost.com: 172-29-171-181.dal-ebis.ihost.com
    STARTUP_MSG: args = [-format]
    STARTUP_MSG: version = 1.1.2.21
    STARTUP_MSG: build = git://localhost.localdomain/ on branch bimota-1.1.2 -r d729878637b5a39cadb8527841c102b5b8feced2; compiled by ‘jenkins’ on Thu Jan 10 02:38:03 PST 2013
    ************************************************************/
    Re-format filesystem in /home/hdfs/hadoop/tmp/dfs/name ? (Y or N) Y
    13/03/13 12:34:42 INFO util.GSet: VM type = 64-bit
    13/03/13 12:34:42 INFO util.GSet: 2% max memory = 20.0 MB
    13/03/13 12:34:42 INFO util.GSet: capacity = 2^21 = 2097152 entries
    13/03/13 12:34:42 INFO util.GSet: recommended=2097152, actual=2097152
    13/03/13 12:34:42 INFO namenode.FSNamesystem: fsOwner=hdfs
    13/03/13 12:34:42 INFO namenode.FSNamesystem: supergroup=supergroup
    13/03/13 12:34:42 INFO namenode.FSNamesystem: isPermissionEnabled=true
    13/03/13 12:34:42 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
    13/03/13 12:34:42 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
    13/03/13 12:34:42 INFO namenode.NameNode: Caching file names occuring more than 10 times
    13/03/13 12:34:42 INFO common.Storage: Image file of size 110 saved in 0 seconds.
    13/03/13 12:34:43 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/hdfs/hadoop/tmp/dfs/name/current/edits
    13/03/13 12:34:43 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/hdfs/hadoop/tmp/dfs/name/current/edits
    13/03/13 12:34:43 INFO common.Storage: Storage directory /home/hdfs/hadoop/tmp/dfs/name has been successfully formatted.
    13/03/13 12:34:43 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: 172-29-171-181.dal-ebis.ihost.com: 172-29-171-181.dal-ebis.ihost.com
    ************************************************************/

    Collapse
    #17273

    Naga Raja
    Member

    From jobtracker logs (part IV)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)

    2013-03-13 12:35:59,407 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for null bad datanode[0] nodes == null
    2013-03-13 12:35:59,407 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file “/home/hdfs/hadoop/tmp/mapred/system/jobtracker.info” – Aborting…
    2013-03-13 12:35:59,407 WARN org.apache.hadoop.mapred.JobTracker: Writing to file hdfs://172.29.171.181:54310/home/hdfs/hadoop/tmp/mapred/system/jobtracker.info failed!
    2013-03-13 12:35:59,408 WARN org.apache.hadoop.mapred.JobTracker: FileSystem is not ready yet!
    2013-03-13 12:35:59,414 WARN org.apache.hadoop.mapred.JobTracker: Failed to initialize recovery manager.
    org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hdfs/hadoop/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1637)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:757)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(AccessController.java:310)
    at javax.security.auth.Subject.doAs(Subject.java:573)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

    Collapse
    #17272

    Naga Raja
    Member

    from jobtracker logs (part III)
    2013-03-13 12:35:59,159 INFO org.apache.hadoop.mapred.JobTracker: Setting safe mode to true. Requested by : hdfs
    2013-03-13 12:35:59,301 INFO org.apache.hadoop.mapred.JobTracker: Setting safe mode to false. Requested by : hdfs
    2013-03-13 12:35:59,339 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory
    2013-03-13 12:35:59,366 INFO org.apache.hadoop.mapred.JobTracker: History server being initialized in embedded mode
    2013-03-13 12:35:59,370 INFO org.apache.hadoop.mapred.JobHistoryServer: Started job history server at: localhost:50030
    2013-03-13 12:35:59,370 INFO org.apache.hadoop.mapred.JobTracker: Job History Server web address: localhost:50030
    2013-03-13 12:35:59,372 INFO org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is inactive
    2013-03-13 12:35:59,407 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hdfs/hadoop/tmp/mapred/system/jobtracker.info
    could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1637)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:757)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
    at java.security.AccessController.doPrivileged(AccessController.java:310)
    at javax.security.auth.Subject.doAs(Subject.java:573)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

    at org.apache.hadoop.ipc.Client.call(Client.java:1107)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
    at com.sun.proxy.$Proxy4.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
    at com.sun.proxy.$Proxy4.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)

    Collapse
    #17270

    Naga Raja
    Member

    From Jobtracker logs (part II)
    2013-03-13 12:35:58,721 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as hdfs
    2013-03-13 12:35:58,744 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
    2013-03-13 12:35:58,746 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort54311 registered.
    2013-03-13 12:35:58,747 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort54311 registered.
    2013-03-13 12:35:58,794 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2013-03-13 12:35:58,829 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2013-03-13 12:35:58,832 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
    2013-03-13 12:35:58,834 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
    2013-03-13 12:35:58,834 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
    2013-03-13 12:35:58,834 INFO org.mortbay.log: jetty-6.1.26
    2013-03-13 12:35:59,128 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
    2013-03-13 12:35:59,134 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
    2013-03-13 12:35:59,135 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered.
    2013-03-13 12:35:59,145 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 54311
    2013-03-13 12:35:59,145 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
    2013-03-13 12:35:59,157 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54311: starting
    2013-03-13 12:35:59,157 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
    2013-03-13 12:35:59,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 54311: starting
    2013-03-13 12:35:59,157 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 54311: starting
    2013-03-13 12:35:59,158 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 54311: starting
    2013-03-13 12:35:59,159 INFO

    Collapse
    #17269

    Naga Raja
    Member

    Hi Ted,
    Just now, I formatted the filesystem and started, I see different errors now, atleast I do not see any connection refused. Seems to me like more of filesystem issue now.

    2013-03-13 12:35:58,140 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2013-03-13 12:35:58,149 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
    2013-03-13 12:35:58,163 ERROR org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Error getting localhost name. Using ‘localhost’…
    java.net.UnknownHostException: 172-29-171-181.dal-ebis.ihost.com: 172-29-171-181.dal-ebis.ihost.com
    at java.net.InetAddress.getLocalHost(InetAddress.java:1532)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.getHostname(MetricsSystemImpl.java:463)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSystem(MetricsSystemImpl.java:394)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:390)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:152)
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:133)
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:311)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:306)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:301)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4815)
    2013-03-13 12:35:58,165 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
    2013-03-13 12:35:58,165 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
    2013-03-13 12:35:58,244 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source QueueMetrics,q=default registered.
    2013-03-13 12:35:58,708 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
    2013-03-13 12:35:58,709 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
    2013-03-13 12:35:58,710 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
    2013-03-13 12:35:58,711 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
    2013-03-13 12:35:58,711 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1
    , -1)
    2013-03-13 12:35:58,712 INFO org.apache.hadoop.util.HostsFileReader: Refreshing

    Collapse
    #17253

    Naga Raja
    Member

    Hi Ted,
    Thanks for responding, yes, I have turned off iptables and SELinux.

    [Host ~]$ cat /etc/selinux/config

    # This file controls the state of SELinux on the system.
    # SELINUX= can take one of these three values:
    # enforcing – SELinux security policy is enforced.
    # permissive – SELinux prints warnings instead of enforcing.
    # disabled – No SELinux policy is loaded.
    SELINUX=disabled
    # SELINUXTYPE= can take one of these two values:
    # targeted – Targeted processes are protected,
    # mls – Multi Level Security protection.
    SELINUXTYPE=targeted

    =======

    [u0012225@172-29-171-181 ~]$ sudo /etc/init.d/iptables status
    Table: filter
    Chain INPUT (policy ACCEPT)
    num target prot opt source destination

    Chain FORWARD (policy ACCEPT)
    num target prot opt source destination

    Chain OUTPUT (policy ACCEPT)
    num target prot opt source destination

    Collapse
    #17240

    tedr
    Member

    Hi Naga,

    Thanks for trying Hortonworks Data Platform.
    Have you turned off the firewall (iptables) and disabled SELinux?

    Thanks,
    Ted.

    Collapse
Viewing 12 replies - 1 through 12 (of 12 total)