Home Forums HDP on Linux – Installation The revisions match but the NN version is 1.2.0.1.3.2.0-110 and the DN version i

This topic contains 17 replies, has 2 voices, and was last updated by  Dave 1 year, 1 month ago.

  • Creator
    Topic
  • #34577

    suaroman
    Participant

    I have an existing Ambari deployed cluster (/w 3 datanodes). Partial version list below

    Cluster Stack Version: HDP-1.3.2
    Service Version Description
    HDFS 1.2.0.1.3.2.0 Apache Hadoop Distributed File System **
    MapReduce 1.2.0.1.3.2.0 Apache Hadoop Distributed Processing Framework
    Nagios 3.5.0 Nagios Monitoring and Alerting system
    Ganglia 3.5.0 Ganglia Metrics Collection system
    . . .
    . . .
    When attempting to provision a new datanode within the current cluster on machine hdpmachine7, it looks like the deployment and everything works fine, but when attempting to start the node, it fails with the following version error:
    2013-09-06 13:39:42,561 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting DataNode
    STARTUP_MSG: host = hdpmachine7/65.53.11.46
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 1.2.0.1.3.2.0-111
    STARTUP_MSG: build = git://c64-s8/ on branch comanche-branch-1 -r 3e43bec958e627d53f02d2842f6fac24a93110a9; compiled by ‘jenkins’ on Mon Aug 19 18:34:32 PDT 2013
    STARTUP_MSG: java = 1.6.0_31
    ************************************************************/
    2013-09-06 13:39:42,685 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
    2013-09-06 13:39:42,700 INFO org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink ganglia started
    2013-09-06 13:39:42,730 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
    2013-09-06 13:39:42,731 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
    2013-09-06 13:39:42,731 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
    2013-09-06 13:39:42,809 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
    2013-09-06 13:39:43,073 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.lang.AssertionError: Invalid build. The revisions match but the NN version is 1.2.0.1.3.2.0-110 and the DN version is 1.2.0.1.3.2.0-111
    at org.apache.hadoop.hdfs.server.datanode.DataNode.isPermittedVersion(DataNode.java:619)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.handshake(DataNode.java:659)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:391)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:321)
    at org.apache.hadoop.hdfs.server.

    All of the other data nodes that were originally created are running: version = 1.2.0.1.3.2.0-110 ; however this new node looks like its runnning version = 1.2.0.1.3.2.0-111.

    The main Ambari machine was not updated since the original deployment so I’m unsure how there could be a version mismatch.

    Is there an easy way to fix this ?
    Any tips greatly appreciated.

Viewing 17 replies - 1 through 17 (of 17 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #35534

    Dave
    Moderator

    Hi Suaroman,

    I’m glad that resolved it!

    Thanks

    Dave

    Collapse
    #35440

    suaroman
    Participant

    Hi Dave

    I ran ‘yum update hadoop’ on all machines in the cluster and everything is working properly now.
    Thanks for the help!

    Collapse
    #35273

    Dave
    Moderator

    Hi Suaroman,

    If your repo is pointing to : http://public-repo-1.hortonworks.com/HDP/centos6/1.x/updates/1.3.2.0

    Then you can do a yum update hadoop

    and then yum update (any other packages you have installed on the machine)

    This will update them to build 111.

    Thanks

    Dave

    Collapse
    #35257

    Dave
    Moderator

    Hi Suaroman,

    Just installed a fresh 1.3.2 stack and it comes out as build 111, so it looks like we changed build (as you said on the 19th of August)

    You could try and “upgrade” your NN or let me know how the jars go.

    Thanks

    Dave

    Collapse
    #35242

    suaroman
    Participant

    Yup, I’ll give this a try and report back

    Collapse
    #35209

    Dave
    Moderator

    Hi Suaroman,

    Thanks for the info, I’ll look into this.

    As a test, can you copy your build 111 Jars into the hadoop directory of the 110 ones (and overwrite them – ie move the 110 and replace with 111) then restart it.

    Does the NN now pick up build 111?

    It’s interesting as my build shows 110, but I’ll download and install another 1.3.2 stack and see what build it gives me.

    Thanks

    Dave

    Collapse
    #35194

    suaroman
    Participant

    One small observation also.
    1. The newly installed datanode (hdpmachine7 ) has two other components running fine (a task tracker instance and ganglia service) .

    2. I installed a client machine (hdpmachine8) about a week ago. This machine is used exclusively to submit jobs to the cluster. I haven’t had any problem with this machine. This machine (hdpmachine8) was also deployed using Ambari and has the following hadoop version info:

    [suaro@hdpmachine8 ~]$ hadoop version
    Hadoop 1.2.0.1.3.2.0-111
    Subversion git://c64-s8/ on branch comanche-branch-1 -r 3e43bec958e627d53f02d2842f6fac24a93110a9
    Compiled by jenkins on Mon Aug 19 18:34:32 PDT 2013
    From source with checksum cf234891d3fd875413caf539bc5aa5ce
    This command was run using /usr/lib/hadoop/hadoop-core-1.2.0.1.3.2.0-111.jar

    Seems like Aug 19 is when the build -111 got into the mix and from that point , any machine deployed using Ambari – received -111 bits. From what I can tell, only the datanode daemon is effected. Everything else seems to be ok.

    Collapse
    #35163

    suaroman
    Participant

    (and again, small correction – hdpmachine1 is the NAMENODE and hdpmachine7 is the newly installed DATANODE )

    Collapse
    #35162

    suaroman
    Participant

    Looks like some of my data got truncated below:

    hdpmachine1 is the datanode:

    ambari=> select * from hostcomponentstate where host_name = ‘hdpmachine1′;
    cluster_id | component_name | current_stack_version | current_state | host_name | service_name
    ————+——————+——————————————–+—————+————-+————–
    2 | HCAT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine1 | HCATALOG
    2 | HIVE_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine1 | HIVE
    2 | NAGIOS_SERVER | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine1 | NAGIOS
    2 | NAMENODE | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine1 | HDFS
    2 | ZOOKEEPER_SERVER | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine1 | ZOOKEEPER
    2 | MAPREDUCE_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine1 | MAPREDUCE
    2 | OOZIE_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine1 | OOZIE
    2 | GANGLIA_MONITOR | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine1 | GANGLIA
    2 | GANGLIA_SERVER | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine1 | GANGLIA
    (9 rows)

    hdpmachine7 is the newly installed node:

    ambari=> select * from hostcomponentstate where host_name = ‘hdpmachine7′;
    cluster_id | component_name | current_stack_version | current_state | host_name | service_name
    ————+—————–+——————————————–+—————+————-+————–
    2 | GANGLIA_MONITOR | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine7 | GANGLIA
    2 | TASKTRACKER | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine7 | MAPREDUCE
    2 | DATANODE | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine7 | HDFS

    Collapse
    #35161

    suaroman
    Participant

    Thanks Dave. Here ya go:

    NAMENODE (hdpmachine1 )
    [root@hdpmachine1 ~]# hadoop version
    Hadoop 1.2.0.1.3.2.0-110
    Subversion git://c64-s8/ on branch comanche-branch-1 -r 3e43bec958e627d53f02d2842f6fac24a93110a9
    Compiled by jenkins on Wed Aug 14 02:53:41 PDT 2013
    From source with checksum cf234891d3fd875413caf539bc5aa5ce
    This command was run using /usr/lib/hadoop/hadoop-core-1.2.0.1.3.2.0-110.jar
    [root@hdpmachine1 ~]#

    New DATANODE Installed using Ambari Provisioning (hdpmachine7)
    ——————————————————————
    [root@hdpmachine7 ~]# hadoop version
    Hadoop 1.2.0.1.3.2.0-111
    Subversion git://c64-s8/ on branch comanche-branch-1 -r 3e43bec958e627d53f02d2842f6fac24a93110a9
    Compiled by jenkins on Mon Aug 19 18:34:32 PDT 2013
    From source with checksum cf234891d3fd875413caf539bc5aa5ce
    This command was run using /usr/lib/hadoop/hadoop-core-1.2.0.1.3.2.0-111.jar
    [root@hdpmachine7 ~]#

    ambari=> select * from hostcomponentstate;
    cluster_id | component_name | current_stack_version | current_state | host_name | service_name
    ————+——————–+——————————————–+—————+————-+————–
    2 | HIVE_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine4 | HIVE
    2 | HIVE_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine3 | HIVE
    2 | ZOOKEEPER_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine4 | ZOOKEEPER
    2 | GANGLIA_MONITOR | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine7 | GANGLIA
    2 | ZOOKEEPER_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine3 | ZOOKEEPER
    2 | ZOOKEEPER_SERVER | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine3 | ZOOKEEPER
    2 | GANGLIA_MONITOR | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine5 | GANGLIA
    2 | MAPREDUCE_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine4 | MAPREDUCE
    2 | OOZIE_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine4 | OOZIE
    2 | HCAT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine1 | HCATALOG
    2 | HIVE_CLIENT | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | INSTALLED | hdpmachine1 | HIVE
    2 | NAGIOS_SERVER | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine1 | NAGIOS
    2 | GANGLIA_MONITOR | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | UNKNOWN | hdpmachine8 | GANGLIA
    2 | NAMENODE | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine1 | HDFS
    2 | ZOOKEEPER_SERVER | {“stackName”:”HDP”,”stackVersion”:”1.3.2″} | STARTED | hdpmachine1 | ZOOKEEPER
    2 |

    Collapse
    #35101

    Dave
    Moderator

    Hi Suaroman,

    Can you attach the output of “hadoop version” from your NameNode and your problematic DataNode here too?

    Thanks

    Dave

    Collapse
    #35100

    Dave
    Moderator

    ok the forum doesn’t like greater than or equal signs.

    insert:

    where host_name=’$hostname’

    Where $hostname is your hostname.

    Collapse
    #35099

    Dave
    Moderator

    Hi Suaroman,

    Sorry my where host_name=” came out as ”

    Basically you need to insert what your hostname is in there, if you do a select * you should see all your servers in your cluster.

    If the server isn’t in there, then this will not be an ambari issue as such but more a communication issue between these versions and I can track that down and see how we fix that.

    Thanks

    Dave

    Collapse
    #35098

    suaroman
    Participant

    I pasted the statement as you have it below and here’s the output:

    root@Ambari ~]# psql -U ambari ambari
    Password for user ambari:
    psql (8.4.13)
    Type “help” for help.

    ambari=> select * from hostcomponentstate where host_name=”
    ambari->
    ambari-> select * from hostcomponentstate where host_name=Ambari
    ambari->

    It looks like no results were returned from the select statement when I ran the above command from my Ambari server :(

    Collapse
    #35065

    Dave
    Moderator

    Hi Suaroman,

    Can you run the following on your ambari-server and list the output here:

    psql -U ambari ambari (passwd = bigdata)

    select * from hostcomponentstate where host_name=”

    I’m interested to see what version of the stack it says here.

    I’ll keep checking to see where this comes from though,

    Thanks

    Dave

    Collapse
    #34991

    suaroman
    Participant

    Thanks Dave.

    I used the instructions located here: http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.2/bk_using_Ambari_book/content/ambari-chap1-1.html to do the install of the 1.3 HDP stack.
    No problems after the install as all aspects of the installation worked perfectly.

    Here’s the repository list:

    [root@Ambari ~]# yum repolist
    Loaded plugins: fastestmirror, security
    Loading mirror speeds from cached hostfile
    epel/metalink | 14 kB 00:00
    * base: http://ftp.usf.edu
    * epel: mirrors.servercentral.net
    * extras: mirror.steadfast.net
    * updates: mirror.lug.udel.edu
    HDP-UTILS-1.1.0.16 | 2.9 kB 00:00
    Updates-ambari-1.x | 1.3 kB 00:00
    ambari-1.x | 1.3 kB 00:00
    base | 3.7 kB 00:00
    extras | 3.4 kB 00:00
    updates | 3.4 kB 00:00
    repo id repo name status
    HDP-UTILS-1.1.0.16 Hortonworks Data Platform Utils Version – HDP-UTILS-1. 61
    Updates-ambari-1.x ambari-1.x – Updates 40
    ambari-1.x Ambari 1.x 5
    base CentOS-6 – Base 6,381
    epel Extra Packages for Enterprise Linux 6 – x86_64 9,570
    extras CentOS-6 – Extras 13
    updates CentOS-6 – Updates 1,179
    repolist: 17,249

    Is there a way to trick the datanode so that it is permitted to connect to the namenode ? (perhaps using configuration key ) ? Not sure if this would be a good idea though.

    Any tips / steps that can help me get to a consistent state ?

    Thanks!

    Collapse
    #34620

    Dave
    Moderator

    Hi suaroman,

    What repository did you download to install build 111?
    I’ll test in my environment with the repository in the documentation as my 1.3.2 are all build 110 but I’m slightly concerned there could be 2 builds floating around.

    Thanks

    Dave

    Collapse
Viewing 17 replies - 1 through 17 (of 17 total)