Home Forums HDP on Windows – Installation single node failure after successful installation

This topic contains 20 replies, has 9 voices, and was last updated by  Seth Lyubich 1 year, 1 month ago.

  • Creator
    Topic
  • #32413

    Eric Xu
    Member

    After I successfully installed the hdp for windows 1.3.
    I run :
    PS C:\hadoop\hdp> ./start_local_hdp_services
    starting namenode
    starting secondarynamenode
    starting datanode
    starting jobtracker
    starting historyserver
    starting tasktracker
    starting zkServer
    starting master
    Start-Service : Service ‘Apache Hadoop Hbase master (master)’ cannot be started due to the following error: Cannot
    start service master on computer ‘.’.
    At C:\hadoop\hdp\manage_local_hdp_services.ps1:77 char:16
    + $foo = Start-Service -Name $serviceName.Name -ErrorAction Continue
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],
    ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

    starting regionserver
    Start-Service : Service ‘Apache Hadoop Hbase regionserver (regionserver)’ cannot be started due to the following
    error: Cannot start service regionserver on computer ‘.’.
    At C:\hadoop\hdp\manage_local_hdp_services.ps1:77 char:16
    + $foo = Start-Service -Name $serviceName.Name -ErrorAction Continue
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],
    ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

    starting hwi
    starting hiveserver
    starting hiveserver2
    starting metastore
    starting derbyserver
    starting templeton
    starting oozieservice
    Sent all start commands.
    total services
    16
    running services
    13
    not yet running services
    3
    Failed_Start namenode master regionserver

    looking forward to someone helping me.!

    there is also something in the log file:

    HBASE: Copying configuration for master regionserver rest thrift thrift2
    HBASE: Creating service config C:\hadoop\hdp\\hbase-0.94.6.1.3.0.0-0380\bin\master.xml
    HBASE: C:\hadoop\hdp\\hbase-0.94.6.1.3.0.0-0380\bin\hbase.cmd –service master start > “C:\hadoop\hdp\\hbase-0.94.6.1.3.0.0-0380\bin\master.xml”
    HBASE-CMD FAILURE: \Microsoft was unexpected at this time.
    HBASE: Creating service regionserver as C:\hadoop\hdp\\hbase-0.94.6.1.3.0.0-0380\bin\regionserver.exe
    HBASE: Adding service regionserver
    HBASE: C:\Windows\system32\sc.exe failure regionserver reset= 30 actions= restart/5000
    [SC] ChangeServiceConfig2 SUCCESS
    HBASE: C:\Windows\system32\sc.exe config regionserver start= demand
    [SC] ChangeServiceConfig SUCCESS
    HBASE: sc sdshow regionserver

Viewing 20 replies - 1 through 20 (of 20 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #40758

    Seth Lyubich
    Keymaster

    Hi Durga,

    Can you please check post below and let us know if it helps with your issue:

    http://hortonworks.com/community/forums/topic/namenode-cannot-be-started-after-successful-hdp-1-3-installation/

    Thanks,
    Seth

    Collapse
    #40726

    Durga Prasad
    Participant

    I have the following error when I try to start the remote hdp services
    C:\HDP\hadoop>start_remote_hdp_services.cmd
    Master nodes: start HDP-Hadoop.fareast.corp.microsoft.com
    0 Master nodes successfully started.
    1 Master nodes failed to start.

    PSComputerName Service Message Status
    ————– ——- ——- ——
    Connecting to re…

    StartStop-HDPservices : Manually start services on Master nodes then retry
    full cluster start. Exiting.
    At C:\HDP\hadoop\manage_remote_hdp_services.ps1:187 char:26
    + if ($mode -eq “start”) { StartStop-HDPservices($mode) }
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,StartStop-HDPServices

    Collapse
    #38708

    Roshan Naik
    Member

    To start flume please ensure the following:
    – You need to create a flume.conf based on your needs. Since the purpose of Flume is to transfer event data from point A to point B, it does not make sense to have a predefined flume config to be shipped as part of HDP.
    – On Windows : HDP Flume installation creates a flume service (see windows service manager). This service is turned off by default (for same reason mentioned above). Assuming Flume is installed in c:\hdp\flume, create a c:\hdp\flume\conf\flume.conf to suit your needs and start the Flume service. The Windows service will automatically pick up that flume.conf.

    NOTE:
    The flume windows service basically starts flume agent as follows:
    flume-ng.cmd agent -f path/conf/flume.conf -n agent

    So ensure that your agent name inside flume.conf is ‘agent’

    Collapse
    #38255

    Ryan Jaeger
    Member

    Hi,

    I had that error today and it is related to the PATH (as Dave describes). Here’s how I fixed the issue:

    1) Navigate to your hbase install: D:\hdp\hadoop\hbase-0.94.6.1.3.0.0-0380\bin
    2) Open the hbase.cmd in a text editor
    3) Look for the line that says:
    set PATH=%PATH%;%HADOOP_HOME%\bin
    4) Delete it or comment it out with a @rem
    5) Open a command prompt and navigate to hbase install: D:\hdp\hadoop\hbase-0.94.6.1.3.0.0-0380\bin
    6) Rebuild the .xml files:
    hbase.cmd –service master start > master.xml
    hbase.cmd –service regionserver start > regionserver.xml
    hbase.cmd –service rest > rest.xml
    hbase.cmd –service thrift > thrift.xml
    hbase.cmd –service thrift2 > thrift2.xml

    7) I also had problems with flume not starting.
    8) Navigate to the flume install: flume-1.3.1.1.3.0.0-0380\bin
    9) If there is no flumeagent.xml copy the flumeservice.xml and name it flumeagent.xml (not sure if this will cause problems later, but it will bring the service up).

    10) While all the services will now start, the Smoke Tests did not pass until i opened the hadoop command line and executed “hadoop namenode –format”.

    Doing these steps solved all my service problems and allowed me to run the smoke tests. The failure seems to be caused by the PATH variable containing spaces….To replicate the problem install Visual Studio on the single server.

    Good luck.

    Ryan

    Collapse
    #33022

    Seth Lyubich
    Keymaster

    Hi,

    This issue appears to be similar to http://hortonworks.com/community/forums/topic/hdp-1-3-flumeagent-and-hbase-services-will-not-start/#post-33021 . I provided some things to try on that post.

    Thanks,
    Seth

    Collapse
    #33015

    Seth Lyubich
    Keymaster

    Hi Pavan,

    Thanks for providing details. Can you please let us know which .xml files are empty? Can you please also check to see what you have in master.xml file? Can you also send us following files?

    HBase-master-*.log
    master.trace.log

    I tried and was not able to reproduce the issue today. Please let me know if you think there are specific steps that I can take to reproduce this.

    Please provide me with anything else which you think might be useful.

    You can upload logs here:

    ftp http://ftp.support.hortonworks.com
    username: dropoff
    password: horton

    Thanks,
    Seth

    Collapse
    #32996

    Pavan Keerthi
    Participant

    I am also seeing all xml files in hbase folders are blank

    Same error when starting server

    HadoopServiceTraceSource Information: 0 : Tracing successfully initialized
    DateTime=2013-08-22T20:51:27.1248131Z
    Timestamp=1926743242
    HadoopServiceTraceSource Information: 0 : Loading service xml: C:\HDP\hbase-0.94.6.1.3.0.0-0380\bin\master.xml
    DateTime=2013-08-22T20:51:27.1297495Z
    Timestamp=1926753587
    HadoopServiceTraceSource Error: 0 : Failed to parse the service xml with exceptionSystem.Xml.XmlException: Root element is missing.
    at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo(String res)
    at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
    at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
    at System.Xml.XmlDocument.Load(XmlReader reader)
    at System.Xml.XmlDocument.Load(String filename)
    at HadoopServiceHost.ServiceHost.InitInternal(String fileName)
    DateTime=2013-08-22T20:51:27.1329075Z
    Timestamp=1926761457

    Collapse
    #32914

    Eric Xu
    Member

    I have checked the master.log in the hbase folder.
    This is the error. Both the master.xml and regionserver.xml are blank.

    HadoopServiceTraceSource Information: 0 : Tracing successfully initialized
    DateTime=2013-08-21T14:04:33.5443497Z
    Timestamp=496290281
    HadoopServiceTraceSource Information: 0 : Loading service xml: C:\hadoop\hdp\hbase-0.94.6.1.3.0.0-0380\bin\master.xml
    DateTime=2013-08-21T14:04:33.6540611Z
    Timestamp=496600382
    HadoopServiceTraceSource Error: 0 : Failed to parse the service xml with exceptionSystem.Xml.XmlException: Root element is missing.
    at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo(String res)
    at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
    at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
    at System.Xml.XmlDocument.Load(XmlReader reader)
    at System.Xml.XmlDocument.Load(String filename)
    at HadoopServiceHost.ServiceHost.InitInternal(String fileName)
    DateTime=2013-08-21T14:04:33.9205117Z
    Timestamp=497299414

    Collapse
    #32864

    Seth Lyubich
    Keymaster

    Hi,

    Can you please check .wrapper file for Hbase Master to see if any obvious issues with the command line that tries to start the service?

    Thanks,
    Seth

    Collapse
    #32861

    Dave Warner
    Participant

    I’m encountering the same problem and believe it is due to the inability of the HBASE cmd file to parse PATH entries that contain embedded spaces. I am installing on a Windows 2012 Server with SQL Server 2012 Developer edition present with the default directories chosen. The pertinent error in the log is
    “HBASE-CMD FAILURE: \Microsoft was unexpected at this time.”

    When I manipulate the PATH I can get the portion following the backslash to change.

    Although the services show as not started, the smoke tests for HBASE pass, with the exception of the WebUI and ZooKeeper.

    Collapse
    #32779

    Eric Xu
    Member

    Hi, Sef
    I am sure that datanode is running. there are only two service not working.

    PS C:\hadoop\hdp> ./start_local_hdp_services
    starting datanode
    starting derbyserver
    starting historyserver
    starting hiveserver
    starting hiveserver2
    starting hwi
    starting jobtracker
    starting master
    Start-Service : Service ‘Apache Hadoop Hbase master (master)’ cannot be started due to the following error: Cannot
    start service master on computer ‘.’.
    At C:\hadoop\hdp\manage_local_hdp_services.ps1:77 char:16
    + $foo = Start-Service -Name $serviceName.Name -ErrorAction Continue
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],
    ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

    starting metastore
    starting namenode
    starting oozieservice
    starting regionserver
    Start-Service : Service ‘Apache Hadoop Hbase regionserver (regionserver)’ cannot be started due to the following
    error: Cannot start service regionserver on computer ‘.’.
    At C:\hadoop\hdp\manage_local_hdp_services.ps1:77 char:16
    + $foo = Start-Service -Name $serviceName.Name -ErrorAction Continue
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],
    ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

    starting secondarynamenode
    starting tasktracker
    starting templeton
    starting zkServer
    Sent all start commands.
    total services
    16
    running services
    13
    not yet running services
    2
    Failed_Start master regionserver

    waiting for your advice

    Collapse
    #32761

    Seth Lyubich
    Keymaster

    Hi,

    Can you please make sure that Datanode service is running? Since you have only one Datanode and it is not running it is possible that system cannot replicate any data to any datanodes.

    Hope this helps,

    Thanks,
    Seth

    Collapse
    #32703

    Eric Xu
    Member

    this is the failed log after format.

    2013-08-20 09:40:50,834 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop cause:java.io.IOException: File /mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    2013-08-20 09:40:50,834 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020, call addBlock(/mapred/system/jobtracker.info, DFSClient_NONMAPREDUCE_-59558164_1, null) from 169.254.152.82:49297: error: java.io.IOException: File /mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    java.io.IOException: File /mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1981)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:826)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1456)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1452)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1450)

    Collapse
    #32669

    Seth Lyubich
    Keymaster

    Hi,

    From last output it appears that Namenode actually started.

    starting namenode

    and only 3 services below failed:

    3
    Failed_Start datanode master regionserver

    Can you please verify to make sure that Namemode service is running? For services that did not start you can also look in component logs to see any hints.

    Can you please check and let us know if this is helpful?

    Thanks,
    Seth

    Collapse
    #32615

    Eric Xu
    Member

    I have tried to format the namenode, but still got the same error.

    Looking forward for another solution.

    Collapse
    #32586


    Member

    Sorry. did not realize previous msg was truncated.

    c:\hdp>start_local_hdp_services.cmd
    starting datanode
    starting derbyserver
    starting historyserver
    starting hiveserver
    starting hiveserver2
    starting hwi
    starting jobtracker
    starting master
    Start-Service : Service ‘Apache Hadoop Hbase master (master)’ cannot be started
    due to the following error: Cannot start service master on computer ‘.’.
    At C:\hdp\manage_local_hdp_services.ps1:77 char:29
    + $foo = Start-Service <<<< -Name $serviceName.Name -ErrorAction Conti
    nue
    + CategoryInfo : OpenError: (System.ServiceProcess.ServiceControl
    ler:ServiceController) [Start-Service], ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Comman
    ds.StartServiceCommand

    starting metastore
    starting namenode
    starting oozieservice
    starting regionserver
    Start-Service : Service 'Apache Hadoop Hbase regionserver (regionserver)' canno
    t be started due to the following error: Cannot start service regionserver on c
    omputer '.'.
    At C:\hdp\manage_local_hdp_services.ps1:77 char:29
    + $foo = Start-Service <<<< -Name $serviceName.Name -ErrorAction Conti
    nue
    + CategoryInfo : OpenError: (System.ServiceProcess.ServiceControl
    ler:ServiceController) [Start-Service], ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Comman
    ds.StartServiceCommand

    starting secondarynamenode
    starting tasktracker
    starting templeton
    starting zkServer
    Sent all start commands.
    total services
    16
    running services
    13
    not yet running services
    3
    Failed_Start datanode master regionserver

    Collapse
    #32585


    Member

    I am also having the same issue. I have ran this command “hadoop namenode -format” and still having the same error.

    ==== namenode format output ====
    C:\hdp\hadoop-1.2.0.1.3.0.0-0380>hadoop namenode -format
    13/08/19 08:51:08 INFO namenode.NameNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting NameNode
    STARTUP_MSG: host = WDMYP6DEVMP01/172.21.13.235
    STARTUP_MSG: args = [-format]
    STARTUP_MSG: version = 1.2.0.1.3.0.0-0380
    STARTUP_MSG: build = git@github.com:hortonworks/hadoop-monarch.git on branch (
    no branch) -r 4c12a850c61d98a885eba4396a4abc145abb65c8; compiled by ‘jenkins’ on
    Tue Aug 06 19:39:01 Coordinated Universal Time 2013
    STARTUP_MSG: java = 1.6.0_31
    ************************************************************/
    Re-format filesystem in c:\hadoop\data\hdfs\nn ? (Y or N) Y
    13/08/19 08:51:10 INFO util.GSet: Computing capacity for map BlocksMap
    13/08/19 08:51:10 INFO util.GSet: VM type = 64-bit
    13/08/19 08:51:10 INFO util.GSet: 2.0% max memory = 4151836672
    13/08/19 08:51:10 INFO util.GSet: capacity = 2^23 = 8388608 entries
    13/08/19 08:51:10 INFO util.GSet: recommended=8388608, actual=8388608
    13/08/19 08:51:11 INFO namenode.FSNamesystem: fsOwner=tang_s
    13/08/19 08:51:11 INFO namenode.FSNamesystem: supergroup=supergroup
    13/08/19 08:51:11 INFO namenode.FSNamesystem: isPermissionEnabled=false
    13/08/19 08:51:11 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
    13/08/19 08:51:11 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessK
    eyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
    13/08/19 08:51:11 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length
    = 0
    13/08/19 08:51:11 INFO namenode.NameNode: Caching file names occuring more than
    10 times
    13/08/19 08:51:11 INFO util.GSet: Computing capacity for map INodeMap
    13/08/19 08:51:11 INFO util.GSet: VM type = 64-bit
    13/08/19 08:51:11 INFO util.GSet: 1.0% max memory = 4151836672
    13/08/19 08:51:11 INFO util.GSet: capacity = 2^22 = 4194304 entries
    13/08/19 08:51:11 INFO util.GSet: recommended=4194304, actual=4194304
    13/08/19 08:51:12 INFO common.Storage: Image file of size 165 saved in 0 seconds
    .
    13/08/19 08:51:12 INFO namenode.FSEditLog: closing edit log: position=4, editlog
    =c:\hadoop\data\hdfs\nn\current\edits
    13/08/19 08:51:12 INFO namenode.FSEditLog: close success: truncate to 4, editlog
    =c:\hadoop\data\hdfs\nn\current\edits
    13/08/19 08:51:12 INFO common.Storage: Storage directory c:\hadoop\data\hdfs\nn
    has been successfully formatted.
    13/08/19 08:51:12 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at WDMYP6DEVMP01/172.21.13.235
    ************************************************************/

    C:\hdp\hadoop-1.2.0.1.3.0.0-0380>

    —– service output —
    starting master
    Start-Service : Service ‘Apache Hadoop Hbase master (master)’ cannot be started
    due to the following error: Cannot start service master

    Collapse
    #32548

    Rohit Bakhshi
    Moderator

    Hi,

    It looks like your NameNode service did not start. There is a possible cause for this – the NameNode data directories were not created.

    Could you please try this:
    1. Open the “Hadoop Command Line” Command Prompt shortcut.
    2. Run the following command that sets up the NameNode directories: “hadoop namenode -format”

    Once that has run successfully, please re-try starting all the services with the start_local_hdp_services

    Let me know if that solves your issue.

    Collapse
    #32530

    Pavan Keerthi
    Participant

    I am noticing the same error

    Collapse
    #32414

    Eric Xu
    Member

    I wonder that hdp for windows should have at least more than 3 nodes?

    Collapse
Viewing 20 replies - 1 through 20 (of 20 total)