Home Forums HDP on Windows – Installation Clusterproperties.txt file for slave node:

This topic contains 19 replies, has 6 voices, and was last updated by  Seth Lyubich 1 year, 5 months ago.

  • Creator
    Topic
  • #20957

    Hi,
    I’m able to install HDP for Windows on single node cluster with the following clusterproperties file:

    HDP_LOG_DIR=C:\hadoop\logs
    HDP_DATA_DIR=C:\hdp\data
    NAMENODE_HOST=NODE1.TECTestLab.local
    SECONDARY_NAMENODE_HOST=NODE1.TECTestLab.local
    JOBTRACKER_HOST=NODE1.TECTestLab.local
    HIVE_SERVER_HOST=NODE1.TECTestLab.local
    OOZIE_SERVER_HOST=NODE1.TECTestLab.local
    TEMPLETON_HOST=NODE1.TECTestLab.local
    SLAVE_HOSTS=NODE1.TECTestLab.local
    DB_FLAVOR=derby
    DB_HOSTNAME=NODE1.TECTestLab.local
    DB_PORT=1527
    HIVE_DB_NAME=hivedb
    HIVE_DB_USERNAME=hive_user
    HIVE_DB_PASSWORD=hive_pass
    OOZIE_DB_NAME=ooziedb
    OOZIE_DB_USERNAME=ooziedb_user
    OOZIE_DB_PASSWORD=ooziedb_pass

    where : NODE1.TECTestLab.local is Master node
    NODE2.TECTestLab.local is Slave node

    If I want NODE1.TECTestLab.local to be Master node and NODE2.TECTestLab.local is slave node, can anybody please let me know what should be the clusterproperties file in NODE2.TECTestLab.local. Also, please do let me know what are services need to be running for slave node?

    Kindly waiting for your response.

    Thanks,
    Santosh

Viewing 19 replies - 1 through 19 (of 19 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #24229

    Seth Lyubich
    Keymaster

    Hi Santosh,

    Can you please check to make sure that hadoop service are running?

    Thanks,
    Seth

    Collapse
    #24159

    Thanks Seth. When I ran the smoke test, I’ve got some messages in Red as below:

    SyntaxError: invalid syntax
    Run-HCatalogSmokeTest : HCatalog Smoke Test: FAILED
    At line:1 char:22
    + Run-HCatalogSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-HCatalogSmokeTest

    + Run-TempletonSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-TempletonSmokeTest
    -1

    Please do let me know what might be wrong. Thank you very much for your much needed help.

    Collapse
    #24137

    Seth Lyubich
    Keymaster

    Hi Santosh,

    You should be able to see output in the command shell you running it from. You should be able to redirect output to a file if needed.

    Hope this helps,

    Seth

    Collapse
    #24030

    Thanks Seth for the response. As you said, I was running from the wrong location.

    After running some tests from C:\hdp\hadoop, where should I test the result? Both the log file as well as the result.

    Thanks,
    Santosh

    Collapse
    #23448

    Seth Lyubich
    Keymaster

    Hi Santosh,

    Can you please check from what location you ran the smoke test?

    Documentation says:
    cd %HADOOP_NODE_INSTALL_ROOT%
    Run-SmokeTests

    looks like you running it form – C:\Users\Administrator>C:\HadoopInstallFiles\HadoopPackages\hdp-1.1.0-winpkg\scr
    ipts\Run-SmokeTests.cmd

    Can you please verify that you are using correct location?

    I also provided some answers below to make sure that you installer installed and started all services. Can you please review and let us know if this is useful.

    Hope this helps.

    Thanks,
    Seth

    Collapse
    #23251

    Thanks Lary for the response.

    Yes, ‘C:\HadoopInstallFiles\HadoopPackages\hdp-1.1.0-winpkg\scripts\cluster.properties’ exists.

    Initially cluster.properties file did not exist. I manually copied in the path and then ran the smoke test. But I’m still getting this error. Can anybody please do let me know what could be the reason?

    Thanks,
    Santosh

    Collapse
    #23169

    Larry Liu
    Moderator

    Hi, Santosh

    Here is the error:
    Get-Content : Cannot find path ‘C:\HadoopInstallFiles\HadoopPackages\hdp-1.1.0-
    winpkg\scripts\cluster.properties’ because it does not exist.

    Can you please verify if C:\HadoopInstallFiles\HadoopPackages\hdp-1.1.0-winpkg\scripts\cluster.properties exists?

    Larry

    Collapse
    #23162

    Here is the log file for Run Smoke Tests:

    2013-04-24 01:13:04,700 INFO org.apache.hadoop.hive.hwi.HWIServer: HWI is starting up
    2013-04-24 01:13:15,810 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2013-04-24 01:13:15,903 INFO org.mortbay.log: jetty-6.1.26
    2013-04-24 01:13:16,308 INFO org.mortbay.log: Extract C:\hdp\hadoop\hive-0.9.0\lib\hive-hwi-0.9.0.war to C:\Users\HADOOP~1.000\AppData\Local\Temp\Jetty_0_0_0_0_9999_hive.hwi.0.9.0.war__hwi__2l99ri\webapp
    2013-04-24 01:13:19,450 INFO org.mortbay.log: Started SocketConnector@0.0.0.0:9999
    2013-04-24 02:49:28,518 INFO org.apache.hadoop.hive.hwi.HWIServer: HWI is starting up
    2013-04-24 02:49:40,971 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2013-04-24 02:49:41,533 INFO org.mortbay.log: jetty-6.1.26
    2013-04-24 02:49:42,737 INFO org.mortbay.log: Extract C:\hdp\hadoop\hive-0.9.0\lib\hive-hwi-0.9.0.war to C:\Users\HADOOP~1.000\AppData\Local\Temp\Jetty_0_0_0_0_9999_hive.hwi.0.9.0.war__hwi__2l99ri\webapp
    2013-04-24 02:49:46,221 INFO org.mortbay.log: Started SocketConnector@0.0.0.0:9999
    2013-04-24 03:07:42,363 INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to process : 1
    2013-04-24 03:07:42,397 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
    2013-04-24 03:07:42,398 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
    2013-04-24 03:07:42,889 INFO org.apache.hadoop.mapred.JobClient: Running job: job_201304240249_0001
    2013-04-24 03:07:43,895 INFO org.apache.hadoop.mapred.JobClient: map 0% reduce 0%
    2013-04-24 03:08:05,215 INFO org.apache.hadoop.mapred.JobClient: map 100% reduce 0%
    2013-04-24 03:08:18,199 INFO org.apache.hadoop.mapred.JobClient: map 100% reduce 100%
    2013-04-24 03:08:19,212 INFO org.apache.hadoop.mapred.JobClient: Job complete: job_201304240249_0001
    2013-04-24 03:08:19,240 INFO org.apache.hadoop.mapred.JobClient: Counters: 29
    2013-04-24 03:08:19,241 INFO org.apache.hadoop.mapred.JobClient: Job Counters
    2013-04-24 03:08:19,241 INFO org.apache.hadoop.mapred.JobClient: Launched reduce tasks=1
    2013-04-24 03:08:19,241 INFO org.apache.hadoop.mapred.JobClient: SLOTS_MILLIS_MAPS=8211
    2013-04-24 03:08:19,241 INFO org.apache.hadoop.mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
    2013-04-24 03:08:19,242 INFO org.apache.hadoop.mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
    2013-04-24 03:08:19,254 INFO org.apache.hadoop.mapred.JobClient: Launched map tasks=1
    2013-04-24 03:08:19,254 INFO org.apache.hadoop.mapred.JobClient: Data-local map tasks=1
    2013-04-24 03:08:19,255 INFO org.apache.hadoop.mapred.JobClient: SLOTS_MILLIS_REDUCES=10015
    2013-04-24 03:08:19,255 INFO org.apache.hadoop.mapred.JobClient: File Output Format Counters

    Collapse
    #23161

    C:\Users\Administrator>C:\HadoopInstallFiles\HadoopPackages\hdp-1.1.0-winpkg\scr
    ipts\Run-SmokeTests.cmd
    Get-Content : Cannot find path ‘C:\HadoopInstallFiles\HadoopPackages\hdp-1.1.0-
    winpkg\scripts\cluster.properties’ because it does not exist.
    At C:\HadoopInstallFiles\HadoopPackages\hdp-1.1.0-winpkg\scripts\Run-SmokeTests
    .ps1:26 char:28
    + $propfile = Get-Content <<<< $filepath
    + CategoryInfo : ObjectNotFound: (C:\HadoopInstal…ster.properti
    es:String) [Get-Content], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetCo
    ntentCommand

    You cannot call a method on a null-valued expression.
    At C:\HadoopInstallFiles\HadoopPackages\hdp-1.1.0-winpkg\scripts\Run-SmokeTests
    .ps1:29 char:25
    + $line=$line.Trim <<<< ()
    + CategoryInfo : InvalidOperation: (Trim:String) [], RuntimeExcep
    tion
    + FullyQualifiedErrorId : InvokeMethodOnNull
    Hadoop smoke test – wordcount using hadoop.cmd file
    13/04/24 03:07:42 INFO input.FileInputFormat: Total input paths to process : 1
    13/04/24 03:07:42 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    13/04/24 03:07:42 WARN snappy.LoadSnappy: Snappy native library not loaded
    13/04/24 03:07:42 INFO mapred.JobClient: Running job: job_201304240249_0001
    13/04/24 03:07:43 INFO mapred.JobClient: map 0% reduce 0%
    13/04/24 03:08:05 INFO mapred.JobClient: map 100% reduce 0%
    13/04/24 03:08:18 INFO mapred.JobClient: map 100% reduce 100%
    13/04/24 03:08:19 INFO mapred.JobClient: Job complete: job_201304240249_0001
    13/04/24 03:08:19 INFO mapred.JobClient: Counters: 29
    13/04/24 03:08:19 INFO mapred.JobClient: Job Counters
    13/04/24 03:08:19 INFO mapred.JobClient: Launched reduce tasks=1
    13/04/24 03:08:19 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=8211
    13/04/24 03:08:19 INFO mapred.JobClient: Total time spent by all reduces wai
    ting after reserving slots (ms)=0
    13/04/24 03:08:19 INFO mapred.JobClient: Total time spent by all maps waitin
    g after reserving slots (ms)=0
    13/04/24 03:08:19 INFO mapred.JobClient: Launched map tasks=1
    13/04/24 03:08:19 INFO mapred.JobClient: Data-local map tasks=1
    13/04/24 03:08:19 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=10015
    13/04/24 03:08:19 INFO mapred.JobClient: File Output Format Counters
    13/04/24 03:08:19 INFO mapred.JobClient: Bytes Written=4756
    13/04/24 03:08:19 INFO mapred.JobClient: FileSystemCounters
    13/04/24 03:08:19 INFO mapred.JobClient: FILE_BYTES_READ=6321
    13/04/24 03:08:19 INFO mapred.JobClient: HDFS_BYTES_READ=7755
    13/04/24 03:08:19 INFO mapred.JobClient: FILE_BYTES_WRITTEN=60501
    13/04/24 03:08:19 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=4756
    13/04/24 03:08:19 INFO mapred.JobClient: File Input Format Counters
    13/04/24 03:08:19 INFO mapred.JobClient: Bytes Read=7614
    13/04/24 03:08:19 INFO mapred.JobClient: Map-Reduce Framework
    13/04/24 03:08:19 INFO mapred.JobClient: Map output materialized bytes=6187
    13/04/24 03:08:19 IN

    Collapse
    #22866

    Hi Santos,

    First Uninstall HDP and modify cluster properties file in line SLAVE_HOSTS to NODE2.TECTestLab.local.
    untouched others properties and execute hdp msi in both PC. Note : Don’t change any thing in properties file for slave and master nodes.

    Hope this will help you.

    Mahabub.

    Collapse
    #22170

    Seth Lyubich
    Keymaster

    Hi Santosh:

    >> how can I test/verify that HDP got installed on two nodes

    You can go to services and check to make sure that HDP service got installed and running.

    >> how will should I test one is master and another is slave node

    You can take a look at which services running on each node (master – NameNode, JobTracker, etc; and slave services – DataNode, TaskTracker).

    You should make sure that services are started and running prior to running smoke test. Can you please check to make sure that all services are running om master and slave node?

    On another note I see that you said “…install HDP for Windows on single node”. Can you please clarify what are you trying to do? Did you install single node, and trying to add a slave node? This might not be the best way to do it as your single node might have slave services installed already.

    Hope this helps.

    Seth

    Collapse
    #22088

    Error:
    C:\>C:\hdp\hadoop\Run-SmokeTests.cmd
    Hadoop smoke test – wordcount using hadoop.cmd file
    The system cannot find the path specified.
    Run-HadoopSmokeTest : Error copying the input file for the Hadoop smoke test
    At line:1 char:20
    + Run-HadoopSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-HadoopSmokeTest
    The system cannot find the path specified.
    Run-HadoopSmokeTest : Hadoop Smoke Test: FAILED
    At line:1 char:20
    + Run-HadoopSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-HadoopSmokeTest
    Pig smoke test – wordcount using hadoop.cmd file
    The system cannot find the path specified.
    The system cannot find the path specified.
    Run-PigSmokeTest : Pig Smoke Test: FAILED
    At line:1 char:17
    + Run-PigSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-PigSmokeTest
    Hcatalog smoke test – show tables, create table, and drop table
    Running hcat command: show tables
    python: can't open file '\bin\hcat.py': [Errno 2] No such file or directory
    Run-HCatalogSmokeTest : HCatalog Smoke Test: FAILED
    At line:1 char:22
    + Run-HCatalogSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-HCatalogSmokeTest
    2
    Hive smoke test – create table and describe table
    Running hive command: create external table if not exists hivesmoke ( foo INT, b
    ar STRING)
    The system cannot find the path specified.
    Run-HiveSmokeTest : Hive Smoke Test: FAILED
    At line:1 char:18
    + Run-HiveSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-HiveSmokeTest
    3
    Sqoop smoke test – version
    The system cannot find the path specified.
    Run-SqoopSmokeTest : Sqoop Smoke Test: FAILED
    At line:1 char:19
    + Run-SqoopSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-SqoopSmokeTest
    Oozie smoke test – status
    The system cannot find the path specified.
    Run-OozieSmokeTest : Oozie Smoke Test: FAILED
    At line:1 char:19
    + Run-OozieSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-OozieSmokeTest
    Templeton smoke test – status, show databases, show tables
    Calling URL: http://NODE1.TECTestLab.local:50111/templeton/v1/status
    Response: 200
    Data: {"status":"ok","version":"v1"}
    Calli

    Collapse
    #21864

    Larry Liu
    Moderator

    Hi, Santosh,

    1. There is no test required. The master node and slave node can be found in the Clusterproperties.txt file.
    2. Can you please provide your steps to run smoke test? Also with the error you get?

    Thanks
    Larry

    Collapse
    #21850

    Thanks for the response.

    Well, how can I test/verify that HDP got installed on two nodes? And how will should I test one is master and another is slave node? (Run-SmokeTests command is giving error to me)
    Please do let me know at the earliest and the detailed steps to validate as its consuming too much time.

    Thanks,
    Santosh

    Collapse
    #21810

    Yi Zhang
    Moderator

    Hi Santosh,

    The live node counts the number of live datanodes, which in your case is 1.

    Regards,
    Yi

    Collapse
    #21682

    Thanks Seth for the response.

    I’m able to install the msi in both master and slave nodes. I’m able to view both the nodes in dashboard in each nodes.

    As you said, I ran only DataNode and TaskTracker services in slave node and all the services in the Master node. But in dashboard of Master node, the LiveNodes is “1″ which is supposed to display as “2″. Let me know if this is the right way validating the no of nodes installed.

    I’m assuming that, the 2 nodes are not communicating to each other somehow. (I’m able to ping both the nodes from one another). Can you please let me know if I should do some other setting.

    Kindly waiting for your response.

    Thanks,
    Santosh

    Collapse
    #21205

    Seth Lyubich
    Keymaster

    Hi Santosh,

    Usually tasktracker and datanode services are running on the slave nodes. When you configure node to be a slave node it will run the DataNode and TaskTracker services.

    Did you try to run installer on the slave node? Can you please let us know if services got installed and if you got any errors and at which point it failed.

    Hope this helps,
    Seth

    Collapse
    #21088

    Thank you Robert for the response.

    I changed the clusterproperties as you mentioned, but didn’t worked. Can you please let me know what all services should be running in slave machine? Also, any other changes need to be done in slave node? (Basically I wanted to know the difference between master and slave node wrt configuration settings)

    Collapse
    #21036

    Robert
    Participant

    Hi Santosh,
    For two node cluster, It’s up to you to split one box as master and the other as slave. For both machines the configuration file should be the same. Hence your slave file I believe would contain the following line:

    SLAVE_HOSTS=NODE1.TECTestLab.local, NODE2.TECTestLab.local

    The FQDN’s of the host are separated via comma.

    Hope that helps.

    Regards,
    Robert Molina

    Collapse
Viewing 19 replies - 1 through 19 (of 19 total)