cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

closeClose button
January 21, 2014
prev slideNext slide

How To Install Hadoop on Windows with HDP 2.0

Installing the Hortonworks Data Platform 2.0 for Windows is straightforward. Lets take a look at how to install a one node cluster on your Windows Server 2012 R2 machine.

To start, download the HDP 2.0 for Windows package. The package is under 1 GB, and will take a few moments to download depending on your internet speed. Documentation for installing a single node instance is located here. This blog post will guide you through that instruction set to get you going with HDP 2.0 for Windows!

Here’s an outline of the process you’ll work through to deploy:

  • Install the prerequisites
  • Deploy HDP on your single node machine
  • Start the services
  • Run smoke tests to validate the install

Install the Pre-requisites

You’ll now install Java, Python, and MSFT C++ run time. Windows Server 2012 already has the up to date .NET runtime, so you can skip that step.

Let’s download the C++ run time, and install that by double clicking the downloaded MSI.

Download Python 2.7.x, and double click the downloaded MSI to install the package.

Once you’ve installed, you’ll need to ensure HDP can find Python – by updating the PATH System Environment variable.

Go to Computer > Properties > Advanced System Settings > Environment variables. Then append the install path to Python, for example C:Python27, to this path after a ‘;’:

()

Verify your path is setup by entering a new Powershell or Command Prompt and typing: python, which should run the python interpreter. Type quit() to exit.

Setup Java, which you can get here. You will also need to setup JAVA_HOME, which Hadoop requires. Make sure to install Java to somewhere without a space in the path – “Program Files” will not work!

To setup JAVA_HOME, in Explorer > right click Computer > Properties > Advanced System Settings > Environment variables. Then setup a new System variable called JAVA_HOME that points to your Java install (in this case, C:javajdk1.6.0_31).

Install the MSI package

Now we have all the pre-requisites installed. The next step is to install the HDP 2.0 for Windows package.

Extract the MSI from the zip package you downloaded earlier. Open a Powershell prompt in Administrator (“Run as Administrator”) mode, and execute the MSI through this command:

   > msiexec /i "hdp-2.0.6.0.winpkg.msi"

The HDP Setup window appears pre-populated with the host name of the server, as well as default installation parameters. Now, complete the form with your parameters:

  • Set the Hadoop User Password. This enables you to log in as the administrative user and perform administrative actions. This must match your local Windows Server password requirements. We recommend a strong pasword. Note the password you set – we’ll use this later.
  • Check ‘Delete Existing HDP Data’. This ensures that HDFS will be formatted and ready to use after you install.
  • Check ‘Install HDP Additional Components’. Select this check box to install Zookeeper, Flume, and HBase as HDP services deployed to the single node server.
  • Set the Hive and Oozie database credentials. Set ‘hive’ for all Hive Metastore entries, and ‘oozie’ for all Oozie Metastore entries.
  • Select DERBY, and not MSSQL, as the DB Flavor in the dropdown selection. This will setup HDP to use an embedded Derby database, which is ideal for the evaluation single node scenario.

When you have finished setting the installation parameters, click ‘Install’ to install HDP.

The HDP Setup window will close, and a progress indicator will be displayed while the installer is running. The installation will take a few minutes – disregard the progress bar expected time display.

The MSI installer window will display an info prompt when the installation is finished and successful.

Start the services and run a jobs

Once the install is successful, you will start the HDP services on the single node.

Open a command prompt, and navigate to the HDP install directory. By default, the location is “C:hdp”, unless you set a different location:

   > cd C:hdp

   > start_local_hdp_services

Validate the install by running the full suite of smoke tests. It’s easiest to run the smoke tests as the HDP super user: ‘hadoop’.

In a command prompt, switch to using the ‘hadoop’ user:

   > runas /user:hadoop cmd

When prompted, enter the password you had set up during install.

Run the provided smoke tests as the hadoop user to verify that the HDP 2.0 services work as expected:

   > cd C:hdp

   > Run-SmokeTests hadoop

This will fire up a Mapreduce job on your freshly set up cluster. If it fails the first time, try running it again with the same command Run-SmokeTests hadoop.

Congratulations, you are now Hadooping on Windows!

If you’d like to learn more about Hadoop, check out the Hortonworks Sandbox, a virtual machine for you to learn Hadoop and sign up for our free ‘Learn Hadoop in 2 Weeks‘ guided tutorial series.

Categories:

Comments

  • I followed the steps in windows server 2012, but I get the installation inition has’ installation failed ‘and I downloaded the 2.0 hdp two occasions thinking it was the executable, but keeps coming’ installation ‘failed’.

  • Hello, I’m trying to install hadoop 2.0 in a windows 2012 DC virtual machine but I got all the time problems with hadoop user password so I can’t proceed with installation.

    I have found this piece of code in the hadoop lo file. It seems that hadoop user does not meet the requirements to be accepted, but I’m using capital letters and number and my password is more than 6 letter so I don’t know what is happening. When I try one more time I got the same problem. It seems that the first password that I’ve entered with no capital letters and numbers has been saved somewhere.

    Could you please help me!?

    CREATE-USER: Creating user hadoop
    CREATE-USER: Setting password for hadoop
    CREATE-USER FAILURE: Exception calling “SetInfo” with “0” argument(s): “The password does not meet the password policy requirements. Check the minimum password length, password complexity and password history requirements.

    At C:HadoopInstallFilesHadoopPackageshdp-2.0.6.0-winpkgscriptscreate_hadoop_user.ps1:88 char:9
    + $user.SetInfo()
    + ~~~~~~~~~~~~~~~

    Best regards,
    Lucho

  • Hello, I was struggling with the installation of the package and I found an error when the instalation package tries to install OOZIE. This below is the error message
    OOZIE: Copy Oozie additional Jars
    OOZIE: Calling Setup script to add Hadoop libs to the generated oozie.war file
    OOZIE FAILURE: Unable to execute ‘jar xvf’. Error (1)
    At D:hdpoozie-4.0.0.2.0.6.0-0009oozie-win-distrobinoozie-setup.ps1:129 char:9
    + throw “Unable to execute ‘jar xvf’. Error ($LASTEXITCODE)”
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Could you please let me know what should I do to fix it?
    Thanks in advanced for your help!
    Lucho Farje

  • If your locale uses , instead of . in numbers, you have to change the Get-TimeStamp function in Run-SmokeTests.ps1 to:

    function Get-TimeStamp
    {
    (Get-Date -UFormat(‘%s’)) -Replace(“.”,””) -Replace(“,”,””)
    }

    @Rohit, you should integrate this by default.

  • How can I add another node to this singlenode install? I cannot a find set of instructions on how to add a node to the cluster on widnows?

  • Hello,

    > The MSI installer window will display an info prompt when the installation is finished and successful

    Well, something gone wrong when trying HDP Sandbox, only because I tried to run the WMWare player appliance, without success (I´m trying to download it again). Tried to install the HDP for Windows, without success either…

    Please advice,
    Sven

  • Need help. I have tried installing and uninstalling HDP 2.0 more than 20 times during last 5 days. So far it installs successfully but SmokeTests fail. I used administrative access for installation.

    1. I am on Windows 7, JDK 1.7.0_02 & phthon 27.
    2. hdp-2.0.6.0.winpkg.install.log shows successful installation and does not show any errors.
    3. Firewall is off for everything
    4. I am on private network
    5. Only Smoke tests passed: Sqoop, Oozie
    6. Extracts of SmokeTest results..
    14/02/07 10:57:01 WARN util.NativeCodeLoader: Unable to load native-hadoop libra
    ry for your platform… using builtin-java classes where applicable
    copyFromLocal: Call From MALT-7539/192.168.2.7 to MALT-7539:8020 failed on conne
    ction exception: java.net.ConnectException: Connection refused: no further infor
    mation; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
    Run-HadoopSmokeTest : Error copying the input file for the Hadoop smoke test
    At line:1 char:20
    + Run-HadoopSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-HadoopSmokeTest

    14/02/07 10:57:06 WARN util.NativeCodeLoader: Unable to load native-hadoop libra
    ry for your platform… using builtin-java classes where applicable
    14/02/07 10:57:06 INFO client.RMProxy: Connecting to ResourceManager at MALT-753
    9/192.168.2.7:8032

  • What’s up, I check your new stuff on a regular basis. Your story-telling style is witty, keep doing what you’re doing!

  • Hi there. I followed all the instructions to the letter and the install still says: “Installation failed. Please see installation log for details: <> This is great but no log is being created in the destination so my troubleshooting options are very limited. BTW: I’m using Win Server 2012 R2.

  • Try removing the space in the folder name. It worked for me. For Program Files, it was only taking “Files” and searching for JDK.

  • I am trying to install HDP on windows but It keeps giving me an error message on the last step that the Password doesnt match the system requirements defined in system policies.

  • Windows 7 Ultimate, 64-bit. JDK 1.8.0.11, Python34 – installation fails, no log written, no information in the system log (beyond the advise to check the non-existent log file.
    To me, free was too expensive this time, wasting about a day to troubleshoot.

  • I installed the Hortonworks Data Platform 2.0 for Windows. The tutorial stops at Run-SmokeTests hadoop. After that there is no information / guide to what we can do on this windows installation .All tutorials are related to virtual sandbox. Is it possible to provide tutorials for Hortonworks Data Platform 2.0 for Windows similar to what you have for virtual sandbox http://hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/ .

  • I am getting a pop-up saying “JAVA_HOME” must be set even when I have set it with a path that has no spaces.
    Please help!!

    • I had the same issue and realized that it was because I had JAVA_HOME set as a user environment variable rather than as a system environment variable.

  • I am getting a pop-up saying “JAVA_HOME” must be set even when I have set it with a path that has no spaces.
    Please help!!

  • Hello guys,

    I am trying to install hdp 2.3.1.0 for windows in a single node. I have already installed Python under Anaconda pck. Anaconda directory is in PATH env. variable. If I run python from a cmd command, python 2.7.7 executes and point to anaconda from any directory I run. Also I have installed Java.

    The fact is that I get an error: No python executable found in path. I am lost…. What could be happened?

    Thanks a lot.

    Regards. JL

  • I read a lot of interesting posts here. Probably you spend a lot of time writing, i know how to save you a lot of time,
    there is an online tool that creates high quality, SEO friendly articles in seconds,
    just type in google – laranitas free content source

  • Exactly where did we download a .zip file? I didn’t see one, also can’t find “hdp-2.0.6.0.winpkg.msi”
    Nothing is working for me or the others here. What a poor tutorial and waste of time

  • Is there a similar thing for Mac OS? I want to install single node on my Mac. I dont want to use the Sandbox VM.

  • Getting below errors while running smoke test
    —————————————————————————————————
    C:hdp>Run-SmokeTests hadoop
    Hadoop smoke test – wordcount using hadoop.cmd file
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.
    copyFromLocal: Call From andromeda/192.168.1.34 to ANDROMEDA:8020 failed on conn
    ection exception: java.net.ConnectException: Connection refused: no further info
    rmation; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
    Run-HadoopSmokeTest : Error copying the input file for the Hadoop smoke test
    At line:1 char:20
    + Run-hadoopSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-HadoopSmokeTest
    ————————————————————–
    ——
    Run-HadoopSmokeTest : Hadoop Smoke Test: FAILED
    At line:1 char:20
    + Run-hadoopSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-HadoopSmokeTest

  • even I installed the Hortonworks Data Platform 2.0 for Windows. The tutorial stops at Run-SmokeTests hadoop. After that what I need to do?There is no information regarding windows installation.help me out.

  • Hi guys , I also tried and getting errors like path for python not able to find .
    So anybody have updated on that.

  • hey rohit,
    thank you for your detailed post.

    i have installed everything successfully. but after that can you tell me the working of this ? how should i make use of this ?
    actually i need to make this hadoop working on my computer lab of my college. so i need some Real details.

  • Getting the following error on trying to run the smoketests as mentioned in the page.

    16/11/11 10:58:10 INFO mapreduce.Job: map 0% reduce 0%
    16/11/11 10:58:10 INFO mapreduce.Job: Job job_1477935038448_0015 failed with sta
    te FAILED due to: Application application_1477935038448_0015 failed 2 times due
    to AM Container for appattempt_1477935038448_0015_000002 exited with exitCode:
    -1000 due to: Could not find any valid local directory for nmPrivate/container_1
    477935038448_0015_02_000001.tokens
    .Failing this attempt.. Failing the application.
    16/11/11 10:58:10 INFO mapreduce.Job: Counters: 0
    Run-HadoopSmokeTest : Hadoop Smoke Test: FAILED
    At line:1 char:20
    + Run-hadoopSmokeTest <<<<
    + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep
    tion
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptio
    n,Run-HadoopSmokeTest

    Please let me know how can I have this resolved.

    I have Windows 2008 Operating System in which I tried above

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    If you have specific technical questions, please post them in the Forums

    You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>