Home Forums HDP on Windows – Installation Zlib for Hadoop 1.1.0 on Windows

This topic contains 4 replies, has 2 voices, and was last updated by  Seth Lyubich 1 year, 1 month ago.

  • Creator
    Topic
  • #36308

    Subroto Sanyal
    Participant

    Hi,

    I have installed Hadoop(hdp-1.1.0-winpkg) on Windows Server 2008 R2 Standard Service Pack 1 64-bit OS . I configure my MR jobs with the following configuration:
    mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec
    mapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec

    The Job succeeds with following warning:
    2013-09-16 01:53:55,817 WARN org.apache.hadoop.io.compress.zlib.ZlibFactory: Failed to load/initialize native-zlib library

    I have put the zlib1.dll in C:\Windows\System32, C:\Windows\SysWOW64 from http://www.zlib.net/.

    Please let me know if there is a configuration problem on my system or a bug?

    Cheers,
    Subroto Sanyal

Viewing 4 replies - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #39067

    Seth Lyubich
    Keymaster

    Hi Subroto,

    You need to configure additional properties in configuration files or pass it with the job like in the example below.

    hadoop jar hadoop-examples-1.1.0-SNAPSHOT.jar sort “-Dmapred.compress.map.output=true” “-Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec” “-Dmapred.output.compress=true” “-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec” -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text input output

    More information can be found here:

    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-Win-1.3.0/bk_sysadmin-guide/content/sysadmin_hdfs_compressions.html

    Hope this helps.

    Thanks,
    Seth

    Collapse
    #39017

    Subroto Sanyal
    Participant

    Hi Seth,

    I have tried putting the file zlib1.dll(from the mentioned download link) in C:\Windows\System32 but, I am facing the same problem.

    Cheers,
    Subroto Sanyal

    Collapse
    #38970

    Seth Lyubich
    Keymaster

    Hi Subroto,

    Can you please try to re-download zlib1.dll file and see if it works? Please use link below for download:

    http://public-repo-1.hortonworks.com/HDP-Win/zlib/1.2.7/zlib1.dll

    Please let me know if this resolves your issue.

    Thanks,
    Seth

    Collapse
    #36982

    Subroto Sanyal
    Participant

    Request help on this…..

    Collapse
Viewing 4 replies - 1 through 4 (of 4 total)