Get fresh updates from Hortonworks by email

Once a month, receive latest insights, trends, analytics information and knowledge of Big Data.

cta

Get Started

cloud

Ready to Get Started?

Download sandbox

How can we help you?

closeClose button
November 09, 2017
prev slideNext slide

Ambari Kerberos support for HBase Part 1

Hortonworks Data Platform(HDP) and the majority of the components in HDP support Kerberos based authentication mechanism. By default, authentication is disabled to allow ease of installation, however for production as well as sensitive data hosting clusters we highly recommend enabling Kerberos based authentication.  While configuring and deploying Kerberos enabled applications might seem challenging and time-consuming task, Ambari makes it extremely simple for HDP by automating provisioning all SPNs and distributing keytab files across the cluster using a wizard. This is first in a series of blog posts to help empower administrators and developers with steps to activate secure Kerberos authentication for their HDP clusters. This blog specifically will cover how to deploy a Kerberos enabled Hadoop (HDFS + YARN) cluster which we will use to then deploy a Kerberos enabled HBase cluster in the next few posts.

What is Kerberos?

Kerberos is a network authentication protocol built with untrusted networks and trusted hosts model in mind. Here are a few Kerberos terms to get familiar with that we will be commonly using in this blog series:

  1. Key Distribution Center (KDC)
  2. Ticket Granting Ticket (TGT)
  3. Service Principal Name (SPN)
  4. Keytab Files

Let’s get started, enable Kerberos

We will start by installing Kerberos KDC Server on the Ambari node, if you have an existing KDC you can use it as well given admin credentials are available for the KDC.

Install Kerberos KDC

$>sudo yum -y install krb5-server

Configure Kerberos KDC

Before we can initialize the KDC database and create a domain, we will need to configure KDC with the name of our Realm (Domain). We will need to edit:

  1. /etc/krb5.conf
  2. /var/kerberos/krb5kdc/kdc.conf
  3. /var/kerberos/krb5kdc/kadm5.acl

We will be specifying what Kerberos domain/realm we are going to use for our cluster and specify the location of the KDC. In our case the KDC is installed on the Ambari-Server node.

Here’s what these files look like for our cluster with highlighted parts that were changed.

$>cat /etc/krb5.conf
#
[libdefaults]
     default_realm = YOUR_DOMAIN.COM
     default_tkt_enctypes = DES-CBC-CRC
     default_tgs_enctypes = DES-CBC-CRC
     ccache_type = 2
[realms]
     YOUR_DOMAIN.COM = {
     kdc = kdc.your_domain.com:88
     admin_server = admin.your_domain.com:749
     }
[domain_realm]
     .your_domain.com = YOUR_DOMAIN.COM
     your_domain.com = YOUR_DOMAIN.COM
[logging]
     kdc = FILE:/var/adm/krb5kdc.log
     admin_server = FILE:/var/log/kadmin.log
     default = FILE:/var/log/krb5lib.log
$>cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
    kdc_ports = 88
    kdc_tcp_ports = 88

[realms]
    YOUR_DOMAIN.COM = {
        kdc_ports = 88
        admin_keytab = /etc/kadm5.keytab
        database_name = /var/kerberos/krb5kdc/principal
        acl_file = /var/kerberos/krb5kdc/kadm5.acl
        key_stash_file = /var/kerberos/krb5kdc/stash
        max_life = 10h 0m 0s
        max_renewable_life = 7d 0h 0m 0s
        master_key_type = des3-hmac-sha1
        supported_enctypes = arcfour-hmac:normal des3-hmac-sha1:normal 
des-cbc-crc:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3
        default_principal_flags = +preauth
    }
$>cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@YOUR_DOMAIN.COM  *

For more details, you can refer to the official documentation here:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/install-kdc.html

Now that the configurations are fixed, let’s initialize the Kerberos database.

Note: if the system entropy for random numbers is low, this can take a while, this command can help speed up the process rngd –r /dev/urandom –o /dev/random –b  (requires rng-tools installed)

$>sudo /usr/sbin/kdb5_util create -s

Setup master password (and store it in a safe place)

$>sudo /usr/sbin/kadmin.local -q “addprinc admin/admin”

Create an additional admin principal (and store password in a safe place, we will need it for finishing the Ambari installation) and start Kerberos KDC and KAdmin servers.

$> sudo service krb5kdc start

$> sudo service kadmin start

Enable Kerberos in Ambari for a Secured HBase Cluster

To start, click the Admin tab and select Kerberos in Ambari Web.

Click Proceed Anyway, as there is no existing data on this new cluster.

Enable Kerberos in Ambari for a Secured HBase Cluster

*If you are doing this on an existing cluster then existing jobs must be stopped to avoid issues.

We have already deployed Kerberos KDC in the previous phase and created the admin credentials. JCE was deployed on all machines as a part of our image baking process. If you are using Active Directory, please refer to the blog post (http://hortonworks.com/blog/enabling-kerberos-hdp-active-directory-integration/)

Next, add details about Realm and location of the KDC (ambari-server in our case) as well as the admin principal that we created while deploying our Kerberos KDC.

After this, click test connection to validate if we are able to connect to the Kerberos KDC and then click Next. Once validated, click Next once the Kerberos client installation is complete.

Enable Kerberos Wizard

 

After this, follow the wizard to complete the rest of the Kerberos setup.

Here you can review the setup

The deployment will take some time to complete.

When completed your screen for Ambari UI should look like below where it states “Kerberos security is enabled”

If you need more specific details, you can follow the official documentation here:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/_launching_the_kerberos_wizard_automated_setup.html

Next, go to Ambari home page and restart all services that need configuration changes.

Finally, we have Kerberos enabled on the cluster, next step is to validate that kerberos is indeed enabled and that we are able to validate authentication.

Using the shell on the Ambari Server:

[centos@ambud-hdp-3 ~]$ ls -lh /etc/security/keytabs/

total 16K

-rw——-. 1 root      root   393 Dec 22 19:50 ambari.server.keytab

-r–r—–. 1 hdfs      hadoop 348 Dec 22 19:50 hdfs.headless.keytab

-r–r—–. 1 ambari-qa hadoop 373 Dec 22 19:50 smokeuser.headless.keytab

-r–r—–. 1 root      hadoop 463 Dec 22 19:50 spnego.service.keytab

Let’s validate the Kerberos authentication is indeed active for the cluster. Without having a TGT we have no credentials to use HDFS, therefore, authentication should fail causing the command to fail as well.

[centos@hdp ~]$> sudo -u hdfs hdfs dfs -ls /

16/12/22 19:56:26 WARN ipc.Client: Exception encountered while connecting to the server :

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

Now let’ get a TGT and then attempt to use simple HDFS commands

$>kinit hdfs-user_local –t /etc/security/keytabs/hdfs.headless.keytab

$>hdfs dfs –ls /

If you can see file listing, that means you’ve successfully enabled and setup Kerberos for your cluster.  In the following parts, we will discuss how to enable authentication and authorization for HBase as well as how to programmatically access the cluster.

Comments

  • Great article, helped me a lot. However, I had a small hiccup with starting Zookeeper service, as I figured out it is also recommended to set udp_preference_limit=1 in /etc/krb5.conf . I’ve enjoyed working with Hortonworks distribution although the documentation is a bit hard to digest, these articles definitely helps.

  • Leave a Reply

    Your email address will not be published. Required fields are marked *