Home Forums Security Kerberos security in HDP, "GSS initiate failed" for the "hdfs" user

Tagged: 

This topic contains 5 replies, has 3 voices, and was last updated by  Yeyun Lu 4 months ago.

  • Creator
    Topic
  • #28100

    I’m trying to enable security in HDP 2.0, deployed using Ambari 1.4.0 (from the developers’ repository), on a virtual machine, in a single-node cluster.
    I have a problem with Kerberos TGT.
    .
    I try to execute the following 2 commands (taken from error messages from Puppet):

    [root@dev01 ~]# /usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs
    [root@dev01 ~]# su hdfs -c “hadoop –config /etc/hadoop/conf fs -mkdir -p /mapred”
    13/06/25 10:14:04 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
    13/06/25 10:14:04 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
    13/06/25 10:14:04 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
    mkdir: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: “dev01.hortonworks.com/192.168.56.101″; destination host is: “dev01.hortonworks.com”:8020;
    [root@dev01 ~]#

    The keytab file (/etc/security/keytabs/hdfs.headless.keytab) is in place, the 1st command finished OK, but the 2nd comand did not work.

    Then I tried:
    [root@dev01 ~]# kinit -R
    kinit: Ticket expired while renewing credentials

    It looks like a ticket has expired immediately after kinit.
    Then I try to check:

    [root@dev01 ~]# klist
    Ticket cache: FILE:/tmp/krb5cc_0
    Default principal: hdfs@EXAMPLE.COM

    Valid starting Expires Service principal
    06/25/13 10:13:46 06/26/13 10:13:46 krbtgt/EXAMPLE.COM@EXAMPLE.COM
    renew until 06/25/13 10:13:46
    [root@dev01 ~]#

    But it looks like the ticket is valid, as far as I understand.
    Now I don’t understand what’s going on with Kerberos TGT here.

    Here is the Kerberos config (/etc/krb5.conf):
    ——–
    [logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

    [libdefaults]
    default_realm = EXAMPLE.COM
    dns_lookup_realm = false
    dns_lookup_kdc = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true

    [realms]
    EXAMPLE.COM = {
    kdc = dev01.hortonworks.com
    admin_server = dev01.hortonworks.com
    }

    [domain_realm]
    .hortonworks.com = EXAMPLE.COM
    dev01.hortonworks.com = EXAMPLE.COM
    ——–

    Can somebody help me?

Viewing 5 replies - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #46106

    Yeyun Lu
    Member

    Have you sovled the problem? I met the similar problem with yours…Everything is like OK, but when I do hdfs operation, then get the same error..Failed to find any Kerberos tgt

    Collapse
    #28381

    Sasha J
    Moderator

    Dmitry,
    take a look to this doc:

    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.3.1/bk_installing_manually_book/content/rpm-chap14.html

    It is for HDP 1.x, but I believe should work same way on HDP 2.x
    All the properties should be installed.

    I set up kerberos number of time, using mentioned document, it works fine for me.
    You may also try to use script setupKerberos.sh ( it coming together with gsINstaller package).
    Always works fine.

    http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_gsInstaller/content/ch_gsInstaller-chp1.html

    Thank you!
    Sasha

    Collapse
    #28373

    Sasha,
    Thank you for your answer.

    I checked, according to this document: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Running_Hadoop_in_Secure_Mode

    core-site.xml has these properties:

    hadoop.security.authentication
    kerberos

    hadoop.security.authorization
    true

    hdfs-site.xml has these properties:

    dfs.secondary.namenode.kerberos.principal
    nn/_HOST@EXAMPLE.COM

    dfs.datanode.kerberos.principal
    dn/_HOST@EXAMPLE.COM

    dfs.namenode.kerberos.principal
    nn/_HOST@EXAMPLE.COM

    These properties are absent:
    dfs.namenode.kerberos.https.principal
    dfs.namenode.secondary.kerberos.https.principal
    dfs.datanode.kerberos.https.principal

    Possibly, it can be the cause of the problem.

    The file named “captured-site.xml” doesn’t exist and I haven’t seen any reference to it before.

    By the way, is this sufficient for the “hdfs” user keytab?
    xst -k /etc/security/keytabs/hdfs.headless.keytab hdfs

    I just thought, maybe I should try also this:
    xst -k /etc/security/keytabs/hdfs.headless.keytab host/dev01.hortonworks.com
    (I didn’t do that yet)

    Collapse
    #28193

    Sasha J
    Moderator

    Dmitry,
    Have you put all the needed properties to core-site.xml, hdfs-site.xml and captured-site.xml?
    All processes have to be restarted .
    Also, this is a wrong thread for HDP 2.0 questions.
    Please, use correct one: http://hortonworks.com/community/forums/forum/hdp-2-0-alpha-feedback-2/

    Thank you!
    Sasha

    Collapse
    #28102

    Checking the keytab itself:

    [root@dev01 ~]# klist -e -k -t /etc/security/keytabs/hdfs.headless.keytab
    Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
    KVNO Timestamp Principal
    —- —————– ——————————————————–
    2 06/20/13 03:47:55 hdfs@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
    2 06/20/13 03:47:55 hdfs@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
    2 06/20/13 03:47:55 hdfs@EXAMPLE.COM (des3-cbc-sha1)
    2 06/20/13 03:47:55 hdfs@EXAMPLE.COM (arcfour-hmac)
    2 06/20/13 03:47:55 hdfs@EXAMPLE.COM (des-hmac-sha1)
    2 06/20/13 03:47:55 hdfs@EXAMPLE.COM (des-cbc-md5)
    [root@dev01 ~]#

    Is everything OK with the keytab?

    Collapse
Viewing 5 replies - 1 through 5 (of 5 total)