The key piece of information here was that kinit was working but HDP was not.
The resolution was to check and set the permissions and owner:group of the keytab files correctly.
This makes sense since kinit from the root account could read the keytab file but trying to read the keytab from another account failed.
HDP on Linux – Installation Forum
HDP 1.3 Kerberos problem
I am installing HDP 1.3 on CentOS 6.4 servers using the Ambari installation method.
The cluster runs fine until I move it to secure mode and then I get
java.io.IOException: Login failure for nn/fqdn@REALM from keytab /etc/security/keytabs/nn.service.keytab
kinit nn/fqdn@REALM -kt /etc/security/keytabs/nn.service.keytab works just fine.
I have checked the hdfs-site.xml and there are no funny characters in the nn/_HOST@REALM entry.
I really don’t know where to look next. Any suggestions would be appreciated.
Support from the Experts
A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.
Become HDP Certified
Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world