All smoke tests are getting failed

Hello everyone,
Yesterday I have installed HDP (all components) on 7 nodes and at that time everything was working fine. But I stopped the services last night before I stopped all the instances (Ec2). This morning I have started all the instances and started servicing in the order written in http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.0/bk_reference/content/reference_chap3_1.html.

I did this couple of times and never faced any issues. But this morning, I’m getting some errors during smoke test. Though the service is started but during smoke test it threw the following error:

notice: /Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln 32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully
err: /Stage[2]/Hdp-hadoop::Hdfs::Service_check/Hdp-hadoop::Exec-hadoop[hdfs::service_check::check_safemode]/Hdp::Exec[hadoop --config /etc/hadoop/conf dfsadmin -safemode get | grep OFF]/Exec[hadoop --config /etc/hadoop/conf dfsadmin -safemode get | grep OFF]/returns: change from notrun to 0 failed: Could not find command ‘hadoop’

Any idea whats going on here? Any information will be highly appreciated.

Thanks in advance.

Topic

[Sasha JModerator]

OK, sounds good!
Keep us informed.

Sasha

[Tanzir MusabbirMember]

I checked those processes too and all seemed to be running. Yes, I also believe that its something related to my EC2 environment. When I installed HDP, it was working fine. I also restarted the services and all services came back as expected. But after I restarted EC2 instances then they started failing. I will install again to see whether I can reproduce the same problem again.

I did try with locally installed VMs and it worked fine. I installed on EC2 instances earlier and it worked at that time too and they are still running without any problems. But this time I tried to install HDP on a new configuration of EC2 instances. I will look into the configurations again. May be I’m missing something there.

Thanks a lot for your help, appreicate it.
-Tanzir

[Sasha JModerator]

You said “all ambari-agents are running”
what about other processes?
Like NameNode, Datanode, etc…

Sasha

[Sasha JModerator]

Tanzir,
you can safely ignore warning…
As of reinstallation, yes, “ambari-server reset” will clean up internal database and your system will looks like a freshly installed OS.

It is hard to say why this happened, this may be related to EC2 specifics…
Why to use EC2 and not locally installed VMs?

Thank you!
Sasha

[Tanzir MusabbirMember]

Interesting when I ran as ambari_qa I didn’t get permission denied error message instead I got warning on the 3rd line.

[ambari_qa:~]$ hadoop –config /etc/hadoop/conf dfs -rmr mapredsmokeoutput mapredsmokeinput ; hadoop dfs -put /etc/passwd mapredsmokeinputrmr: cannot remove mapredsmokeoutput: No such file or directory.
rmr: cannot remove mapredsmokeinput: No such file or directory.
13/02/21 12:54:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

I checked all the hosts and all ambari agents are running. All smoke-tests are getting failed. Do you think the whole setup somehow got corrupted after I started the instances today? Since it’s a fresh copy I can install it again. But wondering if I face the same issue again after instance restart.

One quick question – What is the best practice to erase the whole HDP installation? Just “ambari-server reset”?

Thanks again Sasha.
-Tanzir

[Sasha JModerator]

Most likely thiose commands should be run as ambari_qa user, not hdfs… Sorry for misleading…
However, it seems like problems with environment, as you can run command “hadoop” from the command line, but it can not be found by Ambari-agent…
Is the processes itself running?
Only smoketest failed?

Thank you!
Sasha

[Tanzir MusabbirMember]

Hi Sasha,
Thanks a lot for your quick response. Yes, I was able to execute the “hadoop” command manually. But when I ran it manually I got this:

[hdfs:~]$ hadoop –config /etc/hadoop/conf dfs -rmr mapredsmokeoutput mapredsmokeinput ; hadoop dfs -put /etc/passwd mapredsmokeinput
rmr: cannot remove mapredsmokeoutput: No such file or directory.
rmr: cannot remove mapredsmokeinput: No such file or directory.
put: /usr/lib/hadoop/mapredsmokeinput (Permission denied)

Want to repeat one thing here is, it was working fine after a fresh install. Problem started happening righter after I stopped the services and started again.

Thanks again.

[Sasha JModerator]

Tanzir,
could you check if this command is actually accessible?
login to your ec2 instance and try to run this command manually, from user hdfs.

Thank you!
Sasha

[Author]

Replies