Home Forums HDP on Linux – Installation All smoke tests are getting failed

This topic contains 8 replies, has 2 voices, and was last updated by  Sasha J 1 year, 6 months ago.

  • Creator
    Topic
  • #15585

    Tanzir
    Participant

    Hello everyone,
    Yesterday I have installed HDP (all components) on 7 nodes and at that time everything was working fine. But I stopped the services last night before I stopped all the instances (Ec2). This morning I have started all the instances and started servicing in the order written in http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.0/bk_reference/content/reference_chap3_1.html.

    I did this couple of times and never faced any issues. But this morning, I’m getting some errors during smoke test. Though the service is started but during smoke test it threw the following error:

    notice: /Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln 32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully
    err: /Stage[2]/Hdp-hadoop::Hdfs::Service_check/Hdp-hadoop::Exec-hadoop[hdfs::service_check::check_safemode]/Hdp::Exec[hadoop --config /etc/hadoop/conf dfsadmin -safemode get | grep OFF]/Exec[hadoop --config /etc/hadoop/conf dfsadmin -safemode get | grep OFF]/returns: change from notrun to 0 failed: Could not find command ‘hadoop’

    Any idea whats going on here? Any information will be highly appreciated.

    Thanks in advance.

Viewing 8 replies - 1 through 8 (of 8 total)

You must be logged in to reply to this topic.

  • Author
    Replies
  • #15593

    Sasha J
    Moderator

    OK, sounds good!
    Keep us informed.

    Sasha

    Collapse
    #15592

    Tanzir
    Participant

    I checked those processes too and all seemed to be running. Yes, I also believe that its something related to my EC2 environment. When I installed HDP, it was working fine. I also restarted the services and all services came back as expected. But after I restarted EC2 instances then they started failing. I will install again to see whether I can reproduce the same problem again.

    I did try with locally installed VMs and it worked fine. I installed on EC2 instances earlier and it worked at that time too and they are still running without any problems. But this time I tried to install HDP on a new configuration of EC2 instances. I will look into the configurations again. May be I’m missing something there.

    Thanks a lot for your help, appreicate it.
    -Tanzir

    Collapse
    #15591

    Sasha J
    Moderator

    You said “all ambari-agents are running”
    what about other processes?
    Like NameNode, Datanode, etc…

    Sasha

    Collapse
    #15590

    Sasha J
    Moderator

    Tanzir,
    you can safely ignore warning…
    As of reinstallation, yes, “ambari-server reset” will clean up internal database and your system will looks like a freshly installed OS.

    It is hard to say why this happened, this may be related to EC2 specifics…
    Why to use EC2 and not locally installed VMs?

    Thank you!
    Sasha

    Collapse
    #15589

    Tanzir
    Participant

    Interesting when I ran as ambari_qa I didn’t get permission denied error message instead I got warning on the 3rd line.

    [ambari_qa:~]$ hadoop –config /etc/hadoop/conf dfs -rmr mapredsmokeoutput mapredsmokeinput ; hadoop dfs -put /etc/passwd mapredsmokeinputrmr: cannot remove mapredsmokeoutput: No such file or directory.
    rmr: cannot remove mapredsmokeinput: No such file or directory.
    13/02/21 12:54:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

    I checked all the hosts and all ambari agents are running. All smoke-tests are getting failed. Do you think the whole setup somehow got corrupted after I started the instances today? Since it’s a fresh copy I can install it again. But wondering if I face the same issue again after instance restart.

    One quick question – What is the best practice to erase the whole HDP installation? Just “ambari-server reset”?

    Thanks again Sasha.
    -Tanzir

    Collapse
    #15588

    Sasha J
    Moderator

    Most likely thiose commands should be run as ambari_qa user, not hdfs… Sorry for misleading…
    However, it seems like problems with environment, as you can run command “hadoop” from the command line, but it can not be found by Ambari-agent…
    Is the processes itself running?
    Only smoketest failed?

    Thank you!
    Sasha

    Collapse
    #15587

    Tanzir
    Participant

    Hi Sasha,
    Thanks a lot for your quick response. Yes, I was able to execute the “hadoop” command manually. But when I ran it manually I got this:

    [hdfs:~]$ hadoop –config /etc/hadoop/conf dfs -rmr mapredsmokeoutput mapredsmokeinput ; hadoop dfs -put /etc/passwd mapredsmokeinput
    rmr: cannot remove mapredsmokeoutput: No such file or directory.
    rmr: cannot remove mapredsmokeinput: No such file or directory.
    put: /usr/lib/hadoop/mapredsmokeinput (Permission denied)

    Want to repeat one thing here is, it was working fine after a fresh install. Problem started happening righter after I stopped the services and started again.

    Thanks again.

    Collapse
    #15586

    Sasha J
    Moderator

    Tanzir,
    could you check if this command is actually accessible?
    login to your ec2 instance and try to run this command manually, from user hdfs.

    Thank you!
    Sasha

    Collapse
Viewing 8 replies - 1 through 8 (of 8 total)