I have setup HDP 2.0 cluster using 5 Virtual nodes (OS: CentOS 6.0 using VirtualBox). When I stop and restart ALL processes using Ambari, most of the times I get failure for few demons (randomly). I see following error(s) in logs. As a solution I have to manually delete “rm -f /var/lib/rpm/__db.00*” and restart the processes. After this fix, the processes start normally.
What I have observed that every time Ambari tries to install the packages on the nodes (or atleast check if they are available) and somehow this corrupts the RPMDB.
Please suggest if this behavior is due to environment settings or due to flaw in the cluster setup/configuration. If there is inherent problem with Ambari, is there any workaround/fix ?
err: /Stage/Hdp::Snappy::Package/Hdp::Package[snappy]/Hdp::Package::Process_pkg[snappy]/Package[snappy]/ensure: change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install snappy' returned 1: rpmdb: Thread/process 1757/139873492805376 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 - (-30974)
error: cannot open Packages database in /var/lib/rpm
Error: rpmdb open failed