By the way: The cluster has one NameNode and 7 DataNodes. ...
By the way: The cluster has one NameNode and 7 DataNodes.
see the output of cluster summary.
Security is OFF
69023 files and directories, 68580 blocks = 137603 total.
Heap Memory used 79.92 MB is 45% of Commited Heap Memory 176.75 MB. Max Heap Memory is 1.74 GB.
Non Heap Memory used 36.25 MB is 70% of Commited Non Heap Memory 51.50 MB. Max Non Heap Memory is 130 MB.
Configured Capacity : 1.56 TB
DFS Used : 1.21 TB
Non DFS Used : 16.22 GB
DFS Remaining : 336.78 GB
DFS Used% : 77.87%
DFS Remaining% : 21.11%
Block Pool Used : 1.21 TB
Block Pool Used% : 77.87%
you can see that most disk is taken by Block Pool.
This will causes disk is full issue when running map-reduce.
What’s the root cause about this issue? how to prevent it?
A HDP Support Subscription connects you experts with deep experience running Apache Hadoop in production, at-scale on the most demanding workloads.
Real world training designed by the core architects of Hadoop. Scenario-based training courses are available in-classroom or online from anywhere in the world