Okay the solution was to increase mapred.cluster.reduce.memory.mb
mapred.cluster.reduce.memory.mb=768MB #allowed 2 reducers to run at once
mapred.cluster.reduce.memory.mb=1536MB #allowed all 4 reducers to run at once.
So it seems this is a cluster wide memory setting that sets the limit of reducer task memory across the cluster of each slot allocation. But that’s a guess Feel free to comment if you understand why this is the case.