Change default Block Size

Hello,
Is it possible to change the default block size in hadoop in sandbox , and also is it possible to specify block size for a task.

Topic

[yi zhangMember]

Hi Zero,

Good that you figured it out. The fun of hadoop is to play with it!

Yi

[zero quakeMember]

running it using the pig interface and yes i have changed the default block size in hadoop-site.xml which is located in /etc/hadoop/conf folder and it worked , my bad i was trying it out one same table without creating a new one again.newbie issues , thank you

[yi zhangMember]

Hi Zero Quake,

How are you running the job? Is it a hadoop jar job, or hive, pig, etc?
For example, if you are running a hadoop jar job, you can put all the job-specific configuration in your-job-specific-conf file, then use the -conf to load it; Alternatively you can use -D to define each property:

‘hadoop jar testjob.jar -conf your-job-specific-file -D mapred.map.tasks=xx ….’

A helpful link here about the number of map/reduce tasks:

http://wiki.apache.org/hadoop/HowManyMapsAndReduces

If the sandbox vm has an ip that is accessible, then you can ssh into it. You can check or edit its network configurations to suite your needs.

Hope this helps,
Yi

[zero quakeMember]

SET mapred.min.split.size 20000;
SET mapred.max.split.size 50000;
SET pig.SplitCombination false;
SET default_parallel 30;
SET dfs.block.size 8388608;

tried these , number of maps dint change , tried vi into the hdfs-site.xml for some reason ,it doesnt allow me to properly edit it , can we ssh into the vm from outside ?? and try it out ? any other solutions ?

THank you hortonworks for the sandbox , i learnt lot more through that than 2 days of reading random tutorials.

[Larry LiuModerator]

Hi, Zero

The parameter “mapred.max.split.size” which can be set per job individually is what you looking for. If you want to change default block size, I think you can do it as well. Try to edit hdfs-site.xml for property dfs.block.size.

Thanks

Larry

[Author]

Replies