Add extra Hard Disk/Phisical memory at Data Node
Add extra Hard Disk/Phisical memory at datanode As we are processing data on hadoop cluster we need to configure many things. The main thing is to manage temporary files which is generation in between map reduce. We can set map-reduce intermediate compression and map-reduce output compression. After setting every things we need atleat double/tripple space on datanode with respect to existing dataset on HDFS. So we can add some memory on hadoop cluster in 2 ways: 1. Add 1 or more datanode datanode 2. Add harddisk on the exising datanodes How to add extra HDD: First add a HardDisk on all datanode machine and mount on a point/name(example: /hdd2) Now create datanode directory in new harddisk (example: /hdd2/datanode) Now change ownership of /hdd2/datanode to hdfs/hadoop user then Stop one datanode and add the new Harddisk location in hdfs-site.xml <property> <name>dfs.datanode.data.dir</name> <value>/hdd/datanode,