9.3. Recommended Memory Configurations for the MapReduce Service

The following recommendations can help you determine appropriate memory configurations based on your usage scenario:

  • Make sure that there is enough memory for all the processes. Remember that system processes take around 10% of the available memory.

  • For co-deploying an HBase RegionServer and MapReduce service on the same node, reduce the RegionServer's heap size (use the HBase Settings: RegionServer: HBase Region Servers maximum Java heap size property to modify the RegionServer heap size).

  • For co-deploying an HBase RegionServer and the MapReduce service on the same node, or for memory intensive MapReduce applications, modify the map and reduce slots as suggested in the following example:

EXAMPLE: For co-deploying an HBase RegionServer and the MapReduce service on a machine with 16GB of available memory, the following would be a recommended configuration:

2 GB: system processes

8 GB: MapReduce slots. 6 Map + 2 Reduce slots per 1 GB task

4 GB: HBase RegionServer

1 GB: TaskTracker

1 GB: DataNode

To change the number of Map and Reduce slots based on the memory requirements of your application, use the following properties:

MapReduce Settings: TaskTracker: Number of Map slots per node

MapReduce Settings: TaskTracker: Number of Reduce slots per node


loading table of contents...