Thursday, November 13, 2014

10GB Terasort Benchmark on 5-node Raspberry PI Cluster - 4H 12m 5s

After more than a week of testing, tweeking, and retesting, we were able to successfully run a 10GB Terasort on a 5-node Raspberry PI cluster! Each node has 512MB ram and a 16GB SD card.

Hadoop job_201411131351_0001 on master

User: hadoop
Job Name: TeraSort
Job File: hdfs://pi0:54310/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201411131351_0001/job.xml
Submit Host: pi0
Submit Host Address: 10.1.10.120
Job-ACLs: All users are allowed
Job Setup: Successful
Status: Succeeded
Started at: Thu Nov 13 14:23:35 MST 2014
Finished at: Thu Nov 13 18:35:40 MST 2014
Finished in: 4hrs, 12mins, 5sec
Job Cleanup: Successful

Kind% CompleteNum TasksPendingRunningCompleteKilledFailed/Killed
Task Attempts
map100.00%

1520015200 / 0
reduce100.00%

1520015200 / 0


Counter Map Reduce Total
File Input Format Counters Bytes Read 0 0 10,000,298,372
Job Counters SLOTS_MILLIS_MAPS 0 0 24,993,499
Launched reduce tasks 0 0 152
Total time spent by all reduces waiting after reserving slots (ms) 0 0 0
Rack-local map tasks 0 0 144
Total time spent by all maps waiting after reserving slots (ms) 0 0 0
Launched map tasks 0 0 152
Data-local map tasks 0 0 8
SLOTS_MILLIS_REDUCES 0 0 34,824,665
File Output Format Counters Bytes Written 0 0 10,000,000,000
FileSystemCounters FILE_BYTES_READ 10,341,496,856 10,200,000,912 20,541,497,768
HDFS_BYTES_READ 10,000,315,092 0 10,000,315,092
FILE_BYTES_WRITTEN 20,409,243,506 10,208,123,719 30,617,367,225
HDFS_BYTES_WRITTEN 0 10,000,000,000 10,000,000,000
Map-Reduce Framework Map output materialized bytes 0 0 10,200,138,624
Map input records 0 0 100,000,000
Reduce shuffle bytes 0 0 10,200,138,624
Spilled Records 0 0 300,000,000
Map output bytes 0 0 10,000,000,000
Total committed heap usage (bytes) 0 0 57,912,754,176
CPU time spent (ms) 0 0 40,328,090
Map input bytes 0 0 10,000,000,000
SPLIT_RAW_BYTES 16,720 0 16,720
Combine input records 0 0 0
Reduce input records 0 0 100,000,000
Reduce input groups 0 0 100,000,000
Combine output records 0 0 0
Physical memory (bytes) snapshot 0 0 52,945,952,768
Reduce output records 0 0 100,000,000
Virtual memory (bytes) snapshot 0 0 123,024,928,768
Map output records 0 0 100,000,000

Map Completion Graph
Reduce Completion Graph


Go back to JobTracker

This is Apache Hadoop release 1.2.1

Installing Hadoop on a Raspberry PI cluster

This document details setting up and testing Hadoop on a Raspberry PI cluster. The details are almost exactly the same for any Linux flavor, particularly Debian/Ubuntu. There are only a couple of PI specifics.

In this case, we have a 5-node cluster. If you cluster is a different size, make sure the hadoop distribution is on each of the machines, make sure you update the slaves file, and copy the config files to each machine as shown.
 

Machine Setup

This first thing to do would be to add dns entries for all of your cluster machines. If you have a dns sever available, then that's great. If you don't, then you can edit the hosts file and add the correct entries there. This has to be done for every machine in the cluster. My entries look like this:

10.1.10.120 pi0 master
10.1.10.121 pi1
10.1.10.122 pi2
10.1.10.123 pi3
10.1.10.124 pi4
 
For each machine, the hostname needs to be changed to reflect the hostnames defined above. The default Rasbian hostname is raspberrypi .

Edit the /etc/hostname file and change the entry to the hostname that you want. In my case, the first PI is pi0 . A quick reboot of each node should verify the correct hostname of each node.

 
We want to add a hadoop user to each machine. This is not strictly necessary for hadoop to run, but we do this to keep our hadoop stuff separate from anything else on these machines.

Add a new user to each machine to run hadoop.
pi@pi0 ~ $ sudo adduser hadoop 
Adding user `hadoop' ... 
Adding new group `hadoop' (1004) ... 
Adding new user `hadoop' (1001) with group `hadoop' ... 
Creating home directory `/home/hadoop' ... 
Copying files from `/etc/skel' ... 
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully 
Changing the user information for hadoop 
Enter the new value, or press ENTER for the default 
 Full Name []: Hadoop 
 Room Number []: 
 Work Phone []: 
 Home Phone []: 
 Other []: 
Is the information correct? [Y/n] Y 
 
 
Log out as the pi user and log back in as the hadoop user on your master node. Create a private identity key and copy to your authorized_keys file.
 
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Copy the identity key to all of your other nodes.
ssh-copy-id -i .ssh/id_dsa.pub hadoop@pi1
ssh-copy-id -i .ssh/id_dsa.pub hadoop@pi2
ssh-copy-id -i .ssh/id_dsa.pub hadoop@pi3
ssh-copy-id -i .ssh/id_dsa.pub hadoop@pi4
 
 


You can also copy the keys directly to the new machine with something like this.

scp .ssh/id_dsa.pub hadoop@pi1:.ssh/authorized_keys
 

Hadoop Setup



Copy the distribution to the hadoop user account and decompress.
I always create symlink to make things a bit easier:
ln -s hadoop-1.x.x hadoop


Now we need to edit some hadoop config files to make everything work.

cd hadoop/conf
ls


This should give you a listing that looks like this:
hadoop@raspberrypi ~/hadoop/conf $ ls 
capacity-scheduler.xml  core-site.xml       hadoop-env.sh               hadoop-policy.xml  
log4j.properties       mapred-site.xml  slaves                  ssl-server.xml.example 
configuration.xsl       fair-scheduler.xml  hadoop-metrics2.properties  hdfs-site.xml      
mapred-queue-acls.xml  masters          ssl-client.xml.example  taskcontroller.cfg 


This files we're interested in are masters and slaves. Add the address of the node that will be the primary NameNode and JobTracker to the masters file. Add the addresses of the nodes that will be the DataNodes and TaskTrackers to the slaves file. You can also add the master node address to the slaves file if you want it to store data and run map reduce jobs, but I'm not doing that in this case. These files looks like this for my cluster:
 
hadoop@raspberrypi ~/hadoop/conf $ more masters 
pi0 
hadoop@raspberrypi ~/hadoop/conf $ more slaves 
pi1
pi2
pi3
pi4 


These files need to be copied to all the nodes in the cluster. The easy way to do this would be as follows:

hadoop@pi0 ~/hadoop/conf $ scp masters slaves pi1:hadoop/conf/. 
masters       100%   12     0.0KB/s   00:00                                        
slaves       100%   48     0.1KB/s   00:00    
hadoop@pi0 ~/hadoop/conf $ scp masters slaves pi2:hadoop/conf/. 
masters       100%   12     0.0KB/s   00:00    
slaves       100%   48     0.1KB/s   00:00    
hadoop@pi0 ~/hadoop/conf $ scp masters slaves pi3:hadoop/conf/. 
masters       100%   12     0.0KB/s   00:00    
slaves       100%   48     0.1KB/s   00:00    
hadoop@pi0 ~/hadoop/conf $ scp masters slaves pi4:hadoop/conf/. 
masters       100%   12     0.0KB/s   00:00    
slaves       100%   48     0.1KB/s   00:00


We need to define the JAVA_HOME setting in hadoop-env.sh file. Since Hadoop runs on java, it needs to know where the executable is. The Oracle JDK comes standard with the latest Rasbian release. We can set JAVA_HOME as follows:

# The java implementation to use.  Required. 
export JAVA_HOME=/usr/lib/jvm/jdk-7-oracle-armhf


Next we need to look at the core-site.xml, hdfs-site.xml and mapred-site.xml files. These are the configuration files for the distributed file system HDFS, and the code that runs the map reduce programs that we want to run. By default, both of these processes put their files in temporary file system space. That's fine to play around with, but as soon as you reboot, all of your data disappears and you have to start over again. We first need to tell each machine where the NameNode is. This is configured in the core-site.xml file.

<property>
 <name>fs.default.name</name>
 <value>hdfs://master:54310</value>
</property>

Next we need to tell each node where the
JobTracker is and a safe place to store data so that it stays permanent. This is configured in the mapred-site.xml file.

<property>
 <name>mapred.job.tracker</name>
 <value>master:54311</value>
</property>
 
<property>
 <name>mapred.local.dir</name>
 <value>/home/hadoop/mapred</value>
</property>


Lastly we need to specify safe directories to store our files for HDFS.

<property>
 <name>dfs.name.dir</name>
 <value>/home/hadoop/name</value>
</property>
<property>
 <name>dfs.data.dir</name>
 <value>/home/hadoop/data</value>
</property> 
 
 
All of these files need to be copied to each of the other nodes in the cluster. We can do that like we did before with the masters and slaves files.
 
hadoop@pi0 ~/hadoop/conf $ scp hadoop-env.sh core-site.xml hdfs-site.xml mapred-site.xml hadoop@pi1:hadoop/conf/. 
hadoop-env.sh       100% 2218     2.2KB/s   00:00    
core-site.xml      100%  268     0.3KB/s   00:00    
hdfs-site.xml      100%  347     0.3KB/s   00:00    
mapred-site.xml      100%  356     0.4KB/s   00:00    
hadoop@pi0 ~/hadoop/conf $ scp hadoop-env.sh core-site.xml hdfs-site.xml mapred-site.xml hadoop@pi2:hadoop/conf/. 
hadoop-env.sh       100% 2218     2.2KB/s   00:00    
core-site.xml      100%  268     0.3KB/s   00:00    
hdfs-site.xml      100%  347     0.3KB/s   00:00    
mapred-site.xml      100%  356     0.4KB/s   00:00    
hadoop@pi0 ~/hadoop/conf $ scp hadoop-env.sh core-site.xml hdfs-site.xml mapred-site.xml hadoop@pi3:hadoop/conf/. 
hadoop-env.sh       100% 2218     2.2KB/s   00:00    
core-site.xml      100%  268     0.3KB/s   00:00    
hdfs-site.xml      100%  347     0.3KB/s   00:00    
mapred-site.xml      100%  356     0.4KB/s   00:00    
hadoop@pi0 ~/hadoop/conf $ scp hadoop-env.sh core-site.xml hdfs-site.xml mapred-site.xml hadoop@pi4:hadoop/conf/. 
hadoop-env.sh       100% 2218     2.2KB/s   00:00    
core-site.xml      100%  268     0.3KB/s   00:00    
hdfs-site.xml      100%  347     0.3KB/s   00:00    
mapred-site.xml      100%  356     0.4KB/s   00:00


We have one interesting task left. The hadoop script by default wants to run the datanode in server mode. The Oracle Java distribution does not support this option. We need to edit the bin/hadoop script to remove this option.

cd bin
vi hadoop

Search for “-server”
Change this:
    HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS" 
to this:
    HADOOP_OPTS="$HADOOP_OPTS $HADOOP_DATANODE_OPTS" 
 
 
We need to copy this file to all of the nodes in the cluster.

hadoop@pi0 ~/hadoop/bin $ scp hadoop hadoop@pi1:hadoop/bin/. 
hadoop       100%   14KB  13.8KB/s   00:00    
hadoop@pi0 ~/hadoop/bin $ scp hadoop hadoop@pi2:hadoop/bin/. 
hadoop       100%   14KB  13.8KB/s   00:00    
hadoop@pi0 ~/hadoop/bin $ scp hadoop hadoop@pi3:hadoop/bin/. 
hadoop       100%   14KB  13.8KB/s   00:00    
hadoop@pi0 ~/hadoop/bin $ scp hadoop hadoop@pi4:hadoop/bin/. 
hadoop       100%   14KB  13.8KB/s   00:00    


Now we're ready to get things up and running! We need to format the NameNode this sets up the storage structure HDFS. This is done with the bin/hadoop namenode -format command as follows:

hadoop@pi0 ~/hadoop $ bin/hadoop namenode -format 
14/11/04 10:53:28 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************ 
STARTUP_MSG: Starting NameNode 
STARTUP_MSG:   host = pi0/10.1.10.120 
STARTUP_MSG:   args = [-format] 
STARTUP_MSG:   version = 1.1.1 
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1411108; compiled by 'hortonfo' on Mon Nov 19 10:48:11 UTC 2012 
************************************************************/ 
14/11/04 10:53:30 INFO util.GSet: VM type       = 32-bit 
14/11/04 10:53:30 INFO util.GSet: 2% max memory = 19.335 MB 
14/11/04 10:53:30 INFO util.GSet: capacity      = 2^22 = 4194304 entries 
14/11/04 10:53:30 INFO util.GSet: recommended=4194304, actual=4194304 
14/11/04 10:53:34 INFO namenode.FSNamesystem: fsOwner=hadoop 
14/11/04 10:53:35 INFO namenode.FSNamesystem: supergroup=supergroup 
14/11/04 10:53:35 INFO namenode.FSNamesystem: isPermissionEnabled=true 
14/11/04 10:53:35 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 
14/11/04 10:53:35 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 
14/11/04 10:53:35 INFO namenode.NameNode: Caching file names occuring more than 10 times 
14/11/04 10:53:37 INFO common.Storage: Image file of size 112 saved in 0 seconds. 
14/11/04 10:53:38 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/hadoop/name/current/edits 
14/11/04 10:53:38 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/hadoop/name/current/edits 
14/11/04 10:53:39 INFO common.Storage: Storage directory /home/hadoop/name has been successfully formatted. 
14/11/04 10:53:39 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************ 
SHUTDOWN_MSG: Shutting down NameNode at pi0/10.1.10.120 
************************************************************/
 
 
Since we set up the SSH login niceness before, we cam start the entire cluster now with a single command!
 
hadoop@pi0 ~/hadoop $ bin/start-all.sh 
starting namenode, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-namenode-pi0.out 
pi1: starting datanode, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-datanode-pi1.out 
pi3: starting datanode, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-datanode-pi3.out 
pi4: starting datanode, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-datanode-pi4.out 
pi2: starting datanode, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-datanode-pi2.out 
pi0: starting secondarynamenode, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-secondarynamenode-pi0.out 
starting jobtracker, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-jobtracker-pi0.out 
pi1: starting tasktracker, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-tasktracker-pi1.out 
pi2: starting tasktracker, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-tasktracker-pi2.out 
pi4: starting tasktracker, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-tasktracker-pi4.out 
pi3: starting tasktracker, logging to /home/hadoop/hadoop-1.1.1/libexec/../logs/hadoop-hadoop-tasktracker-pi3.out 


Open up a browser and go to the following URLs: pi0 is the master or namenode in your cluster.
http://pi0:50070/dfshealth.jsp
http://pi0:50030/jobtracker.jsp

Testing the Setup

Now we can run something simple and quick and make sure things are working.

hadoop@pi0 ~/hadoop $ bin/hadoop jar hadoop-examples-1.1.1.jar pi 4 1000 
Number of Maps  = 4 
Samples per Map = 1000 
Wrote input for Map #0 
Wrote input for Map #1 
Wrote input for Map #2 
Wrote input for Map #3 
Starting Job 
14/11/04 11:50:50 INFO mapred.FileInputFormat: Total input paths to process : 4 
14/11/04 11:50:55 INFO mapred.JobClient: Running job: job_201411041127_0001 
14/11/04 11:50:56 INFO mapred.JobClient:  map 0% reduce 0% 
14/11/04 11:51:49 INFO mapred.JobClient:  map 25% reduce 0% 
14/11/04 11:51:54 INFO mapred.JobClient:  map 100% reduce 0% 
14/11/04 11:52:22 INFO mapred.JobClient:  map 100% reduce 66% 
14/11/04 11:52:25 INFO mapred.JobClient:  map 100% reduce 100% 
14/11/04 11:52:44 INFO mapred.JobClient: Job complete: job_201411041127_0001 
14/11/04 11:52:44 INFO mapred.JobClient: Counters: 30 
14/11/04 11:52:44 INFO mapred.JobClient:   Job Counters 
14/11/04 11:52:44 INFO mapred.JobClient:     Launched reduce tasks=1 
14/11/04 11:52:44 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=129653 
14/11/04 11:52:44 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0 
14/11/04 11:52:44 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0 
14/11/04 11:52:44 INFO mapred.JobClient:     Launched map tasks=4 
14/11/04 11:52:44 INFO mapred.JobClient:     Data-local map tasks=4 
14/11/04 11:52:44 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=34464 
14/11/04 11:52:44 INFO mapred.JobClient:   File Input Format Counters 
14/11/04 11:52:44 INFO mapred.JobClient:     Bytes Read=472 
14/11/04 11:52:44 INFO mapred.JobClient:   File Output Format Counters 
14/11/04 11:52:44 INFO mapred.JobClient:     Bytes Written=97 
14/11/04 11:52:44 INFO mapred.JobClient:   FileSystemCounters 
14/11/04 11:52:44 INFO mapred.JobClient:     FILE_BYTES_READ=94 
14/11/04 11:52:44 INFO mapred.JobClient:     HDFS_BYTES_READ=956 
14/11/04 11:52:44 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=119665 
14/11/04 11:52:44 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=215 
14/11/04 11:52:44 INFO mapred.JobClient:   Map-Reduce Framework 
14/11/04 11:52:44 INFO mapred.JobClient:     Map output materialized bytes=112 
14/11/04 11:52:44 INFO mapred.JobClient:     Map input records=4 
14/11/04 11:52:44 INFO mapred.JobClient:     Reduce shuffle bytes=112 
14/11/04 11:52:44 INFO mapred.JobClient:     Spilled Records=16 
14/11/04 11:52:44 INFO mapred.JobClient:     Map output bytes=72 
14/11/04 11:52:44 INFO mapred.JobClient:     Total committed heap usage (bytes)=819818496 
14/11/04 11:52:44 INFO mapred.JobClient:     CPU time spent (ms)=16580 
14/11/04 11:52:44 INFO mapred.JobClient:     Map input bytes=96 
14/11/04 11:52:44 INFO mapred.JobClient:     SPLIT_RAW_BYTES=484 
14/11/04 11:52:44 INFO mapred.JobClient:     Combine input records=0 
14/11/04 11:52:44 INFO mapred.JobClient:     Reduce input records=8 
14/11/04 11:52:44 INFO mapred.JobClient:     Reduce input groups=8 
14/11/04 11:52:44 INFO mapred.JobClient:     Combine output records=0 
14/11/04 11:52:44 INFO mapred.JobClient:     Physical memory (bytes) snapshot=586375168 
14/11/04 11:52:44 INFO mapred.JobClient:     Reduce output records=0 
14/11/04 11:52:44 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=1716187136 
14/11/04 11:52:44 INFO mapred.JobClient:     Map output records=8 
Job Finished in 116.488 seconds 
Estimated value of Pi is 3.14000000000000000000