Thursday, November 13, 2014

10GB Terasort Benchmark on 5-node Raspberry PI Cluster - 4H 12m 5s

After more than a week of testing, tweeking, and retesting, we were able to successfully run a 10GB Terasort on a 5-node Raspberry PI cluster! Each node has 512MB ram and a 16GB SD card.

Hadoop job_201411131351_0001 on master

User: hadoop
Job Name: TeraSort
Job File: hdfs://pi0:54310/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201411131351_0001/job.xml
Submit Host: pi0
Submit Host Address: 10.1.10.120
Job-ACLs: All users are allowed
Job Setup: Successful
Status: Succeeded
Started at: Thu Nov 13 14:23:35 MST 2014
Finished at: Thu Nov 13 18:35:40 MST 2014
Finished in: 4hrs, 12mins, 5sec
Job Cleanup: Successful

Kind% CompleteNum TasksPendingRunningCompleteKilledFailed/Killed
Task Attempts
map100.00%

1520015200 / 0
reduce100.00%

1520015200 / 0


Counter Map Reduce Total
File Input Format Counters Bytes Read 0 0 10,000,298,372
Job Counters SLOTS_MILLIS_MAPS 0 0 24,993,499
Launched reduce tasks 0 0 152
Total time spent by all reduces waiting after reserving slots (ms) 0 0 0
Rack-local map tasks 0 0 144
Total time spent by all maps waiting after reserving slots (ms) 0 0 0
Launched map tasks 0 0 152
Data-local map tasks 0 0 8
SLOTS_MILLIS_REDUCES 0 0 34,824,665
File Output Format Counters Bytes Written 0 0 10,000,000,000
FileSystemCounters FILE_BYTES_READ 10,341,496,856 10,200,000,912 20,541,497,768
HDFS_BYTES_READ 10,000,315,092 0 10,000,315,092
FILE_BYTES_WRITTEN 20,409,243,506 10,208,123,719 30,617,367,225
HDFS_BYTES_WRITTEN 0 10,000,000,000 10,000,000,000
Map-Reduce Framework Map output materialized bytes 0 0 10,200,138,624
Map input records 0 0 100,000,000
Reduce shuffle bytes 0 0 10,200,138,624
Spilled Records 0 0 300,000,000
Map output bytes 0 0 10,000,000,000
Total committed heap usage (bytes) 0 0 57,912,754,176
CPU time spent (ms) 0 0 40,328,090
Map input bytes 0 0 10,000,000,000
SPLIT_RAW_BYTES 16,720 0 16,720
Combine input records 0 0 0
Reduce input records 0 0 100,000,000
Reduce input groups 0 0 100,000,000
Combine output records 0 0 0
Physical memory (bytes) snapshot 0 0 52,945,952,768
Reduce output records 0 0 100,000,000
Virtual memory (bytes) snapshot 0 0 123,024,928,768
Map output records 0 0 100,000,000

Map Completion Graph
Reduce Completion Graph


Go back to JobTracker

This is Apache Hadoop release 1.2.1

No comments:

Post a Comment