Thursday, December 18, 2014

10GB Terasort Benchmark on 5-node Raspberry PI Cluster - 2H 52m 56s

We just set a new record for the 10GB terasort on a 5-node PicoCluster! We cut over an our off the benchmark time bringing the total to under 3 hours! Pretty amazing!

Hadoop job_201412181311_0002 on master User: hadoop
Job Name: TeraSort
Job File: hdfs://pi0:54310/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201412181311_0002/job.xml
Submit Host: pi0
Submit Host Address: 10.1.10.120
Job-ACLs: All users are allowed
Job Setup: Successful
Status: Succeeded
Started at: Thu Dec 18 14:54:20 MST 2014
Finished at: Thu Dec 18 17:47:16 MST 2014
Finished in: 2hrs, 52mins, 56sec
Job Cleanup: Successful

Kind% CompleteNum TasksPendingRunningCompleteKilledFailed/Killed
Task Attempts
map100.00%

80008000 / 0
reduce100.00%

80008000 / 0


Counter Map Reduce Total
Map-Reduce Framework Spilled Records 0 0 300,000,000
Map output materialized bytes 0 0 10,200,038,400
Reduce input records 0 0 100,000,000
Virtual memory (bytes) snapshot 0 0 46,356,074,496
Map input records 0 0 100,000,000
SPLIT_RAW_BYTES 8,800 0 8,800
Map output bytes 0 0 10,000,000,000
Reduce shuffle bytes 0 0 10,200,038,400
Physical memory (bytes) snapshot 0 0 32,931,528,704
Map input bytes 0 0 10,000,000,000
Reduce input groups 0 0 100,000,000
Combine output records 0 0 0
Reduce output records 0 0 100,000,000
Map output records 0 0 100,000,000
Combine input records 0 0 0
CPU time spent (ms) 0 0 27,827,080
Total committed heap usage (bytes) 0 0 32,344,113,152
File Input Format Counters Bytes Read 0 0 10,000,144,320
FileSystemCounters HDFS_BYTES_READ 10,000,153,120 0 10,000,153,120
FILE_BYTES_WRITTEN 20,404,679,750 10,204,290,230 30,608,969,980
FILE_BYTES_READ 10,265,248,834 10,200,000,960 20,465,249,794
HDFS_BYTES_WRITTEN 0 10,000,000,000 10,000,000,000
File Output Format Counters Bytes Written 0 0 10,000,000,000
Job Counters Launched map tasks 0 0 80
Launched reduce tasks 0 0 80
SLOTS_MILLIS_REDUCES 0 0 28,079,434
Total time spent by all reduces waiting after reserving slots (ms) 0 0 0
SLOTS_MILLIS_MAPS 0 0 22,051,330
Total time spent by all maps waiting after reserving slots (ms) 0 0 0
Rack-local map tasks 0 0 30
Data-local map tasks 0 0 50

Map Completion Graph - close

Reduce Completion Graph - close



Go back to JobTracker

This is Apache Hadoop release 1.2.1

No comments:

Post a Comment