Powered by OPF
OPF WIKI STATIC ARCHIVE
See the original ↗Page history
2,681 pages · 153 spaces · 776 tags · 4,025 history records · 96.2% of the original wiki recovered
Archived copy. This page was recovered from the Internet Archive snapshot of /display/SP/raw2nexus migration large dataset copied from small dataset taken on 2016-09-11. The original wiki at wiki.opf-labs.org is being decommissioned.

raw2nexus migration large dataset copied from small dataset

Created by Alastair Duncan on Jun 23, 2014 · last edited by Alastair Duncan · on Jun 30, 2014 (view history) · 14 versions

Evaluator(s)

Alastair Duncan

Evaluation points

Metric Description Metric baseline Metric goal 19/06/2014
Maps per node 4 Split 1
20/06/2014
Maps per node 8 Split 1 		23/24/06/2014
Maps per Node 1 split 1 		25/06/2014
Maps per Node 1 split 33 		26/06/2014
Maps per Node 4 split 33
27/06/2014
Maps per Node 8 split 33
NumberOfObjectsPerHour Number of objects processed in one hour     1667.09 1789.48 874.16
1518.95 1746.88
2820.24
MaxObjectSizeHandledInGbytes Max size of raw files     0.16689453 0.16689453 0.16689453 0.16689453 0.16689453 0.16689453
MinObjectSizeHandledInGbytes Min size of raw files     2.24113e-5 2.24113e-5 2.24113e-5 2.24113e-5 2.24113e-5 2.24113e-5
ThroughputGbytesPerMinute The throughput of data measured in Gybtes per minute     0.856
0.919 0.449 0.780 0.897
1.45
ThroughputGbytesPerHour The throughput of data measured in Gybtes per hour     51.379 55.151 26.94 46.81 53.84
86.91
ReliableAndStableAssessment Manual assessment on if the experiment performed reliable and stable     true true true true true
true
NumberOfFailedFiles Number of files that failed in the workflow     1000 1000 1000 1000 1000 1000
AverageRuntimePerItemInSeconds The average processing time in seconds per item     2.16 2.01 4.12 2.37 2.06
1.28
throughput in bytes per second The throughput of data measured in bytes per second     15324467.037 16449541.634 8035585.716 13962718.892 16057937.712
25924702.545
number of  objects per second Number of objects that can be processed per second     0.46
0.49
0.24
0.42 0.48
0.78
max object size handled in bytes Max size of raw files     160312320 160312320 160312320 160312320 160312320 160312320
min object size handled in bytes Min size of raw files     24064 24064 24064 24064 24064 24064
WebDAV logs and configuration

http://fue.onb.ac.at/scape-tb-evaluation/stfc/raw2nexus/largedatasetmap8split1/

http://fue.onb.ac.at/scape-tb-evaluation/stfc/raw2nexus/largedatasetmap4split1/

http://fue.onb.ac.at/scape-tb-evaluation/stfc/raw2nexus/largedatasetmap1split1/

http://fue.onb.ac.at/scape-tb-evaluation/stfc/raw2nexus/largedatasetmap1split33/

http://fue.onb.ac.at/scape-tb-evaluation/stfc/raw2nexus/largedatasetmap4split33/

http://fue.onb.ac.at/scape-tb-evaluation/stfc/raw2nexus/largedatasetmap8split33/

Evaluation notes

Timings for creating the copied dataset and generating the input files for ToMaR are not included in the results above.

There are 1000 failed executions because one of the test set(33 files) of files has a metadata file missing. The data set is 1000 copies of the 33 files.