Archived copy. This page was recovered from the Internet Archive snapshot of /display/SP/Fedora 4 Ingest Throughput taken on 2016-09-11. The original wiki at wiki.opf-labs.org is being decommissioned.

Fedora 4 Ingest Throughput

Created by Matthias Hahn on May 07, 2014 · last edited by Matthias Hahn · on May 07, 2014 (view history) · 5 versions

Investigator(s)

Frank Asseg, frank.asseg(at)fiz-karlsruhe.de
Matthias Hahn, matthias.hahn(at)fit-karlsruhe.de

Dataset

The Dataset used was a random set with different file sizes

Platform

We tested on diverse Cluster at the Steinbuch Center for Computing SCC, on a Amazon AWS instance, on a local Cluster at FIZ Karlsruhe and on a Cluster at Timisoara (a SCAPE Partner)

Workflow

We first deployed Fedora 4 in clustered mode on the Cluster Nodes with the help of Shell- and Puppet scripts to do this efficiently. We used a benchtool to measure the ingest performance of the datasets.

Evaluation Summaries

Links to results of the experiment using the evaluation template.

Experiment Fedora 4 at SCC Cluster: https://wiki.duraspace.org/display/FF/Performance+evaluation+on+the+SCC+Cluster

Experiment Fedora 4 at AWS Cluster: https://wiki.duraspace.org/display/FF/Performance+evaluation+on+AWS

Experiment Fedora 4 at FIZ Cluster: https://wiki.duraspace.org/display/FF/Performance+evaluation+on+the+FIZ+cluster

Experiment Fedora 4 at Timisoar Cluster: for this test we do not have a Wiki Page at Duraspace, instead we provide the numbers here:

The cluster is currently running on these three nodes:
   http://scape-fiz-1.info.uvt.ro:8080/fcrepo/rest
   http://scape-fiz-3.info.uvt.ro:8080/fcrepo/rest
   http://scape-fiz-4.info.uvt.ro:8080/fcrepo/rest

The first small ingest test of a single file with 10m xielded this result:
17:15 INFO Found Fedora 4 at http://localhost:8080/fcrepo
16:17:15 INFO Running 1 INGEST action(s) against FCREPO4 with a binary size of 10.0 MB using 1 thread(s)
16:17:16 INFO The Fedora cluster has 3 node(s) before the benchmark
16:17:16 INFO preparing 1 objects
16:17:16 INFO creating 1 objects took 57 ms
16:37:16 INFO scheduling 1 actions
17:08:55 INFO purging 1 objects and datastreams
17:08:55 INFO Completed 1 INGEST action(s) executed in 1898784 ms
17:08:55 INFO The Fedora cluster has 3 node(s) after the benchmark
17:08:55 INFO Throughput was 0.01 MB/sec
17:08:55 INFO Condensed results:
17:08:55 INFO 1 10485760 1 INGEST 1898784 0.0052665286 no-tx
17:08:55 INFO All operations completed in 3099875 ms

So it did work but the throughput with one file was only 0.1mb/s.
Running a larger test ingesting 100 Objects with 10mb each gives the same result.

The profile of the cluster is available here:
https://wiki.duraspace.org/pages/viewpage.action?pageId=50528479