Powered by OPF
OPF WIKI STATIC ARCHIVE
See the original ↗Page history
2,681 pages · 153 spaces · 776 tags · 4,025 history records · 96.2% of the original wiki recovered
Archived copy. This page was recovered from the Internet Archive snapshot of /display/SP/Validate JPEG2000 Newspapers Using Jpylyzer taken on 2016-09-11. The original wiki at wiki.opf-labs.org is being decommissioned.

Validate JPEG2000 Newspapers Using Jpylyzer

Created by Bolette Ammitzbøll Jurik on Dec 06, 2013 · last edited by Rune Bruun Ferneke-Nielsen · on Sep 26, 2014 (view history) · 34 versions

Investigator(s)

Rune Ferneke-Nielsen (SB)

Dataset

Danish newspaper - Morgenavisen Jyllandsposten

Platform

SB Hadoop Platform

Workflow

The idea behind this experiment is that you have a digital newspaper collection, in JPEG 2000 format, and you want to verify that certain properties hold true for every file in the collection. The properties that should hold true are specified in a control policy, which at least contains information about the digital newspaper collection.

Iteration 1

The first iteration of the experiment will use a very simple setup and focus on processing the files using jpylyzer - we would like to get a first indication of the performance without bringing extra complexity into the equation. Therefore, files will be read from local storage instead of using our repositories as would normally be the case. Moreover, output from the processing will be discarded - failing processes being the exception - instead of being stored in our repositories. The Hadoop configuration has not been altered, apart from the necessary settings that corresponds to our cluster.

About the details

First step will extract meta data, using jpylyzer, from each file in the newspaper collection. Second step will compare the extracted meta-data against the control policy, and report on any differences.

Considerations

  1. Where are newspaper collection files stored - in a repository, on local storage (outside of Hadoop), hdfs storage?
  2. What is the appropriate number of concurrently running tasks? One way to handle this is by specifying split size, which will determine how many map tasks to start. Moreover, jpylyzer is able to handle several input paths, which is kind of a second level in handling concurrently running tasks. Specification of hardware should off course be taken into consideration in this discussion (available nodes, cpu cores & threads, memory, e.t.c.).
  3. Should generated meta-data be stored, and where - in a repository, on local storage, hdfs storage, discard it?

Improvements & suggestions

Iteration 2

Building upon the results from the first iteration, we want the experiment to reflect our reality more. We extend the setup further by adding repositories, where data will be read from and written to. In details, we will use a Fedora-based repository for reading and writing content meta-data, and a bit repository for reading content. By adding these systems, we need to extend the experiment with components that can load and store data in an efficient manner.

About the details

The environment has been extending to also include the two repositories, containing metadata and content of the images.

  1. Extracting metadata from Fedora-based repository
  2. Performing quality assurance on Hadoop platform
  3. Storing metadata into Fedora-based repository

Step 1 can be split into:

Step 2 can be split into:

Step 3 can be split into:

The SCAPE Stager & Loader components
Extracting and storing data in the repository is handled by the SCAPE Stager and Loader components, which both interact with the repository through a SCAPE Data Connector.

Requirements and Policies

Policy statements that relate to this experiment and any evaluation criteria taken from SCAPE metrics

Evaluations