Archived copy. This page was recovered from the Internet Archive snapshot of /display/AQuA/Scenario D - Fingerprinting for deduplication taken on 2011-05-09. The original wiki at wiki.opf-labs.org is being decommissioned.

Scenario D - Fingerprinting for deduplication

Added by Becky McGuinness · last edited by Maurice de Rooij · on Apr 18, 2011 (view change)

AQuA scenario summary	Fingerprinting for deduplication
AQuA collection champion	Jodie
AQuA techie / other involved parties	Maurice
Basic description of a collection (or collections) that this scenario relates to	eg. 1000 19th century digitised newspaper issues. Tiff masters, JPEG service images, METS metadata
Licensing details for the related collections	Leeds content will require clearance from collection owner
Institution (owner of the content)	BL and Leeds
The scenario's issues, challenges or collection problems to be addressed. How does, or might, the problem manifest itself? What is the implication of the problem? Is manual QA sufficient to catch this problem?	Digitising large quantities of material will inherently create an occassional duplicated page or image. Books in particular when manually scanned may have a duplicated page on occassion. Catching these errors prior to publishing is a time consuming manual process. Problems arise due to operator error or machine error. The implication of the problem is the final package is incomplete and not a representation of the physical object. An additional scenario is a duplicated image as a result of depositing content twice. This creates a confusing experience for the user .... hmmmmm will have a think on this one and write some more....
Possible approaches to solving the scenario problem. This might include how to do detect the problems or how to fix them.	Converting images to fingerprints using Wavelet Scalar Quantization (WSQ) or 8-bit ASCII. Comparing images in memory using Perceptual Image Difference Tool
Context to the scenario problem and/or solution. How does the scenario fit within the owning institution's wider activities and workflows? Are there any technical constraints on the solutions?	With the increase in self-deposit of collections and content duplication and embedded metadata will be an increasing issue to manage in the repositories. Current systems do not support this workflow but is part of the development plan to add on and enhance preservation tools already in place. One of the repositories will not support this workflow but the plan is to switch platforms thereby enabling tools for support.
Summary of any AQuA solutions that were developed (include links to git repository)
Summary of AQuA event evaluation of developed solutions	Duplicate images within a collection or job Perceptual Image Diff comparison