Archived copy. This page was recovered from the Internet Archive snapshot of /display/SP/WCT2 Comparison of Web Archive with Quality Assured Snapshot taken on 2013-01-13. The original wiki at wiki.opf-labs.org is being decommissioned.

WCT2 Comparison of Web Archive with Quality Assured Snapshot

Added by Peter Cliff · last edited by Peter Cliff · on Jan 11, 2013

User Story

In order to be confident that we have preserved a website we need a digital preservation system that can automate the comparison of the harvested copy (WARC/ARC file) to a previous harvested copy that has been manually verified as an accurate representation of the site and provide a similarity score which can be used to assess the quality of the copy.

Component(?) Stories

I need a tool to generate an image of a Web page found in a WARC/ARC file so that I can compare this image with the live copy.

I need a tool to compare two images and provide a similarity score so I can assess how closely the live site and the harvested site match visually.

User Requirements

MUST be able to read WARC files
SHOULD be able to read ARC files
MUST continue after any network downtime
SHOULD be robots.txt aware
Similarity score MUST be normalised between 0 and 1, where 1 is identical and 0 is no similarity at all
WARC comparison MUST occur within the update frequency of the live website