Archived copy. This page was recovered from the Internet Archive snapshot of /display/SP/WCT1 Comparison of Web Archive with Live Sites taken on 2013-01-13. The original wiki at wiki.opf-labs.org is being decommissioned.

WCT1 Comparison of Web Archive with Live Sites

Added by Peter Cliff · last edited by Peter Cliff · on Jan 11, 2013

User Story

In order to be confident that we have preserved a website we need a digital preservation system that can automate the comparison of the harvested copy (WARC/ARC file) to the live site and provide a similarity score which can be used to assess the quality of the copy.

Component(?) Stories

I need a tool to generate an image of a live Web page so that I can compare this image with the harvested copy.

I need a tool to generate an image of a Web page found in a WARC/ARC file so that I can compare this image with the live copy.

I need a tool to compare two images and provide a similarity score so I can assess how closely the live site and the harvested site match visually.

User Requirements

MUST be able to read WARC files
SHOULD be able to read ARC files
MUST continue after any network downtime
MUST be able to be throttled to meet site access requirements
SHOULD be robots.txt aware
Similarity score MUST be normalised between 0 and 1, where 1 is identical and 0 is no similarity at all
WARC comparison MUST occur within the update frequency of the live website