Archived copy. This page was recovered from the Internet Archive snapshot of /display/SP/WCT EX2 File ID at SB taken on 2016-09-11. The original wiki at wiki.opf-labs.org is being decommissioned.

WCT EX2 File ID at SB

Created by Peter Cliff on Feb 22, 2013 · last edited by Rune Bruun Ferneke-Nielsen · on Jul 16, 2014 (view history) · 3 versions

Investigator(s)

Per Møldrup-Dalum, SB, [email protected]

Dataset

http://wiki.opf-labs.org/display/SP/SB+Web+Archive+Data

Platform

http://wiki.opf-labs.org/display/SP/Platform+SB+1

may have been renamed to http://wiki.opf-labs.org/display/SP/SB+Test+Platform

Workflow

Since November 2011 we have been running FITS (link to component @ myexperiment) on a selection of our web content spread over the years from 2005 up till 2011.

The data is stored in ARC files on a SAN. These ARC files are fetched from this SAN, unpacked and the FITS are run on each ARC record.

Running FITS on a ARC record produces an XML file. These XML files from a single ARC are packed into TGZ files and made available to the Planning and Watch subproject.

Requirements and Policies

ThroughputGbytesPerHour >= 60
OrganisationalFit = ?

FITS is already in use within this institution, so file format ID using FITS would be useful. However, other tools may be used.

Evaluations

http://wiki.opf-labs.org/display/SP/EVAL-WCT3-1