Archived copy. This page was recovered from the Internet Archive snapshot of /display/SP/LSDRT9 Characterisation of large amounts of wav audio taken on 2012-11-18. The original wiki at wiki.opf-labs.org is being decommissioned.

LSDRT9 Characterisation of large amounts of wav audio

Created by Bjarne Andersen on Sep 29, 2011 · last edited by Maureen Pennock · on Jun 29, 2012 (view history) · 3 versions

Collection:

Title	WAV with Danish Radio broadcasts, ripped audio CD’s, and SB in-house audio digitization
Description	sample of a 200 TB collection In the sample 200 Gbytes WAV (22.05khz, 16 bit) with Danish Radio broadcasts number of files: 119 average file size: 1.7 Gb largest file: 7.6 Gb 200 Gbytes WAV (44.1khz, 16 bit) with ripped audio CD’s number of files: 4500 average file size: 44 Mb largest file: 420 Mb 200 Gbytes WAV (96khz, 24 bit) from in-house audio digitization number of files: 298 average file size: 671 Mb largest file: 2 Gb
Licensing	Sample only available to SCAPE partners that sign licence available at: SCAPE Agreement Sharing testdata with Statsbiblioteket
Owner	The State and University Library, Aarhus (SB)
Dataset Location	https://scape.statsbiblioteket.dk/data/wav22khz/ https://scape.statsbiblioteket.dk/data/wav44khz/ https://scape.statsbiblioteket.dk/data/wav96khz/
Collection expert	Bjarne Andersen (SB)
Issues brainstorm	Simple Characterisation (characteristics can normally be extracted from header information in the wav format). This is needed when ingesting the collection into a DOMS.
List of Issues	IS24 Characterisation of large amounts of wav audio

Issue:

Title	IS24 Characterisation of large amounts of wav audio
Detailed description	SB holds large amounts of WAV audio (200Tb +) in different resolutions (ranging from 22Khz 16 bit to 96Khz 24 bit). Different resolutions have been choosen over the years for different reasons (equipment, budgets for storage space, quality of original media in digitisation). Before we ingest all these older collections into our new DOMS we need to do simple characterisation on the files to ensure to generate correct technical metadata (in PREMIS format) for the files. We know that cirtain collections that claim to hold only eg. 48Khz 16 bit files have files in other resolutions - most likely as a result of mis-operation of the digitisation equipment.
Scalability Challenge	Large amounts of data (200Tbytes +). For simple characterisation not much CPU is required but a lot of I/O is needed. Some of the files are rather large (8Gbytes) - could be a problem for some characterisation tools (not problematic for tools that only reads header information and magic bytes)
Issue champion	Gry Elstrøm (SB)
Other interested parties
Possible Solution approaches	Should be simple. 1. Find appropriate characterisation tool that supports wav in a suitable manner * XC*L framework seems to have good wav support - evaluate and test. 2. Ensure this tools runs within the SCAPE platform
Context
Lessons Learned
Training Needs
Datasets	WAV with Danish Radio broadcasts, ripped audio CD’s, and SB in-house audio digitization
Solutions	SO06 Use Ffprobe to characterise audio+video SO06 Use Ffprobe to characterise audio+video SO25 Rosetta v3.0 Implementation Integrated with DROID 6, JHOVE1, NLNZ tool and more...

Evaluation

Objectives	This is about scaleability and functionality
Success criteria	We will have a workflow that can process WAV (and BWF) files - also larger files up to 10Gb
Automatic measures	1. Support for both WAV and BWF 2. Support for larger files - up to 10Gb 3. Process 2Tbytes of sample content in less than 24 hours 4. 100% of the files are identified correctly 5. 100% of the files gets useful and correct characterisation output
Manual assessment	1. Sample checking of the generated characterisation output
Actual evaluations	links to acutual evaluations of this Issue/Scenario

Solutions:

Unable to render {include} Couldn't find a page to include called: SO06 Use Ffprobe to characterise Wav