List of Digital Preservation Tools, Services and Specifications

Based on http://agogified.com/tools-and-services 

File format management utilities

• DROID (Digital Record Object Identification): Automated identification of file formats.
• JHOVE (JSTOR/Harvard Object Validation Environment) & JHOVE2: Provides functions to perform format-specific identification, validation, and characterization of digital objects.
• XENA (XML Electronic Normalising for Archives): Detects the file formats of digital objects; converts digital objects into open formats for preservation.  Part of NAA DSPS tool suite.
• Fido: A python implementation of the DROID identification system, using regular expressions.
• Fine Free File Command: Identifies a very wide range of files, although not focused on preservation issues.
• FITS: The File Information Tool Set (FITS) identifies, validates, and extracts technical metadata for various file formats. Aggregates results from other tools.
• NZNL Metadata Extraction Tool: See here for further information.
• JMimeMagic: Although not in development at present, this open source identifier is very close to 'file' in structure, and may provide a useful base for new developments.

File format reference

• Library of Congress Digital Formats Website: Provides information about digital content formats, including recommendations for preservation formats.
• PRONOM: Supplies technical information about the structure of those file formats, and the software products which support them.
• MIME Media Types
• Freebase: File Format
• Wikipedia: List of file formats
• GDFR, superceded by UDFR
• Note that some Operating Systems and other pieces of software (like web servers) include 'MIME Map' files, that map between file extensions and MIME Types/
• And of course Operating Systems tend to use file extension as the authoritative format identifier. Apple have a powerful hierarchical identifier scheme.
• A number of sites allow formats to be looked-up by extensions, e.g. http://www.wotsit.org/ or http://www.fileinfo.com/

File integrity utilities

• ACE (Audit Control Environment): Validates the integrity of digital files through mathematical techniques.
  • Note that Andrew Jackson uses ACE-AM to manage multi-Terabyte data prior to curation and later ingest into our digital library system.
• Checksum Checker: Monitors the contents of a digital archive for data loss or corruption.  Part of NAA DSPS tool suite.
• FastSum: Manifest creation and checking with a reasonably usable GUI.
• Manifest Maker: Supports the transfer of data objects by producing a manifest file which satisfies the requirements for a digital transfer. Part of NAA DSPS tool suite.

File transfer specifications

• BagIt:  Packaging of digital content for transfer. Content is packaged (the bag) along with a small amount of machine-readable text (the tag) to help automate the content’s receipt, storage and retrieval.
• SWORD (Simple Web-service Offering Repository Deposit):  Lightweight protocol for depositing content from one location to another.  a profile of the Atom Publishing Protocol

File transfer utilities

• BagIt Library:  Java software library that supports the creation, manipulation and validation of BagIt bags.
• BagIt Transfer Utilities: Collection of tools for validation and transfer of BagIt bags.
• GNU Wget: File transfer utility: Permits retrieval of files using HTTP, HTTPS and FTP
• rsync: Checksum-validated robust re-startable transfers of all sorts. Useful for cloning large collections reliably over a flaky connection.