Archived copy. This page was recovered from the Internet Archive snapshot of /pages/viewpage.action?pageId=36012136 taken on 2022-08-07. The original wiki at wiki.opf-labs.org is being decommissioned.

PIG User Defined Functions to operate on extracted web content

Created by Sven Schlarb on Dec 03, 2013 · last edited by Andrew Jackson · on Dec 04, 2013 (view history) · 13 versions

Working plan

Language detection

Pig example

For now clone https://github.com/cneud/warcbase and checkout the pig-integration branch. Running the unit tests will run the above Pig Latin script on the provided test gzip'ed ARC file. The language distribution reported by Tika is:

The UDF is added to the piggy bank of the warcbase project by adding this class that leverages Tika

MIME type detection

Goals

Create UDFs for

language detection using Tika
identification using Tika

Pig scripts

Combined script for mime type and language detection

Example output from above script: output.txt