Archived copy. This page was recovered from the Internet Archive snapshot of /pages/viewpage.action?pageId=36012111 taken on 2022-09-30. The original wiki at wiki.opf-labs.org is being decommissioned.

PIG User Defined Functions to operate on extracted web content

Added by Sven Schlarb · last edited by Per Møldrup-Dalum · on Dec 03, 2013

Working plan

Pig example

For now clone https://github.com/perdalum/warcbase and checkout the pig-integration branch. Running the unit tests will run the above Pig Latin script on the provided test gzip'ed ARC file. The language distribution reported by Tika is:

Goals

Create UDFs for

language detection using Tika
identification using Tika