Archived copy. This page was recovered from the Internet Archive snapshot of /display/AQuA/Identifying the content of Email Mailboxes - Solution taken on 2011-06-28. The original wiki at wiki.opf-labs.org is being decommissioned.

Identifying the content of Email Mailboxes - Solution

Created by Mette van Essen on Jun 15, 2011 · last edited by Paul Wheatley · on Jun 15, 2011 (view history) · 11 versions

One line summary	A Python script using the standard Python mbox handler can be used to count individual mail messages, export messages and attachments. The script calls DROID to identify the exported files after which we read the results and output a report listing the number of messages and the number of messages with attachments, showing which attachments belong to which mail.
Detailed description	Step 1: To obtain an accurate count of the individual email messages contained within email mailbox files Step 2: Provide accurate identification of any email attachments also contained within mailbox files Step 3: Provide automatic reporting of the above We concentrated on the .mbox file format, although we believe that the methods employed could also be used for other mailbox formats Solution: Message Count: Analysed internal bitstream of .mbox file, using visual checking in hex editor, and reference to mbox specification documentation (http://tools.ietf.org/html/rfc4155). Identified recurring start sequences for each message. The identification of such a sequence enabled us to develop an automated method of counting, based on the number of times that sequence appeared in the file. Developed a Python script (add link to script), which was able to run across the mbox file, counting individual messages, and return a total. Initially, count returned did not match expected number of email messages which were present when mbox file was opened in Eudora. Further testing was undertaken to verify count. Mbox file was opened in Thunderbird, Eudora OSE, Eudora Light 3.0.3 (where the source file came from), separate email analysis script http://www.stanford.edu/~pgbovine/mbox-analysis.htm - all resulted in same count as the Python script Further investigation revealed that source file in Eudora did contain the same number of email messages as the copy file analysed, but that certain messages had been "deleted" (or sent to Trash folder) in Eudora, and did not display in the user interface view, while still being present in the actual file (which opens up a whole new area of investigation... Accurate Identification of attachments: Using email start sequence bitstream (as described above), a new signature has been created for.mbox files for the DROID file format identification tool. Although not yet complete a Python script will be used to extract mail and attachments and to execute DROID on those files. The DROID output will be parsed and used to generate a report. Providing reporting tools: The script will produce an aggregate report providing some of the stats we've taken from the standard Python email handling library and the results of identification via DROID.
Solution champion	Andrew & Ross
Git link
Evaluation	Solution champion keen to do some further work on the script for the issue champion and provide full report of attachment breakdown, in the next week or two Would the "deleted" emails be part of the collection? Obviously depends on collection policy, but the tool has potential to enable filtering There is hope that a solution here would have some applicability for scanning other mailbox formats
Tool (link)