/display/SP/Cross Project Integration Issues taken on 2016-09-12.
The original wiki at wiki.opf-labs.org is being decommissioned.
Cross Project Integration Issues
Contents
- Contents
- Task List
- SCAPE components TODO list
- "Things" under development
- Components specs
- Component profiles spec
- MyExperiment API
- Component plugin for taverna
- Toolwrapper - Wrapped tools from PC
- Execution module to run components on the platform
- Integration of Plato and Components
- Integration issues
Task List
SCAPE components TODO list
-
handler
Support annotation extraction and query in myExperiment (MAN)
-
handler
Implement, test and release extended myExperiment API – working on component plugin, myE API to go live this week
-
handler
Generate Components for existings tools and publish to MyExperiment (PC developers)
-
handler
Finalize component profile specification
-
handler
Validate workflow according to profile
-
handler
Review component end-2-end workflow whitepaper comments and update
-
handler
Report status of support for complex annotations to Markus P
-
handler
Extract and validate dependencies on the Platform (AIT)
-
handler
Enhance toolwrapper (KEEPS, started)
-
handler
Component Validation on Taverna (MAN)
-
handler
Clarification from Per about external post-processing step for command line tool components – Miguel talked to Per and point was taken. Post-processing will happen within the component.
-
handler
Check status of tasks in Preservation Components, Workflows, Planning and Platform integration – outdated page
-
handler
[All] Update "developments" as per Toolwrapper example
"Things" under development
Components specs
Examples of SCAPE components: https://github.com/openplanets/scape-component-profiles/tree/master/examples
Component profiles spec
The component profiles are available in the component profile repository.
The used ontology is also developed in this repository and published on http://purl.org/DP/components alongside with documentation.
MyExperiment API
(supports creating, searching and managing components)
This is anticipated to go live this week (by 25/10/13). (Documentation link.)
Component plugin for taverna
Currently testing version 1.1.0 (will release once it works!) (Browse source.)
Toolwrapper - Wrapped tools from PC
| Module name | Toolwrapper |
| Description | The toolwrapper is a tool that, based on a small specification file, is able to produce the following outputs:
|
| WP | PC.WP.2 - Action Services |
| Responsible | Miguel Ferreira (WP lead) |
Tasks
| Task | Depends of | Status | Responsible | Due date |
|---|---|---|---|---|
| Development of new version of the toolwrapper | Component examples | DONE (see github) | Hélder Silva (KEEPS) | December |
| Development of new tool that publishes SCAPE components to MyExperiment | MyExperiment API | UNDER DEVELOPMENT |
Hélder Silva (KEEPS) |
December |
References
| Useful information | Description | URL |
|---|---|---|
| SCAPE Component examples | Examples of how components should look like. These will be used as templates by the new version of the ToolWrapper so that Components are produced automatically by this tool |
Github: SCAPE Component Profiles examples |
| SCAPE Component profiles | Profiles that validate that a SCAPE component is according to the right specification. There are 3 types of profiles. |
Github: SCAPE Component Profiles |
| MyExperiment API documentation | Documentation of the API provided by MyExperiment to support the publication of Components by the new tool to be developed |
MyExperiment: Components API |
Issues
| Issue description | Reported by | Caused by | Solution owner |
|---|---|---|---|
Execution module to run components on the platform
We need feedback from Rainer.
Main question: Will we be able to submit a SCAPE component to the Job execution service and have it run on the platform.
Integration of Plato and Components
| Module name | Plato |
| Description | Preservation planning currently does
|
| WP | PW.WP.3 Automated Planning |
| Responsible | Kresimir Duretec (WP lead) |
Tasks
| Task | Depends on | Status | Responsible | Due date |
|---|---|---|---|---|
| Executable Plan Template | MyExperiment API, Component Profiles | PENDING | Markus Plangg (TUW) | December |
References
| Useful information | Description | URL |
|---|---|---|
Issues
| Issue description | Reported by | Caused by | Solution owner |
|---|---|---|---|
Integration issues
PC Requirements
On a previous PC meeting held in Paris the following whitepaper has been drafted according to PC members perspective. SCAPE_Preservation Compoments Profiles_KEEPS_V0.1.docx
The document has been shared with PW at the time.
PC/PT
How to handle non-1-to-1 migrations? (Current migration component profile handles this correctly; output files are presented component outputs, and both inputs and outputs can be singletons (=filename) or lists of singletons, so 1-many, many-1 and many-many transformations can be represented.)
PT/PW
How to express measures so that they can be assessed ("do we write this object back to the repository?")
- Proposal: Produce a common format to contain the measures. This common container format can then be combined (using some sort of standardised combiner component) so that a single document is presented to an assessment component; the boolean output of that assessment component is then used to guide what happens next (i.e., “write back to repository” or “complain to user about a failed migration”.)
- Needs basic worker components for container document construction, container document combination, container document assessment
- Container format needs to be logically a collection of triples: subject, property type, measured value
- Having an explicit subject allows reference to the input object, the output object, the comparison of the two, or anything else (e.g., intermediate transformations)
- Property types should be mentioned in relevant ontology; not sure that subjects can be in general (too many possibilities, too dependent on the structure of the overall migration workflow)
- Measured values should be simple strings; meaning determined by property type
- Container format/assessment suggestion: make it be XML and use something like Schematron to do the assessment.
- Specific format example (proposed schema):
- Note: no caution was taken over the subject and type attribute values.
- Specific suggested operations:
- Workflow Entry: SCAPE Metric Document Builder
- Workflow Entry: SCAPE Metric Document Joiner
- Workflow Entry: SCAPE Assess Metrics (note that this workflow is currently known to not work)
- Workflow Entry: SCAPE Metric Document Builder
- Consequence: The characterisation and QA components have to change to (also?) produce a metric document output.
How to get it so that PW can ask the platform to perform a migration?
- Plan, part A: It is currently planned (check with
Matthias Hahn) to have a standardised migration execution service interface that will allow a migration to be requested (probably by POSTing a description document to a service endpoint). The document will contain references to any subordinate resources required, such as:
- the migration plan to enact,
- the collection of digital objects to apply the migration plan to, and
- the acceptance criteria for judging that the plan successfully applied to a particular digital object.
- Plan, part B: It is currently planned to enhance Taverna Server to support that standardised migration execution service interface (in addition to the current service interfaces it has).
- Note that it's not currently decided what the relationship of the requested plan is to the actual workflow to be executed. Does the plan in/referred-to the request include the staging and assessment steps?
PC/PW
How to handle similarity measures?
A question is if the two ontologies/vocabularies that being developed in SCAPE are compatible or even the same:
1. The Component measures vocabulary (http://ifs.tuwien.ac.at/dp/vocabulary/quality/measures)
2. The Policies representation ontology developed in WP13.
The question is... are both of these ontology structures compatible? Let me put it in another way, are the measures defined by the first ontology compatible with the lower nodes of the second ontology (i.e. the nodes that represent operational measures of the policy representation)?