Teamware: A Collaborative, Teamware: A Collaborative, Web-based - - PowerPoint PPT Presentation

▶
teamware a collaborative teamware a collaborative web
SMART_READER_LITE
LIVE PREVIEW

Teamware: A Collaborative, Teamware: A Collaborative, Web-based - - PowerPoint PPT Presentation

Aug 27, 2022 276 likes •576 views

Teamware: A Collaborative, Teamware: A Collaborative, Web-based Annotation Environment Web-based Annotation Environment Kalina Bontcheva, Milan Agatonovic University of Sheffield University of Sheffield NLP Outline Outline Why Teamware?

slide-1
SLIDE 1

Teamware: A Collaborative, Teamware: A Collaborative, Web-based Annotation Environment Web-based Annotation Environment

Kalina Bontcheva, Milan Agatonovic University of Sheffield

slide-2
SLIDE 2

University of Sheffield NLP

2 GATE Summer School - July 27-31, 2009

Outline Outline

  • Why Teamware?
  • What’s Teamware?
  • Teamware for annotation
  • Teamware for quality assurance and curation
  • Teamware for defining workflows, running

automatic services, managing annotation projects

  • Outlook
slide-3
SLIDE 3

University of Sheffield NLP

3 GATE Summer School - July 27-31, 2009

From Annotation Tools to From Annotation Tools to Collaborative Annotation Workflows Collaborative Annotation Workflows

We have lots and lots of tools and algorithms for annotation; what we need is

1.methodological instead of purely technological 2.multi-role instead of single role 3.assistive instead of autonomous 4.service-orientated, not monolithic 5.usable by non-specialists

GATE Teamware

Research users in several EU projects External users at IRF and Matrixware Interest from other commercial users as well

slide-4
SLIDE 4

University of Sheffield NLP

4 GATE Summer School - July 27-31, 2009

GATE Teamware: Annotation GATE Teamware: Annotation Workflows on the Web Workflows on the Web

GATE Teamware is:

□Collaborative, social, Web 2.0, has behaviour

mining using Machine Learning

□Parallel and distributed (using web services) □Scalable (via service replication) □Workflow based with business process

integration via BPEL

slide-5
SLIDE 5

University of Sheffield NLP

5 GATE Summer School - July 27-31, 2009

Teamware – Teamware – Layer Cake Layer Cake

Teamware Executive Layer Workflow Management Authentication And User Management Services Layer GATE Document Service GATE Annotation Services GATE Ontology Service GATE Machine Learning API User Interface Layer Manual Annotation User Interface

Schema Annotation UI Ontology Annotation UI

Data Curation User Interface

Annotation Diff UI ANNIC UI Document Browser

Language Engineer User Interface

GATE Developer UI

slide-6
SLIDE 6

University of Sheffield NLP

6 GATE Summer School - July 27-31, 2009

Division of Labour: Division of Labour: A Multi-role A Multi-role Methodology Methodology

  • (Human) Annotators - labour has to be cheap!

Bootstrap annotation process with JAPE rules or mixed-initiative learning

  • Curators (or super-annotators)

Reconcile differences between annotators, using IAA, AnnDiff, curator UI

Manager

Defining annotation guidelines and schemas Choose relevant automatic services to pre-process Toolset including performance benchmarking, progress monitoring

tools, small linguistic customisations

Define workflow, manage annotators, liaise with language engineers and

sys admins

  • Sys admin

Setup the Teamware system, users, etc.

Language engineer

Uses GATE Developer to create bespoke services and deploy online

slide-7
SLIDE 7

University of Sheffield NLP

7 GATE Summer School - July 27-31, 2009

Teamware: Teamware: Manual Annotation Tool Manual Annotation Tool

slide-8
SLIDE 8

University of Sheffield NLP

8 GATE Summer School - July 27-31, 2009

Manual Annotation Process Manual Annotation Process

  • Annotator logs into Teamware
  • Clicks on “Open Annotation Editor”
  • Requests an annotation task (first button)
  • Annotates the assigned document
  • When done, presses the “Finish task” button
  • If wants to save work and return to this task later –

“Save” button, then close the UI. Next time a task is requested, the same document will be assigned, so it can be finished

  • Depending on the project setup, it might be

possible to reject a document and then ask for another one to annotate (Reject button)

slide-9
SLIDE 9

University of Sheffield NLP

9 GATE Summer School - July 27-31, 2009

Hands-on Hands-on

  • Open a web browser and Teamware
  • Login using you user name (from your reg.pack):

<cics-account-id>-annotator

  • Open the annotation UI
  • Try requesting tasks, editing annotations,

saving your work, asking for another task, etc.

  • This is what Teamware looks like to a human

annotator

slide-10
SLIDE 10

University of Sheffield NLP

10 GATE Summer School - July 27-31, 2009

Teamware for Curators Teamware for Curators

  • Still being developed, so UI is in transition
  • Identify if there are differences between

annotators using IAA

  • Inspect differences in detail using AnnDiff
  • Edit and reconcile differences if required

New curator UI in Teamware under development Currently available in Developer

slide-11
SLIDE 11

University of Sheffield NLP

11 GATE Summer School - July 27-31, 2009

IAA: Do my annotators agree? IAA: Do my annotators agree?

slide-12
SLIDE 12

University of Sheffield NLP

12 GATE Summer School - July 27-31, 2009

IAA: Results IAA: Results

slide-13
SLIDE 13

University of Sheffield NLP

13 GATE Summer School - July 27-31, 2009

IAA: Recap IAA: Recap

  • The IAA on IE tasks, such as named entity

recognition, should be measured using f-measure across all annotators

  • For classification tasks, use Kappa to

measure IAA

  • For details, see the evaluation lecture and

the GATE user guide

slide-14
SLIDE 14

University of Sheffield NLP

14 GATE Summer School - July 27-31, 2009

AnnDiff: Finding the differences AnnDiff: Finding the differences

slide-15
SLIDE 15

University of Sheffield NLP

15 GATE Summer School - July 27-31, 2009

Where are these in Teamware? Where are these in Teamware?

  • Only visible to curators and their managers
  • Resources/Documents menu
  • Select the corpus worked on
  • Iterate through each document
  • Run IAA and AnnDiff, as required
  • Try for yourself:

Login as <cics-user-name>-curator Corpus: annie-demo The first or second documents

slide-16
SLIDE 16

University of Sheffield NLP

16 GATE Summer School - July 27-31, 2009

Forthcoming curator facilities Forthcoming curator facilities

  • Have a corpus-level view of IAA
  • Extended AnnDiff to allow easy reconciliation
  • f the differences between 2 annotators
  • Currently prototyped in Developer
  • Will be made available in Teamware soon
slide-17
SLIDE 17

University of Sheffield NLP

17 GATE Summer School - July 27-31, 2009

New AnnDiff in Developer New AnnDiff in Developer

slide-18
SLIDE 18

University of Sheffield NLP

18 GATE Summer School - July 27-31, 2009

Beyond Pair-wise Reconciliation Beyond Pair-wise Reconciliation

  • AnnDiff only handles 2 sets of annotations at

a time – we often need more!

  • Towards an in-place, content-based

reconciliation interface

slide-19
SLIDE 19

University of Sheffield NLP

19 GATE Summer School - July 27-31, 2009

Current UI Prototype Current UI Prototype

slide-20
SLIDE 20

University of Sheffield NLP

20 GATE Summer School - July 27-31, 2009

Teamware for Managers Teamware for Managers

  • Defining workflows
  • Running annotation projects
  • Tracking progress
slide-21
SLIDE 21

University of Sheffield NLP

21 GATE Summer School - July 27-31, 2009

Teamware Workflows Teamware Workflows

  • Whole process is controlled by a workflow manager
  • Workflow may be simple:

Give the document to a human annotator Information curator checks a sample of documents for QC

  • or more complex

Invoke one or more web services to produce automatic annotations Pass each document to 2 annotators Information curator to quickly check level of agreement between

the annotators and reconcile any differences

Annotated documents used to train an ML model When model is good enough, start making suggestions to the

annotators

slide-22
SLIDE 22

University of Sheffield NLP

22 GATE Summer School - July 27-31, 2009

Workflow Templates Workflow Templates

slide-23
SLIDE 23

University of Sheffield NLP

23 GATE Summer School - July 27-31, 2009

Defining new workflows Defining new workflows

  • Select Projects/WF Templates
  • Opens the WF wizard
  • Choose which services you want to run
  • Choose whether you want manual

annotation, how many annotators per doc, …

slide-24
SLIDE 24

University of Sheffield NLP

24 GATE Summer School - July 27-31, 2009

Setting up a Manual Annotation Setting up a Manual Annotation Project Project

  • Upload the schemas
  • Upload the documents
  • Define the Workflow template
  • Run the project, choosing the corpus, the

annotators, curators, etc.

  • DEMO!
slide-25
SLIDE 25

University of Sheffield NLP

25 GATE Summer School - July 27-31, 2009

Setting up an Automatic Setting up an Automatic Annotation Project Annotation Project

  • Configure the web service(s)
  • Define the Workflow template
  • Run the project, choosing the corpus
  • DEMO!
slide-26
SLIDE 26

University of Sheffield NLP

26 GATE Summer School - July 27-31, 2009

Semi-automatic Projects Semi-automatic Projects

  • Just combine the two sets of steps
slide-27
SLIDE 27

University of Sheffield NLP

27 GATE Summer School - July 27-31, 2009

Teamware: Teamware: Monitoring Project Progress Monitoring Project Progress

slide-28
SLIDE 28

University of Sheffield NLP

28 GATE Summer School - July 27-31, 2009

Outlook Outlook

  • Teamware is still under active development
  • Many features subject to change
  • If you’d like further information or to try it with

your data for a particular project, please contact Hamish and Kalina