Oral presentation at MIE 2011: Factuality Levels of Diagnoses in - - PowerPoint PPT Presentation

▶

Apr 04, 2024 133 likes •247 views

Oral presentation at MIE 2011: Factuality Levels of Diagnoses in Swedish Clinical Text Sumithra VELUPILLAI, Fil. Lic. (presenter) Hercules DALIANIS, Ph.D. Maria KVIST, M. D., PhD Oslo, August 30 th 2011 IT for Health, Dept. of Computer and

SLIDE 1

Oral presentation at MIE 2011:

Factuality Levels of Diagnoses in Swedish Clinical Text

Sumithra VELUPILLAI, Fil. Lic. (presenter) Hercules DALIANIS, Ph.D. Maria KVIST, M. D., PhD Oslo, August 30th 2011

IT for Health, Dept. of Computer and Systems Sciences (DSV), SU,

Dept. of clinical immunology and transfusion medicine, Karolinska

University Hospital

SLIDE 2

Motivation

Improve information access

– Reasoning documented in (EPR) free-text

speculations, negations, affirmations
important to distinguish

– Accurate and situation-specific information – Overviews/summaries: these diagnoses have been affirmed, negated, … – Capture reasoning à deepened knowledge

Sumithra Velupillai MIE 2011 2

SLIDE 3

Aim and Objectives

Build automated information access systems

– create an annotation schema for modeling factuality levels – applying this on Swedish clinical documentation – gain empirical understanding for qualitative analysis and use for automatic classifiers

Sumithra Velupillai MIE 2011 3

SLIDE 4

Methods

Creation of diagnosis list (~300 diagnoses)

– Kvarst susp <Diagnosis>borrelia</Diagnosis> Remains (abbr.) suspected (abbr) borrelia

Annotation guidelines and annotations

– Two senior physicians (“domain experts”)

Emergency ward, assessment entries

– Stockholm EPR Corpus

Sumithra Velupillai MIE 2011 4

SLIDE 5

Certainly ¡ ¡ Posi.ve ¡ ¡ Probably ¡ ¡ Posi.ve ¡ ¡ Possibly ¡ Posi.ve ¡ ¡ Certainly ¡ ¡ Nega.ve ¡ ¡ Probably ¡ Nega.ve ¡ ¡ Possibly ¡ Nega.ve ¡ ¡

+ ¡

‑ ¡

Sumithra Velupillai MIE 2011 5

Patient has Parkinsons disease. Physical examination strongly suggests Parkinson. Patient possibly has Parkinson. Parkinson cannot yet be outruled. No support for Parkinson. Parkinsson can be excluded.

SLIDE 6

Results

Sumithra Velupillai MIE 2011 6

Intra- and Inter-Annotator Agreement

– 0.7/0.58 F-measure, 0.73/0.6 Cohen’s κ

Certainly Positive clear majority (approx. 50%)

– High IAA: 0.9 F-measure

Possibly Negative very rare
Discrepancies in intermediate classes

SLIDE 7

0% ¡ 10% ¡ 20% ¡ 30% ¡ 40% ¡ 50% ¡ 60% ¡ 70% ¡ 80% ¡ 90% ¡ 100% ¡

Annota.on ¡classes ¡for ¡15 ¡diagnoses ¡

Sumithra Velupillai

Certainly ¡ ¡ Posi.ve ¡ ¡ Probably ¡ ¡ Posi.ve ¡ ¡ Possibly ¡ Posi.ve ¡ ¡ Certainly ¡ ¡ Nega.ve ¡ ¡ Possibly ¡ Nega.ve ¡ ¡ Probably ¡ Nega.ve ¡ ¡

MIE 2011 7

SLIDE 8

Conclusion and discussion

Functional and agreeable model for annotators
IAA results suggest that this model can be used

for developing automated systems

Different types of “cues” (not only linguistic)

Sumithra Velupillai MIE 2011 8

SLIDE 9

Where to go next?

Automatic classification

– Keep model or collapse classes? – Classification methods?

Preliminary results promising (CRF)
Qualitative analysis

– Subjective differences – Cues: linguistic, domain-specific, more?

Sumithra Velupillai MIE 2011 9

SLIDE 10

Thank you for your attention

Some key references
V. Vincze, G. Szarvas, R. Farkas, G. Móra, J. Csirik,

The Bioscope Corpus: Biomedical Texts Annotated for Uncertainty, Negation and their Scopes BMC Bioinformatics 9(S-11) (2008)

R. Saurí, J. Pustejovsky

FactBank: a corpus annotated with event factuality Language Resources & Evaluation 43 (2009), 227–268

Ideas and comments welcome! sumithra@dsv.su.se

Sumithra Velupillai MIE 2011 10