Medical visual information retrieval Henning Mller HES-SO//Valais - - PowerPoint PPT Presentation

Medical visual information retrieval

Henning Müller HES-SO//Valais Sierre, Switzerland

Overview

• Motivation for visual retrieval

– Differences between text and visual search – A model for CBIR

• Main techniques

– Context vs. content – Detection vs. retrieval – Semantics vs. free text

• Current example systems

– Springer images, Goldminer, MedSearch, Yottalook

• Upcoming challenges
• Conclusions

Motivation

• Enormous amounts of images are

being produced digitally and are available for analysis

– Part of patient records – Part of scientific articles, videos in addition to content

• New machines and protocols produce an increasing

variety of image types that require explanations

• Images are needed and often require visualization

– Thin slices, reconstruction in the brain of clinicians

• Annotated data can be used to train detection

– Data from detection can then also be used for retrieval

Differences text and visual search

• Text contains semantic information directly

– … mapping on controlled vocabularies may help retrieval

• Visual features are fairly low-level

– Colors, textures, basic shapes – Automatic segmentation is an ill-posed problem – Visual words can help solving some of the problems – Simple mapping to terminologies can also help

• Annotation of images improves retrieval quality

– But most often annotations is not of the images but rather puts the images into a specific context

Content-based image retrieval

Similarity calculation Relevance feedback Feature extraction Image feature Database

?

Content vs. context

• Visual features

describe global or regional content

– Annotation of the content exists only rarely

• Clinical records and articles often describe small

parts of the content and rather put them in context

– Which were the problems? – Why was the image taken? – Age of patient, anamnesis, …

• A partial description of the ROI is often available
• Figure captions describe also part of the content
• Both content and context are complementary

Detection vs. retrieval

• Detection

– Classification of small anomalies in usually a region of interest (ROI) or globally in the image – Annotated training data is necessary – Interpretation needs to be given, so the system is not a black box

• Retrieval

– Training data is not necessary, or not available – Search for similarity

• For clearly defined concepts or visual data
• Can be seen as a classification relevant/non-relevant

Visual words

• N-dimensional feature vector for

• Division of the feature space into

• Clustering (k-means)
• Cluster centers are the words

• Optimal number of words needs

• For each image a histogram can

• Analogy to text words

• Histogram of words for an image
• r an image region based of the

• M-Dimensional feature vector

Semantics vs. free text

• Free text is also treated before retrieval

– Stemming, stop word removal, phonetic retrieval – Translation is possible but sometimes hard – All data are readily available, little treatment

• Semantic data

– Extracted from text

• Probability to be true, relatively high for a few terms in

medical texts

– Synonyms, hypernyms, hyponyms can then be used – Existing ontologies are available (LinkedLifeData) – Understanding what particular parts are about

Goldminer.arrs.org (249,000 images)

www.springerimages.com (3.3 mio images)

medgift.hevs.ch/demos/ (231,000 images)

www.yottalook.com (70,000 images)

Upcoming challenges

• Develop simple interfaces for many types of

browsing and limiting the results set

– Faceted browsing, interactive changes, propositions – Also on small screens for mobile devices – Include 3D and 4D data

• Linking visual retrieval/detection with semantic data

– Better understanding visual data and its context – Filtering out unwanted information

• Finding small ROIs for an in depth analysis
• Treating extremely large databases
• Quickly changing modalities (limited training data)

Regions of interest are often small

Conclusions

• Images are still not fully integrated into retrieval

– Not only in the medical field – Mobile devices and detection techniques change this – Visual data is analyzed much faster than text

• Regions of interest are often very small

– Context is needed to find and analyze them – Interactivity can help as well

• Many new tools are becoming available

– LinkedLife data and semantics are required for images – Many publishers are now making also images available as these are valuable

Questions

• ?

http://www.khresmoi.eu/ http://medgift.hevs.ch/