Language-Specific and Multilingual Full-Text Searching Markus - - PowerPoint PPT Presentation

Drupal City Berlin 2011

Language-Specific and Multilingual Full-Text Searching

Markus Kalkbrenner (mkalkbrenner)

“Apache Solr Search Integration“ is awesome!* * ... if you're running an English website :-(

What's the Problem?

• Stop-words
• Stemming
• Compound words
• Spell checking
• Synonyms
• …

=> everything is preconfigured for English

Stop-words

• Exclude words from your index that are too unspecific
• Definite articles are typical stop-words:
• English: „the“
• German: „der“, „die“, „das“
• Stop words have to be language specific

(False Friends)

• Stop words also depend on the purpose of your site!

Mummy, that one, that one, that one ...

Stemming

• Reducing a word to its stem enables the user to

find content, independent of the keyword's inflection, e.g. singular or plural

• tomato => tomato
• tomatoes => tomato
• The stemming algorithm differs from language

to language!

• For some languages there's no stemmer!

German Stemming

• English stemming:

tomato => tomato tomatoes => tomato Tomate => tomat Tomaten => tomaten

• German stemming:

Tomate => tomat Tomaten => tomat tomato => tomato tomatoes => tomato

Synonyms

• In some cases, stemming does not solve a

problem.

• English:

goose, geese mouse, mice

• German:

Kartoffel, Kartoffeln => provide language-specific synonyms

Protected Words

• Depending on the purpose of your site, it makes

sense to prevent stemming of some words.

• Often product names or brands contain a plural

form or stop words:

• drupal gardens
• Pittsburgh Steelers
• The Who
• Protected words might be language specific.

Compound Words

• In German, „Dampfschifffahrt“ should be found

if you search for

• Dampf
• Dampfschiff
• Schiff
• Schifffahrt
• Solr provides a CompoundWordFilter, Drupal

Apache Solr Search Integration does not.

Spell Checking

• “Did you mean ...”
• Backend for auto-complete
• No doubt that spell checking should be

language-specific.

• Spell checking uses stop-words, too.

ISOLatin1Accent

• By default, „Ä“ becomes „A“
• Take this into account when defining your stop

words, synonyms, etc.

• German stemmer in combination with

ISOLatin1AccentFilter:

• Kuchen => kuch
• Küche => kuch
• Küchen => kuch

Apache Solr Multilingual Module

• Configures Solr according to your site's

language without hacking XML-config-files

• Offers additional advanced configuration
• ptions
• Handles multiple languages in one Solr index
• Provides better language facet
• Implements (basic) „CLIR“ - Cross Language

Information Retrieval

Reliable Setup for Drupal 6

• Apache Solr Multilingual 6.x-2.0-beta1
• Apache Solr Search Integration 6.x-2.0-beta5

Drupal 7?

• Translation process changed dramatically

compared to Drupal 6

• There's more than one way, fields in core, i18n, …
• Field-translation has been “fixed/reverted/removed”

from 7.0 to 7.8

• Apache Solr Search Integration, itself, has been a

moving target => we stopped developing, but now it seems that things stabilize now

Contributing

• Code ;-)
• Default stop word lists, synonym lists,

compound word lists

• Language-specific default configurations for all

the advanced options not mentioned in this session

Questions?

• Apache Solr Search Integration

http://drupal.org/project/apachesolr

• Apache Solr Multilingual

http://drupal.org/project/apachesolr_multilingual

• bio.logis GmbH

http://bio.logis.de