APPROACHES TO IMPLEMENT SEMANTIC SEARCH Johannes Peter Product - PowerPoint PPT Presentation

APPROACHES TO IMPLEMENT SEMANTIC SEARCH Johannes Peter Product Owner / Architect for Search 1

WHAT IS SEMANTIC SEARCH ? 2

Success of search • Interface of shops to brains of customers • Wide range of usage • Success depends on a proper understanding ? Search 3

Simple keyword search mymobile 7 without contract title type description attribute MyMobile 7 Smartphone ... with a contract ... contract MyMobile 7 Smartphone ... Marriage without contract DVD ... MyMobile 6 Smartphone ... Sitcom season 7 DVD ... 7 … MyMobile 6 Smartphone ... with a contract ... contract 4

Identifying entities mymobile 7 without contract product / product group without certain attribute Entity Example Products mymobile 7 Attributes contract Product with / without attribute mymobile 7 without contract Product group with approximate price mymobile under 300 euro 5

Semantic search mymobile 7 without contract product / product group without certain attribute title type description attribute MyMobile 7 smartphone ... MyMobile 6 smartphone ... MyMobile 7 smartphone ... with a contract ... contract MyMobile 6 smartphone ... with a contract ... contract 6

Core benefits Better Facilitated search Better precision recommendation management 7

Future perspectives Sophisticated sales Voice search Chat bots advisors 8

APPROACHES 9

ONTOLOGIES & RULE COLLECTIONS 10

Ontologies & rule collections mymobile 7 without contract Step Example Identify entities product (mymobile 7) without attribute (contract) Execute rules to combine product (mymobile 7) not ( attribute (contract)) entities Translate into search query title:("mymobile 7") AND NOT flag:(contract) 11

Ontologies • Hierarchies of entities • Products, attributes and relations product attribute mymobile color mymobile 6 mymobile 7 black white 12

Rule collections mymobile 7 without contract • Condition: There is the term without between a product and an attribute • Action: Negate the attribute pink dvd • Pink: color or artist? à Disambiguation • Condition: The term pink appears together with entities related to music or movies • Action: Annotate the term pink as artist 13

Implementation • Two parts of implementation - Development of the application - Information extraction part (creation of ontologies & rule collections) • Service for ontology extraction - Solr and Elasticsearch are not suitable - Highly scalable and performant solution with Spring Boot & Apache Lucene (using term vectors as payloads) • Rule engine - Configurable rulesets - Routing concept 15

Implementation • Well suited for agile development • Pieces of information can be extracted fairly independently Sprint(s) Extract prices Ontology for products … Sprint(s) Rules for products Combinations of products & prices … 16

Implementation • More complex cases - Extract information out of product descriptions - Understanding of natural language Developers Analysts / Linguists • Requires maintenance for ontologies and rule collections 17

MACHINE LEARNING 18

Machine learning training data model new query 19

Machine learning training data model new query term mymobile 7 without contract part of speech noun digit preposition noun relation head mymobile contract mymobile chunks noun phrase noun with negation entity product with negated attribute 20

Machine learning – NLP • How natural is the language used for queries? • Considering grammatical information can be complicated • Disambiguation is very difficult for some cases term term pink mymobile pink dvd part of speech part of speech adjective noun proper noun noun • Natural language processing: - "The label saw potential in Pink and offered her a contract." 21

Implementation • Established procedures from the area of natural language processing • Libraries (e. g. spaCy) providing - Functionalities fairly easy to use - High performance - Customizations • All discussed steps require their own model (training + evaluation data) • Still highly experimental - Fail early? - Continuous delivery? 22

TERM CO-OCCURRENCES 23

Term co-occurrences • Enrich documents by contextual information • Using collaborative filters (recommendation) • Which terms / attributes appear in the context of a product? 24

Term co-occurrences mymobile 7 title category color description MyMobile 7 MyMobile black Smartphone MyMobile 7 black with 128 gb MyMobile 7 MyMobile white New smartphone MyMobile 7, 64 gb, white Sitcom season 7 DVD Season number 7 of the sitcom … MyMobile 6 MyMobile black MyMobile 6 – smartphone – 32 gb – black MyMobile 6 MyMobile white MyMobile 6, smartphone black with 128 gb • Co-occurring terms for category MyMobile: Ø Term "smartphone": 7, black, white 26

Term co-occurrences mymobile 7 title category color context MyMobile 7 MyMobile black 6, white MyMobile 7 MyMobile white 6, black MyMobile 6 MyMobile black 7, white MyMobile 6 MyMobile white 7, black Sitcom season 7 DVD … 27

Implementation • Fairly easy to implement • Generic • Produces side effects • Requires high data quality • Only partially solves problems related to semantic search • Not suitable for complex cases 29

Conclusion Term co-occurrences Ontologies + rules Machine learning Effort moderate high high Holistic solution no yes yes Suitable for complex cases no yes yes Maintenance effort low high low High data quality Ability of linguists • Ability of data scientists • • Agile development Quality of rules • Quality of training data Success factors • • Agile development • • Never-ending generation Never-ending rule- • of training data Risk factors Side effects • building • Too high expectations 30

THANK YOU !! BTW: We are hiring … peterj@mediamarktsaturn.com 31