F ROM WEBSITES TO APPS , AND NOW FROM APPS TO CHATBOTS ? A NTNIO B - - PowerPoint PPT Presentation

▶
f rom websites to apps and now from apps to chatbots
SMART_READER_LITE
LIVE PREVIEW

F ROM WEBSITES TO APPS , AND NOW FROM APPS TO CHATBOTS ? A NTNIO B - - PowerPoint PPT Presentation

Apr 12, 2023 348 likes •499 views

F ROM WEBSITES TO APPS , AND NOW FROM APPS TO CHATBOTS ? A NTNIO B RANCO FROM APPS TO CHATBOTS CEO of large social network in annual development conference 2 months ago Shared vision for next 2 decades: past : with advent of PCs, companies

slide-1
SLIDE 1

FROM WEBSITES TO APPS,

AND NOW FROM APPS TO CHATBOTS?

ANTÓNIO BRANCO

slide-2
SLIDE 2

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

FROM APPS TO CHATBOTS

2

CEO of large social network in annual development conference 2 months ago Shared vision for next 2 decades: past: with advent of PCs, companies reached customers with websites currently: with smartphones, are reaching customers with apps future: will be reaching with chatbots

slide-3
SLIDE 3

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

A REAL USAGE SCENARIO

3

Provide accurate support to end-users via chat channel When moving to another linguistic market, accumulated advantage vanishes to zero??

slide-4
SLIDE 4

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

MACHINE TRANSLATION AT WORK

4

Not necessarily, if one resorts to machine translation: But how much effective is using MT in day 1 in a new linguistic market?

slide-5
SLIDE 5

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

HOW MUCH CAN MT HELP ?

5

Cost with human resources are the lion’s share in contact centers… … MT contribution is as much effective as more calls to human

  • perators can be spared in day 1:

For the extrinsic evaluation methodology:

Gaudio, Burchardt and Branco, 2016, "Evaluating Machine Translation in a Usage Scenario" , LREC2016.

Probability* calling*operator*

Avg.*

EU! BG! CS! NL! DE! PT! ES!

low 42.2%

33.3% 47.4% 54.5% 30.4% 47.8% 21.5% 60.4%

medium 20.5%

28.1% 30.6% 17.9% 21.9% 22.0% 15.8% 7.0%

high 37.1%

37.0% 22.0% 27.5% 47.7% 30.1% 62.7% 32.7%

*

slide-6
SLIDE 6

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

TAKE HOME MESSAGE

6

When porting your MT-supported chat-based contact center to a new linguistic market, at day 1 of its operation, the overall chance of dispensing human operator intervention and thus the language specific costs that are spared by using MT are on average at least 40% and up to 60% of the costs of doing it without MT

slide-7
SLIDE 7

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

WAIT, WAIT…

7

  • … 40%-60% of costs spared with which MT system by the way?
  • Off-the-shelf SMT
  • So, even with a less performant system than your QTLeap MT

system?

  • Right
slide-8
SLIDE 8

THANK YOU

slide-9
SLIDE 9

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

9

EMULATE REAL USAGE

Step 1: Review answer A (MT) without any reference:

§ It would clearly help me solve my problem / answer my question § It might help, but would require some thinking to understand it. § Is not helpful / I don't understand it

Step2: Compare answers A and B (human reference), (re-)evaluate A selecting one of the following options:

§ A gives the right advice. § A gets minor points wrong. § A gets important points wrong.

slide-10
SLIDE 10

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

RESULTS OF STEP 1 AND 2

10

slide-11
SLIDE 11

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

ESTIMATING OPERATOR INVENTION PROBABILITY

11

Probability* EU! BG! CS! NL! DE! PT! ES! Avg.*

low 33.3% 47.4% 54.5% 30.4% 47.8% 21.5% 60.4% 42.2% medium 28.1% 30.6% 17.9% 21.9% 22.0% 15.8% 7.0% 20.5% high 37.0% 22.0% 27.5% 47.7% 30.1% 62.7% 32.7% 37.1%

*

slide-12
SLIDE 12

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

The Baseline

  • Baseline system

– 7 language pairs: phrase-based SMT (Moses)

  • Two models: transla=on model, (mono, target) language model
  • Training

– Europarl and other parallel and monolingual corpora

  • Tuning

– MERT, 1 Ksentences in-domain data

  • Evalua=on

– Automa=c metrics (as usual: BLEU, METEOR), 1Ksentences in- domain

12

slide-13
SLIDE 13

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

Baseline – training seJngs

13

slide-14
SLIDE 14

António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016

Baseline - datasets

  • Basque
  • 1.5 Msentences bilingual corpora (Elhuyar Founda=on, in-domain, etc.)
  • 2.2 M monolingual corpora
  • Bulgarian
  • 600 K bilingual (Europarl, in-domain LibreOffice, etc)
  • 3.4 M monolingual (+ Bulgarian Ref Corpus)
  • Czech
  • 15 M bilingual (Czech-English Parallel Corpus)
  • 18 M monolingual (+ Europarl, etc)
  • Dutch
  • 370 K bi & mono (Dutch Parallel Corpus, ½ in-domain)
  • German
  • 4.5 M bi & mono (Europarl, in-domain, etc.)
  • Portuguese
  • 2 M bi & mono (out-domain Europarl)
  • Spanish
  • 15 M bi & mono (Europarl, UN, in-domain, etc.)

14