1. 2. - - PDF document

▶
1 2
SMART_READER_LITE
LIVE PREVIEW

1. 2. - - PDF document

Oct 19, 2022 14 likes •277 views

1. 2. 3. 4. Big Data

slide-1
SLIDE 1
  • Beihang
  • 1.

“ 2.

  • 3.
  • 4.

Big Data“ 5. GoogleBaidu 6.

  • 102

“

  • 20160923
slide-2
SLIDE 2
  • Beihang
  • ”
  • Beihang

()

  • Evaluation

– –

slide-3
SLIDE 3
  • Beihang

“(1)

–

–

  • Beihang

“(2)

– v.s.

  • Indexing structures
  • Interaction with OS
  • Communication delays
  • Other overheads

– “retrieval performance evaluation

  • IR: Relevance
slide-4
SLIDE 4
  • Beihang

: Relevance

  • ““

– Answer precise question precisely. – Partially answer question. – Suggest a source for more information. – Give background information. – Remind the user of other knowledge.

  • Beihang
  • “

“

–

  • 01

–

  • 0123

4

–

  • 1994Stefana Mizzaro4
  • <>
  • http://www.psy.gla.ac.uk/~steve/stefano.html
slide-5
SLIDE 5
  • Beihang

“

  • “

– Batch mode

– “Interactive retrieval

– –

  • Beihang
  • ”
slide-6
SLIDE 6
  • Beihang
  • Beihang
slide-7
SLIDE 7
  • Beihang
  • Beihang
slide-8
SLIDE 8
  • Beihang
  • /(Recall rate)

– “

  • /(Precision)

– ““

  • CC

CR CA CRa

Recall = Ra R A Ra = Precision

  • Beihang
  • →q

– Rq={d3,d5,d9,d25,d44,d56,d71,d89,d123,d23} – “

{ d123, d84, d56, d6, d8, d9, d511, d129, d187, d25, d38, d48,d250, d113,d3 }

  • 11(11 standard recall levels)0%,10%,

20%...90%, 100%

  • ))

(

  • %
  • )
  • %
  • %
  • (

61027254 .1033 610272547.1033861

slide-9
SLIDE 9
  • Beihang

A problem

  • 11

– 11

  • – Rq={d3, d56, d129}

– “

{ d123, d84, d56, d6, d8, d9, d511, d129, d187, d25, d38, d48,d250, d113,d3 }

Recall: 33.3%, 66.7%, 100% Precision: 33.3%, 25%, 20%

  • %

%

  • (()
  • .05032

.11 .0503275.1167

  • Beihang

: Interpolation

  • rjjj=0,1,…,10

) ( max ) (

1

r P r P

j j j

r r r

+

≤ ≤

=

  • (
  • (

% %(

  • (
  • %
  • (

)

  • .72138365

21044 .72138365821044972

  • %
%
  • (()
  • .05032
.11 .0503275.1167
slide-10
SLIDE 10
  • Beihang
  • Average Precision

– “ –

∑

=

=

Nq i q i

N (r) P (r) P

1

NqPi(r)r

  • Beihang

ROC/AUC

  • TPR: True Positive Rate, Recall/Sensitivity

FPR: Fall-out FNR: Missing rate TNR: Specificity

Precision v.s. Accuracy ROC: Receiver Operating Curve AUC: Area Under the Curve

63 37 72 28

slide-11
SLIDE 11
  • Beihang
  • “
  • “
  • Beihang

→

  • P@5/P@10/P@N

– 5/10

  • R(R-precision)

– “RR

R R Precision R

  • =

−

slide-12
SLIDE 12
  • Beihang

→

  • (Mean Average Precision)

– APri,

  • – MAP: AP

– , MAP

  • A(q1): d1,d2,d3,d4,d5
  • A(q2): d1,d3,d4,d2,d5
  • ∑

∑

= =

× =

  • Beihang

– “

065240-3.

  • 061

0-34.

slide-13
SLIDE 13
  • Beihang

– – E

  • b=1E=1-F, EF
  • b>1pr
  • b<1rp

r p pr r p + = ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ + = 2 1 1 2 F

( )

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ + + − = p r b b 1 1 1 E

2 2

  • Beihang
slide-14
SLIDE 14
  • Beihang
  • Beihang
  • Discounted Cumulated

Gain

– CG – DCG – NDCG

  • BPREF

– /

  • –
slide-15
SLIDE 15
  • Beihang
  • Beihang
slide-16
SLIDE 16
  • Beihang

– C=Rk/U

– novelty=Ru/(Rk+Ru)

  • (relative recall)

– “

  • (recall effort)

– “

  • CU
  • CRk
  • CRu
  • CR
  • CA
  • Beihang
  • ”
slide-17
SLIDE 17
  • Beihang

TREC

  • TREC

– Text REtrieval Conference“ – “

– NIST(National Institute of Standards and Technology) – U.S. Department of Defense

– “ – 1992~201221

  • Beihang

TREC

  • “

“

  • – “

–

  • “
  • ““
slide-18
SLIDE 18
  • Beihang
  • Track

– TREC

  • Topic

– “ – topicèquery () – Question (QA)

  • Document

–

  • Relevance Judgments

–

  • Beihang

TREC

  • TREC(

)

  • TREC

– : NIST – : – : NIST – : NIST

  • –
slide-19
SLIDE 19
  • Beihang

TREC

– GB – – SGML (Standard Generalized Markup Language)

  • Topic

– – SGML

–

  • Beihang

Topic

  • Title
  • DescriptionTitle

Title

  • Narrative
slide-20
SLIDE 20
  • Beihang

Topic

<topic number="2" type="diagnosis"> <description> A 62 yo male presents with four days of non-productive cough and one day of fever. He is on immunosuppressive medications, including prednisone. He is admitted to the hospital, and his work-up includes bronchoscopy with bronchoalveolar lavage (BAL). BAL fluid examination reveals

  • wl's eye inclusion bodies in the nuclei of infection cells.

</description> <summary> A 62-year-old immunosuppressed male with fever, cough and intranuclear inclusion bodies in bronchoalveolar lavage </summary> </topic>

  • Beihang

Topic

  • Topic
slide-21
SLIDE 21
  • Beihang

– Set Precision/Set Recall

– P@n/Average Precision/Reciprocal Rank

– Filtering Utility

  • Beihang

(1)

  • topicNIST
  • 100
  • Pooling

– n

slide-22
SLIDE 22
  • Beihang

(2)

  • NISTtrec_eval
  • precision

recall)

  • track
  • Beihang

TREC

  • Ad hoc

–

  • Information Routing

–

slide-23
SLIDE 23
  • Beihang

TREC

  • Beihang

TREC 2016-Tracks

  • Clinical Decision Support Track
  • Contextual Suggestion Track
  • Dynamic Domain Track
  • Live QA Track
  • OpenSearch Track
  • Real-Time Summarization Track
  • Tasks Track
  • Total Recall Track
slide-24
SLIDE 24
  • Beihang
  • Beihang

NTCIR

  • NII Test Collection for IR Systems

– NII (National Institute of Informatics) “ – 1998 – “ –

  • “
  • “
slide-25
SLIDE 25
  • Beihang

CLEF

  • Cross-Language Evaluation Forum

– 2000 –

  • –
  • “
  • “
  • “
  • “
  • “
  • “
  • “
  • Beihang

User-Based Evaluation

  • Human experimentation in the lab
  • Side-by-side panels
  • A/B testing
  • Crowdsourcing
  • Using clickthrough data
slide-26
SLIDE 26
  • Beihang
  • Beihang

Q&A