SearchEngineArchitecture CISC489/689010,Lecture#2 Wednesday,Feb.11 - - PDF document

3/17/09
 1


Search
Engine
Architecture


CISC489/689‐010,
Lecture
#2
 Wednesday,
Feb.
11
 Ben
CartereGe


Search
Engine
Architecture


• A
soIware
architecture
consists
of
soIware


components,
the
interfaces
provided
by
those
 components,
and
the
relaPonships
between
 them


– describes
a
system
at
a
parPcular
level
of
abstracPon


• Architecture
of
a
search
engine
determined
by
2


requirements


– effecPveness
(quality
of
results)
and
efficiency
 (response
Pme
and
throughput)


3/17/09
 2


Indexing
Process


Documents


(E‐mails,
web
pages,

 Word
docs,
news
arPcles,
…)


Text
acquisiPon


(Crawler,
feeds,

 filter,
…)


Corpus


Accessible
data
store


Text
transformaPon


(Parsing,
stopping,

 stemming,
extracPon,
…)


Index
creaPon


(Document/term
stats,

 weighPng,
inversion,
…)


Server(s)


Indexing
Process


• Text
acquisiPon


– idenPfies
and
stores
documents
for
indexing


• Text
transformaPon


– transforms
documents
into
index terms or
 features

• Index
creaPon


– takes
index
terms
and
creates
data
structures
 (indexes)
to
support
fast
searching


3/17/09
 3


Query
Process


Corpus


Accessible
data
store


Server(s)
 Ranking
 f(Q,D) EvaluaPon


(Precision,
recall,

 clicks,
…)


Query
Process


• User
interacPon


– supports
creaPon
and
refinement
of
query,
display


• f
results

• Ranking


– uses
query
and
indexes
to
generate
ranked
list
of
 documents


• EvaluaPon


– monitors
and
measures
effecPveness
and
 efficiency
(primarily
offline)


3/17/09
 4


Details:
Text
AcquisiPon


• Crawler


– IdenPfies
and
acquires
documents
for
search
 engine
 – Many
types
–
web,
enterprise,
desktop
 – Web
crawlers
follow
links
to
find
documents

• Must
efficiently
find
huge
numbers
of
web
pages


(coverage)
and
keep
them
up‐to‐date
(freshness)


• Single
site
crawlers
for
site search
• Topical or focused crawlers
for
verPcal search


– Document
crawlers
for
enterprise
and
desktop
 search


• Follow
links
and
scan
directories


Text
AcquisiPon


• Feeds



– Real‐Pme
streams
of
documents


• e.g.,
web
feeds
for
news,
blogs,
video,
radio,
tv


– RSS
is
common
standard


• RSS
“reader”
can
provide
new
XML
documents
to
search


engine


• Conversion


– Convert
variety
of
documents
into
a
consistent
text
 plus
metadata
format


• e.g.
HTML,
XML,
Word,
PDF,
etc.
→
XML


– Convert
text
encoding
for
different
languages


• Using
a
Unicode
standard
like
UTF‐8


3/17/09
 5


Text
AcquisiPon


• Document
data
store


– Stores
text,
metadata,
and
other
related
content
 for
documents



• Metadata
is
informaPon
about
document
such
as
type


and
creaPon
date


• Other
content
includes
links,
anchor
text


– Provides
fast
access
to
document
contents
for
 search
engine
components


• e.g.
result
list
generaPon


– Could
use
relaPonal
database
system



• More
typically,
a
simpler,
more
efficient
storage
system


is
used
due
to
huge
numbers
of
documents


Text
TransformaPon


• Parser


– Processing
the
sequence
of
text
tokens in
the
 document
to
recognize
structural
elements


• e.g.,
Ptles,
links,
headings,
etc.


– Tokenizer
recognizes
“words”
in
the
text


• must
consider
issues
like
capitalizaPon,
hyphens,


apostrophes,
non‐alpha
characters,
separators


– Markup languages such
as
HTML,
XML
oIen
used
to
 specify
structure


• Tags
used
to
specify
document
elements

– E.g.,
<h2>
Overview
</h2>


• Document
parser
uses
syntax
of
markup
language
(or
other


formanng)
to
idenPfy
structure


3/17/09
 6


Text
TransformaPon


• Stopping


– Remove
common
words

• e.g.,
“and”,
“or”,
“the”,
“in”


– Some
impact
on
efficiency
and
effecPveness
 – Can
be
a
problem
for
some
queries


• Stemming


– Group
words
derived
from
a
common
stem

• e.g.,
“computer”,
“computers”,
“compuPng”,
“compute”


– Usually
effecPve,
but
not
for
all
queries
 – Benefits
vary
for
different
languages


Text
TransformaPon


• Link
Analysis


– Makes
use
of
links
and
anchor text in
web
pages
 – Link
analysis
idenPfies
popularity
and
community
 informaPon


• e.g.,
PageRank


– Anchor
text
can
significantly
enhance
the
 representaPon
of
pages
pointed
to
by
links
 – Significant
impact
on
web
search


• Less
importance
in
other
applicaPons


3/17/09
 7


Text
TransformaPon


• InformaPon
ExtracPon


– IdenPfy
classes
of
index
terms
that
are
important
 for
some
applicaPons
 – e.g.,
named en;ty recognizers idenPfy
classes
 such
as
people, loca;ons, companies, dates,
etc.


• Classifier


– IdenPfies
class‐related
metadata
for
documents


• i.e.,
assigns
labels
to
documents

• e.g.,
topics,
reading
levels,
senPment,
genre


– Use
depends
on
applicaPon


Index
CreaPon


• Document
StaPsPcs


– Gathers
counts
and
posiPons
of
words
and
other
 features
 – Used
in
ranking
algorithm


• WeighPng


– Computes
weights
for
index
terms
 – Used
in
ranking
algorithm
 – e.g.,
=.idf
weight


• CombinaPon
of
term frequency in
document
and


inverse document frequency in
the
collecPon


3/17/09
 8


Index
CreaPon


• Inversion


– Core
of
indexing
process
 – Converts
document‐term
informaPon
to
term‐ document
for
indexing


• Difficult
for
very
large
numbers
of
documents


– Format
of
inverted
file
is
designed
for
fast
query
 processing


• Must
also
handle
updates

• Compression
used
for
efficiency


Index
CreaPon


• Index
DistribuPon


– Distributes
indexes
across
mulPple
computers
 and/or
mulPple
sites
 – EssenPal
for
fast
query
processing
with
large
 numbers
of
documents
 – Many
variaPons


• Document
distribuPon,
term
distribuPon,
replicaPon


– P2P
and
distributed IR
involve
search
across
 mulPple
sites

3/17/09
 9


User
InteracPon


• Query
input


– Provides
interface
and
parser
for
query language
 – Most
web
queries
are
very
simple,
other
 applicaPons
may
use
forms
 – Query
language
used
to
describe
more
complex
 queries
and
results
of
query
transformaPon


• e.g.,
Boolean
queries,
Indri
and
Galago
query
languages

• similar
to
SQL
language
used
in
database
applicaPons

• IR
query
languages
also
allow
content
and
structure


specificaPons,
but
focus
on
content


User
InteracPon


• Query
transformaPon


– Improves
iniPal
query,
both
before
and
aIer
iniPal
 search
 – Includes
text
transformaPon
techniques
used
for
 documents
 – Spell checking and
query sugges;on
provide
 alternaPves
to
original
query
 – Query expansion and
relevance feedback
modify
 the
original
query
with
addiPonal
terms

3/17/09
 10


User
InteracPon


• Results
output


– Constructs
the
display
of
ranked
documents
for
a
 query
 – Generates
snippets
to
show
how
queries
match
 documents
 – Highlights
important
words
and
passages
 – Retrieves
appropriate
adver;sing
in
many
 applicaPons
 – May
provide
clustering
and
other
visualizaPon
 tools


Ranking


• Scoring


– Calculates
scores
for
documents
using
a
ranking
 algorithm
 – Core
component
of
search
engine
 – Basic
form
of
score
is



• qt
and
dt
are
query
and
document
term
weights
for


term
t


– Many
variaPons
of
ranking
algorithms
and
 retrieval
models


3/17/09
 11


Ranking


• Performance
opPmizaPon


– Designing
ranking
algorithms
for
efficient
 processing


• Term‐at‐a ;me vs.
document‐at‐a‐;me
processing
• Safe
vs.
unsafe
opPmizaPons

• DistribuPon


– Processing
queries
in
a
distributed
environment
 – Query broker distributes
queries
and
assembles
 results
 – Caching
is
a
form
of
distributed
searching


EvaluaPon


• Logging


– Logging
user
queries
and
interacPon
is
crucial
for
 improving
search
effecPveness
and
efficiency
 – Query logs and
clickthrough data used
for
query
 suggesPon,
spell
checking,
query
caching,
ranking,
 adverPsing
search,
and
other
components


• Ranking
analysis


– Measuring
and
tuning
ranking
effecPveness


• Performance
analysis


– Measuring
and
tuning
system
efficiency


3/17/09
 12


How
Does
It
Really
Work?


• This
course
explains
these
components
of
a


search
engine
in
more
detail


• OIen
many
possible
approaches
and
techniques


for
a
given
component


– Focus
is
on
the
most
important
alternaPves
 – i.e.,
explain
a
small
number
of
approaches
in
detail
 rather
than
many
approaches
 – “Importance”
based
on
research
results
and
use
in
 actual
search
engines
 – AlternaPves
described
in
references


Text
AcquisiPon


Web
Crawling,
Feeds,
and
Storage


3/17/09
 13


Web
Crawler


• Finds
and
downloads
web
pages
automaPcally


– provides
the
collecPon
for
searching


• Web
is
huge
and
constantly
growing

• Web
is
not
under
the
control
of
search
engine


providers


• Web
pages
are
constantly
changing

• Crawlers
also
used
for
other
types
of
data


Retrieving
Web
Pages


• Every
page
has
a
unique
uniform resource

locator
(URL)


• Web
pages
are
stored
on
web
servers
that
use


HTTP
to
exchange
informaPon
with
client
 soIware


• e.g.,


3/17/09
 14


Retrieving
Web
Pages


• Web
crawler
client
program
connects
to
a


domain name system (DNS)
server


• DNS
server
translates
the
hostname
into
an


internet protocol (IP)
address


• Crawler
then
aGempts
to
connect
to
server


host
using
specific
port

• AIer
connecPon,
crawler
sends
an
HTTP


request
to
the
web
server
to
request
a
page


– usually
a
GET
request


Crawling
the
Web


3/17/09
 15


Web
Crawler


• Starts
with
a
set
of
seeds,
which
are
a
set
of


URLs
given
to
it
as parameters


• Seeds
are
added
to
a
URL
request
queue

• Crawler
starts
fetching
pages
from
the
request


queue


• Downloaded
pages
are
parsed
to
find
link
tags


that
might
contain
other
useful
URLs
to
fetch


• New
URLs
added
to
the
crawler’s
request


queue,
or
fron;er

• ConPnue
unPl
no
more
new
URLs
or
disk
full


Web
Crawling


• The
“long
tail”
of
web
pages.


Number
of
pages
 UPlity
 Unordered
crawling
produces
these.


3/17/09
 16


Web
Crawling


• Ordering
URLs


– Crawl
URLs
in
some
order
of
“importance”
 – “Random
surfer”
model:


• A
user
starts
on
a
page
and
randomly
clicks
links.

• Occasionally
switches
to
a
different
page
with
no
click.

• What
is
the
probability
the
user
will
land
on
any
given


page?


– Higher
probability

greater
importance.
 – PageRank


Web
Crawling


• Web
crawlers
spend
a
lot
of
Pme
waiPng
for


responses
to
requests


• To
reduce
this
inefficiency,
web
crawlers
use


threads
and
fetch
hundreds
of
pages
at
once


• Crawlers
could
potenPally
flood
sites
with


requests
for
pages


• To
avoid
this
problem,
web
crawlers
use


politeness policies

– e.g.,
delay
between
requests
to
same
web
server


3/17/09
 17


Controlling
Crawling


• Even
crawling
a
site
slowly
will
anger
some


web
server
administrators,
who
object
to
any
 copying
of
their
data


• Robots.txt
file
can
be
used
to
control
crawlers


Simple
Crawler
Thread


3/17/09
 18


Freshness


• Web
pages
are
constantly
being
added,


deleted,
and
modified


• Web
crawler
must
conPnually
revisit
pages
it


has
already
crawled
to
see
if
they
have
 changed
in
order
to
maintain
the
freshness of
 the
document
collecPon


– stale
copies
no
longer
reflect
the
real
contents
of
 the
web
pages


Freshness


• HTTP
protocol
has
a
special
request
type


called
HEAD
that
makes
it
easy
to
check
for
 page
changes


– returns
informaPon
about
page,
not
page
itself


3/17/09
 19


Freshness


• Not
possible
to
constantly
check
all
pages


– must
check
important
pages
and
pages
that
 change
frequently


• Freshness
is
the
proporPon
of
pages
that
are


fresh


• OpPmizing
for
this
metric
can
lead
to
bad


decisions,
such
as
not
crawling
popular
sites


• Age
is
a
beGer
metric


Age


• Expected
age
of
a
page
t
days
aIer
it
was
last


crawled:


• Web
page
updates
follow
the
Poisson


distribuPon
on
average


– Pme
unPl
the
next
update
is
governed
by
an
 exponenPal
distribuPon


3/17/09
 20


Age


• Older
a
page
gets,
the
more
it
costs
not
to


crawl
it


– e.g.,
expected
age
with
mean
change
frequency



 λ =
1/7
(one
change
per
week)


Focused
Crawling


• AGempts
to
download
only
those
pages
that


are
about
a
parPcular
topic


– used
by
ver;cal search applicaPons


• Rely
on
the
fact
that
pages
about
a
topic
tend


to
have
links
to
other
pages
on
the
same
topic


– popular
pages
for
a
topic
are
typically
used
as
 seeds


• Crawler
uses
text classifier to
decide
whether


a
page
is
on
topic


3/17/09
 21


Deep
Web


• Sites
that
are
difficult
for
a
crawler
to
find
are


collecPvely
referred
to
as
the
deep (or
hidden) Web

– much
larger
than
convenPonal
Web


• Three
broad
categories:


– private
sites


• no
incoming
links,
or
may
require
log
in
with
a
valid
account


– form
results


• sites
that
can
be
reached
only
aIer
entering
some
data
into


a
form


– scripted
pages


• pages
that
use
JavaScript,
Flash,
or
another
client‐side


language
to
generate
links


Document
Feeds


• Many
documents
are
published

– created
at
a
fixed
Pme
and
rarely
updated
again
 – e.g.,
news
arPcles,
blog
posts,
press
releases,
 email


• Published
documents
from
a
single
source
can


be
ordered
in
a
sequence
called
a
document feed

– new
documents
found
by
examining
the
end
of
 the
feed


3/17/09
 22


Document
Feeds


• Two
types:


– A
push feed alerts
the
subscriber
to
new
 documents
 – A
pull feed requires
the
subscriber
to
check
 periodically
for new
documents


• Most
common
format
for
pull
feeds
is
called


RSS

– Really
Simple
SyndicaPon,
RDF
Site
Summary,
Rich
 Site
Summary,
or
...


Conversion


• Text
is
stored
in
hundreds
of
incompaPble
file


formats


– e.g.,
raw
text,
RTF,
HTML,
XML,
MicrosoI
Word,
ODF,
 PDF


• Other
types
of
files
also
important


– e.g.,
PowerPoint,
Excel


• Typically
use
a
conversion
tool


– converts
the
document
content
into
a
tagged
text
 format
such
as
HTML
or
XML
 – retains
some
of
the
important
formanng
informaPon


3/17/09
 23


Character
Encoding


• A
character
encoding
is
a
mapping
between


bits
and
glyphs


– i.e.,
genng
from
bits
in
a
file
to
characters
on
a
 screen
 – Can
be
a
major
source
of
incompaPbility


• ASCII
is
basic
character
encoding
scheme
for


English


– encodes
128
leGers,
numbers,
special
characters,
 and
control
characters
in
7
bits,
extended
with
an
 extra
bit
for
storage
in
bytes


Character
Encoding


• Other
languages
can
have
many
more
glyphs


– e.g.,
Chinese
has
more
than
40,000
characters,
with


• ver
3,000
in
common
use

• Many
languages
have
mulPple
encoding
schemes


– e.g.,
CJK
(Chinese‐Japanese‐Korean)
family
of
East
 Asian
languages,
Hindi,
Arabic
 – must
specify
encoding
 – can’t
have
mulPple
languages
in
one
file


• Unicode
developed
to
address
encoding


problems


3/17/09
 24


Storing
the
Documents


• Many
reasons
to
store
converted
document


text


– saves
crawling
Pme
when
page
is
not
updated
 – provides
efficient
access
to
text
for
snippet
 generaPon,
informaPon
extracPon,
etc.


• Database
systems
can
provide
document


storage
for
some
applicaPons


– web
search
engines
use
customized
document
 storage
systems


Storing
the
Documents


• Requirements
for
document
storage
system:


– Random
access


• request
the
content
of
a
document
based
on
its
URL

• hash
funcPon
based
on
URL
is
typical


– Compression
and
large
files


• reducing
storage
requirements
and
efficient
access


– Update


• handling
large
volumes
of
new
and
modified


documents


• adding
new
anchor
text


3/17/09
 25


Large
Files


• Store
many
documents
in
large
files,
rather


than
each
document
in
a
file


– avoids
overhead
in
opening
and
closing
files
 – reduces
seek
Pme
relaPve
to
read
Pme


• Compound
documents
formats


– used
to
store
mulPple
documents
in
a
file
 – e.g.,
TREC
Web,
Wikipedia
XML