[PDF] - Extending the P o w er and Capacit y of Constrain t PDF Document

SLIDE 1 Extending the P

w

er and Capacit y

f

Constrain t Satisfaction Net w

rks

Xinc h uan Zeng and T

n

y R Martinez Computer Science Departmen t Brigham Y

ung

Univ ersit y July

SLIDE 2 Abstract This w

rk

fo cuses

n

impro ving the Hopeld network for solv ing

ptimization

problems Although m uc h w

rk

has b een done in this area the p erformance

f

the Hopeld net w

rk

is still not satisfactory in terms

f

v alid con v ergence and qualit y

f

solutions W e address this issue in this w

rk

b y com bing a new activ ation function E B A and a new relax ation pro cedure C R

in
rder

to impro v e the p erformance

f

the Hopeld net w

rk

Eac h

f

E B A and C R has b een in dividually demonstrated capable

f

substan tially impro ving the p erformance The com bined approac h has b een ev al uated through

sim

ulations based

n
randomly

generated cit y distributions

f

the cit y tr aveling salesman pr

blem

The result sho ws that com bining the t w

metho

ds

SLIDE 3 is able to further impro v e the p erformance Compared to C R without com bining with E B A the com bined approac h increases the p ercen tage

f

v alid tou rs b y

and

de creases the error rate b y

As

compared to the

riginal

Hopeld metho d using neither E B A nor C R

the

com bined approac h increases the p ercen tage

f

v alid tours b y

and

decreases the error rate b y

SLIDE 4 Com binatorial

ptimization

problems

Most
f

them are N P problems

Man

y maxim um

r

minim um p

in

ts

Examples
T

ra v eling Salesman Problem T S P

Routing

in comm unication net w

rks
Graph

partitioning in circuit design Wh y study T S P b y Hopeld net w

rk
T

S P is

ne

early application b y Hopeld net w

rk

HN

T

S P is represen tativ e

f

com binational problems

T

S P is a b enc hmark for comparing algorithms

SLIDE 5

A

go

d

algorithm for T S P could b e mapp ed to solv e

ther

com binatorial problems Hopeld net w

rks

vs

ther

heuristics

Examples
f
ther

heuristics DivideandConquer F

rm
utline

con tour

The

pro cedure

f

HN is more gener al while

ther

metho ds are more pr

blemsp

e cic

HN

can use p ar al lel pro cessing while

ther

metho ds usu ally use se quential pro cessing

HN

can ha v e con v enien t hardw are implemen tation

HN

can ac hiev e an

ptimal
r

suboptimal solution in short time

SLIDE 6 Example Arc hitecture for a cit y TSP

E6

POSITION CITY A B C D E F G 1 2 3 4 5 6 7

V V

B2

F

ully connected based

n

an energy function

Prop

er random initial activ ations for neurons

Net

w

rk

is relaxed un til reac hing an equilibrium

SLIDE 7 Energy function for a N cit y TSP

E
A
N

X X

N

X i N X j j i V X i V X j

B
N

X i N X X

N

X Y Y X V X i V Y i

C
N

X X

N

X i V X i

N
D
N

X X

N

X i N X Y Y X d X Y V X i V Y i

V

Y i

where

A

B
D
C
N
Corresp
nding

constrain ts

One

and

nly
ne
n

neuron eac h ro w

One

and

nly
ne
n

neuron eac h column

T
tal

n um b er

f
n

neuron

Encourage

short tour based

n

the distance matrix

SLIDE 8 Pro cedure

f

the Hopeld Net wrok

Eac

h neuron X

i

has an input v alue U X i and an activ a tion

utput

v alue V X i

Connecting

w eigh t b et w een X

i

and Y

j
W

X iY j

A

X Y

ij
B
ij
X

Y

C
D

d X Y

ji
ji
Where
ij
if

i

j
and
ij
therwise
Neuron

X

i

is also connected to an external input cur ren t bias I X i

C

N

Initial

v alue

f

U X i is set to b e a constan t v alue deter mined b y P N X

P

N i V X i

N
and

is then p erturb ed with small random noise

SLIDE 9

During

relaxation U n X i at step n is up dated b y U n X i

U

n X i

U

X i

U

X i is giv en b y the equation U X i

U

X i

N

X Y

N

X j

W

X iY j V Y j

I

X i

n

t

where
is

time constan t

f

R C circuit

Activ

ation V n X i is determined b y U n X i through an activation function whic h is sigmoid in HN V n X i

tanh

U n X i u

Hopeld

and T ank

sho

w ed that the net w

rk

is guaran teed to con v erge to a lo cal minim um for sym metric connecting w eigh ts

SLIDE 10 Problems with the Hopeld Net w

rk
F

requency

f

p

r

qualit y and in v alid solutions

Sensitivit

y to v ariations

f

cit y distributions in T S P

Sensitivit

y to the c hoice

f

parameters

Result
f

Wilson and P a wley

v

alid tours

froze

in to in v alid tours

did

not con v erge in

iterations

Based

n
sets
f

randomly cit y TSP

Previous

w

rk

has fo cused

n

impro ving the Hopeld net w

rk

b y mo difying the energy function

SLIDE 11 Purp

se

This w

rk

fo cuses

n

impro ving the Hopeld network for solv ing

ptimization

problems Although m uc h w

rk

has b een done in this area the p erformance

f

the Hopeld net w

rk

is still not satisfactory in terms

f

v alid con v ergence and qualit y

f

solutions In particular w

rk

follo wing the initial use

f

the Hopeld net w

rk

for the T S P tr aveling salesman pr

blem

demonstrated that the net w

rk

con v erged to v alid tours

nly

a small p ercen tage

f

the time W e address this issue in this w

rk

b y com bing an Evidence Based Activ ation F unction E B A and a Con trolled Relaxation C R

pro

ce dure in

rder

to impro v e the p erformance

f

the Hopeld net w

rk

SLIDE 12 Metho ds This w

rk

impro v es the p erformance

f

the Hopeld net w

rk

b y com bing an Evidence Based Activ ation F unction E B A and a Con trolled Relaxation C R

Evidence

Based Activ ation F unction The

riginal

sigmoid activ ation function is giv en b y Eq

Evidence

Based Activ ation F unction E B A V X i

tanh

U X i x

u
tanh

x

u
U

X i

V

X i

tanh

x

u
tanh

U X i x

u
tanh

x

u
U

X i

SLIDE 13 Comparison b et w een t w

functions

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0.1
0.05

0.05 0.1 V U A B C A: sigmoid B: EBA (x0=0.03) C: EBA (x0=0.05)

E

B A has a larger threshold con trolled b y x

and

is more robust against noise

E

B A increases the p ercen tage

f

v alid tours b y

and

decreases the error rate

f

tour length b y

SLIDE 14 Con trolled Relaxation The pro cedure

f

Con trolled Relaxation C R

N

et X i

N

X Y

N

X j

W

X iY j V Y j

I

X i net v al ue

T

X i

tanh

N et X i u

g
al

activ ation

V

n X i

V

n X i

R

T X i

V

n X i

utput

v al ue

Dierences

from

riginal

pro cedure

Up

date V X i directly do es not use U X i

Use

a relaxation rate R

No

parameter t

u
and

u

ha

v e dieren t scales

Drop

the time constan t

to
small

C R increases v alid tours b y

and

decreases error rate

f

tour length b y

SLIDE 15 Com bine C R with E B A

Using

the folo wing activ ation function using E B A in Eq

T

X i

tanh

N et X i x

u
tanh

x

u
U

X i

T

X i

tanh

x

u
tanh

N et X i x

u
tanh

x

u
U

X i

All

the

ther

steps in the com bined approac h are the same as those using C R

SLIDE 16 Results Sim ulation Conditions

random

cit y T S P cit y distributions

runs

for eac h cit y distribution

runs

for

cit

y distributions P erformance Measuremen ts

P

ercen tage

f

v alid tours

V

alid

for

cit y dist i V al id i

N

v al idi N total i

V

alid

a

v eraged

v

er

cit

y dist V al id

P

N C ity D ist i V al id i N C ity D ist

SLIDE 17

Error

rate

Error
for

tour j

f

cit y dist i E r r ij

d

ij

d

iopt d iopt

Error
for

cit y dist i E r r i

P

N v al idi j

E

r r ij N v al idi

Error
a

v eraged

v

er

cit

y dist E r r

P

N C ity D ist i V al id i E r r i

P

N C ity D ist i V al id i

SLIDE 18 V alid tour p ercen tage

20

40 60 80 100 120 0.04 0.06 0.08 0.1 0.12 0.14 0.16 Valid Tour(%) R A B C A: CR with EBA B: CR without EBA C: nither CR nor EBA

V

alid tours increase b y another

at

R

com

pared to those without using E B A

V

alid tours increase b y total

f
compared

to those using the

riginal

Hopeld net w

rk

using neither E B A nor C R

SLIDE 19 Error rate

f

tour length

2

4 6 8 10 0.04 0.06 0.08 0.1 0.12 0.14 0.16 Error(%) R A B C A: CR with EBA B: CR without EBA C: nither CR nor EBA

Error

rate decreases b y another

at

R

com

pared to those without using E B A

Error

rate decreases b y total

f
compared

to those using the

riginal

Hopeld net w

rk

using neither E B A nor C R

SLIDE 20 New Asp ects

f

W

rk

Long standing problems with the Hopeld net w

rks

include lo w v alid con v ergence rate lo w qualit y nal states and ex treme sensitivit y to w eigh t parameters The impro v ed ac tiv ation function E B A and relaxation pro cess C R

allo

w substan tial impro v emen ts in b

th

con v ergence rate and qual it y

W

e com bine E B A and C R in

rder

to impro v e the p er formance

f

the Hopeld net w

rk

The results based

n

a large n um b er

f

sim ulations sho w that com bining E B A and C R can further signican tly impro v e the p erformance

SLIDE 21 Conclusion E B A has the capabilit y

f

reducing the eects

f

noise and C R enables a smo

ther

relaxation pro cess Both E B A and C R are capable

f

signican tly impro ving the p erformance

f

the Hopeld net w

rk

Com bining E B A and C R can further impro v e the p erfor mance

leads

to another

total
f
increase

in v alid tours and another

total
f
decrease

in error rate

SLIDE 22 F uture W

rk
Ev

aluate its p erformance

n
ther
ptimization

prob lems

Use

adaptiv e relaxation rates to ac hiev e a more

ptimal

relaxation pro cess

In

tro duce a learning mec hanism to determine the pa rameters in the net w

rk
Exp

erimen t with the bip

lar

v ersion

f

the E B A function and ev aluate its p erformance

Apply

it in to

ther

realw

rld

application domains in particular con tin uous sp eec h recognition