Language as an Algebra Daoud Clarke Department of Computer Science - - PowerPoint PPT Presentation

▶

Mar 07, 2023 256 likes •829 views

Background Context Theories Tools Language as an Algebra Daoud Clarke Department of Computer Science University of Hertfordshire The Categorical Flow of Information in Quantum Physics and Linguistics, Oxford, 2010 Daoud Clarke Language as

SLIDE 1

Background Context Theories Tools

Language as an Algebra

Daoud Clarke

Department of Computer Science University of Hertfordshire

The Categorical Flow of Information in Quantum Physics and Linguistics, Oxford, 2010

Daoud Clarke Language as an Algebra

SLIDE 2

Background Context Theories Tools

Overview

1

Background Distributional Semantics Context-theoretic Semantics

2

Context Theories Motivating Example Example

3

Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Daoud Clarke Language as an Algebra

SLIDE 3

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Beyond lexical distributional semantics

Distributional semantics:

Hypothesis of Harris (1968) LSA, distributional similarity etc. Many applications Good for words/short phrases

How can we go beyond the lexical domain?

Data sparseness

Daoud Clarke Language as an Algebra

SLIDE 4

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Beyond lexical distributional semantics

Distributional semantics:

Hypothesis of Harris (1968) LSA, distributional similarity etc. Many applications Good for words/short phrases

How can we go beyond the lexical domain?

Data sparseness

Daoud Clarke Language as an Algebra

SLIDE 5

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Physicists (xkcd.com)

Daoud Clarke Language as an Algebra

SLIDE 6

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Physicists (xkcd.com)

Daoud Clarke Language as an Algebra

SLIDE 7

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Context-theoretic Semantics

Suggestion: explore the use of unital associative algebras

ver the real numbers

An algebra is a vector space together with a bilinear multiplication: x · (y + z) = x · y + x · z (x + y) · z = x · z + y · z Associative algebras form a monoidal category K-Alg

Daoud Clarke Language as an Algebra

SLIDE 8

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Context-theoretic Semantics

Suggestion: explore the use of unital associative algebras

ver the real numbers

An algebra is a vector space together with a bilinear multiplication: x · (y + z) = x · y + x · z (x + y) · z = x · z + y · z Associative algebras form a monoidal category K-Alg

Daoud Clarke Language as an Algebra

SLIDE 9

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Context-theoretic Semantics

Suggestion: explore the use of unital associative algebras

ver the real numbers

An algebra is a vector space together with a bilinear multiplication: x · (y + z) = x · y + x · z (x + y) · z = x · z + y · z Associative algebras form a monoidal category K-Alg

Daoud Clarke Language as an Algebra

SLIDE 10

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Al-kit¯ ab al-mukhtas .ar f¯ ı h .is¯ abi-l-jabr wa’l-muq¯ abala

Daoud Clarke Language as an Algebra

SLIDE 11

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Algebra over a field

An algebra over a field:

abstraction of the space of operators on a vector space

Hilbert space operators → C∗-algebras Vector lattice operators → lattice-ordered algebras Matrices of order n form an algebra under normal matrix multiplication

Daoud Clarke Language as an Algebra

SLIDE 12

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Algebra over a field

An algebra over a field:

abstraction of the space of operators on a vector space

Hilbert space operators → C∗-algebras Vector lattice operators → lattice-ordered algebras Matrices of order n form an algebra under normal matrix multiplication

Daoud Clarke Language as an Algebra

SLIDE 13

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Algebra over a field

An algebra over a field:

abstraction of the space of operators on a vector space

Hilbert space operators → C∗-algebras Vector lattice operators → lattice-ordered algebras Matrices of order n form an algebra under normal matrix multiplication

Daoud Clarke Language as an Algebra

SLIDE 14

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Algebra over a field

An algebra over a field:

abstraction of the space of operators on a vector space

Hilbert space operators → C∗-algebras Vector lattice operators → lattice-ordered algebras Matrices of order n form an algebra under normal matrix multiplication

Daoud Clarke Language as an Algebra

SLIDE 15

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Context-theoretic Framework

A context theory is a tuple A, A,ˆ, φ The meaning of a string x ∈ A∗ is a vector ˆ x ∈ A

A∗ is the free monoid on an alphabet A A is a real associative unital algebra

The mapping ˆ from A∗ to A is a monoid homomorphism

the cat sat =

the · cat · sat

Multiplication · is distributive with respect to vector space addition (bilinearity) φ is a linear functional on A indicating probability

Daoud Clarke Language as an Algebra

SLIDE 16

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Context-theoretic Framework

A context theory is a tuple A, A,ˆ, φ The meaning of a string x ∈ A∗ is a vector ˆ x ∈ A

A∗ is the free monoid on an alphabet A A is a real associative unital algebra

The mapping ˆ from A∗ to A is a monoid homomorphism

the cat sat =

the · cat · sat

Multiplication · is distributive with respect to vector space addition (bilinearity) φ is a linear functional on A indicating probability

Daoud Clarke Language as an Algebra

SLIDE 17

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Context-theoretic Framework

A context theory is a tuple A, A,ˆ, φ The meaning of a string x ∈ A∗ is a vector ˆ x ∈ A

A∗ is the free monoid on an alphabet A A is a real associative unital algebra

The mapping ˆ from A∗ to A is a monoid homomorphism

the cat sat =

the · cat · sat

Multiplication · is distributive with respect to vector space addition (bilinearity) φ is a linear functional on A indicating probability

Daoud Clarke Language as an Algebra

SLIDE 18

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Context-theoretic Framework

A context theory is a tuple A, A,ˆ, φ The meaning of a string x ∈ A∗ is a vector ˆ x ∈ A

A∗ is the free monoid on an alphabet A A is a real associative unital algebra

The mapping ˆ from A∗ to A is a monoid homomorphism

the cat sat =

the · cat · sat

Multiplication · is distributive with respect to vector space addition (bilinearity) φ is a linear functional on A indicating probability

Daoud Clarke Language as an Algebra

SLIDE 19

Background Context Theories Tools Distributional Semantics Context-theoretic Semantics

Context-theoretic Framework

A context theory is a tuple A, A,ˆ, φ The meaning of a string x ∈ A∗ is a vector ˆ x ∈ A

A∗ is the free monoid on an alphabet A A is a real associative unital algebra

The mapping ˆ from A∗ to A is a monoid homomorphism

the cat sat =

the · cat · sat

Multiplication · is distributive with respect to vector space addition (bilinearity) φ is a linear functional on A indicating probability

Daoud Clarke Language as an Algebra

SLIDE 20

Background Context Theories Tools Motivating Example Example

Motivating Example: Context Algebras

Meaning as context Formal language → syntactic monoid Fuzzy language → context algebra E.g. Latent Dirichlet Allocation → commutative algebra

Daoud Clarke Language as an Algebra

SLIDE 21

Background Context Theories Tools Motivating Example Example

Motivating Example: Context Algebras

Meaning as context Formal language → syntactic monoid Fuzzy language → context algebra E.g. Latent Dirichlet Allocation → commutative algebra

Daoud Clarke Language as an Algebra

SLIDE 22

Background Context Theories Tools Motivating Example Example

Motivating Example: Context Algebras

Meaning as context Formal language → syntactic monoid Fuzzy language → context algebra E.g. Latent Dirichlet Allocation → commutative algebra

Daoud Clarke Language as an Algebra

SLIDE 23

Background Context Theories Tools Motivating Example Example

Motivating Example: Context Algebras

Meaning as context Formal language → syntactic monoid Fuzzy language → context algebra E.g. Latent Dirichlet Allocation → commutative algebra

Daoud Clarke Language as an Algebra

SLIDE 24

Background Context Theories Tools Motivating Example Example

Context Algebras: How it works

Fuzzy language C : A∗ → [0, 1] For x ∈ A∗, define ˆ x : A∗ × A∗ → [0, 1] by ˆ x(y, z) = C(yxz) Define A as vector space generated by {ˆ x : x ∈ A∗} Choose a basis and use multiplication on A∗ to define multiplication on A

Multiplication is the same, no matter what basis we choose!

Daoud Clarke Language as an Algebra

SLIDE 25

Background Context Theories Tools Motivating Example Example

Context Algebras: How it works

Fuzzy language C : A∗ → [0, 1] For x ∈ A∗, define ˆ x : A∗ × A∗ → [0, 1] by ˆ x(y, z) = C(yxz) Define A as vector space generated by {ˆ x : x ∈ A∗} Choose a basis and use multiplication on A∗ to define multiplication on A

Multiplication is the same, no matter what basis we choose!

Daoud Clarke Language as an Algebra

SLIDE 26

Background Context Theories Tools Motivating Example Example

Context Algebras: How it works

Fuzzy language C : A∗ → [0, 1] For x ∈ A∗, define ˆ x : A∗ × A∗ → [0, 1] by ˆ x(y, z) = C(yxz) Define A as vector space generated by {ˆ x : x ∈ A∗} Choose a basis and use multiplication on A∗ to define multiplication on A

Multiplication is the same, no matter what basis we choose!

Daoud Clarke Language as an Algebra

SLIDE 27

Background Context Theories Tools Motivating Example Example

Context Algebras: How it works

Fuzzy language C : A∗ → [0, 1] For x ∈ A∗, define ˆ x : A∗ × A∗ → [0, 1] by ˆ x(y, z) = C(yxz) Define A as vector space generated by {ˆ x : x ∈ A∗} Choose a basis and use multiplication on A∗ to define multiplication on A

Multiplication is the same, no matter what basis we choose!

Daoud Clarke Language as an Algebra

SLIDE 28

Background Context Theories Tools Motivating Example Example

Context Algebras: Example

Corpus defined by C(the cat sat) = 0.8 C(the big cat sat) = 0.2 Then

= 0.8e(the, sat) + 0.2e(the big, sat)

big ·

cat = big cat = 0.2e(the, sat) where e(x, y) is the basis element corresponding to (x, y) ∈ A∗ × A∗.

Daoud Clarke Language as an Algebra

SLIDE 29

Background Context Theories Tools Motivating Example Example

Context Algebras: Example

Corpus defined by C(the cat sat) = 0.8 C(the big cat sat) = 0.2 Then

= 0.8e(the, sat) + 0.2e(the big, sat)

big ·

cat = big cat = 0.2e(the, sat) where e(x, y) is the basis element corresponding to (x, y) ∈ A∗ × A∗.

Daoud Clarke Language as an Algebra

SLIDE 30

Background Context Theories Tools Motivating Example Example

Context-theoretic Semantics: Summary

Map strings to elements of an algebra Motivating example:

Meaning as context Fuzzy language → context algebra

Lots of other examples in Clarke (2007)

Daoud Clarke Language as an Algebra

SLIDE 31

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Tools for Constructing Algebras

Quotient Algebras (Clarke, Lutz and Weir 2010) Finite dimensional algebras

with David Weir, Rudi Lutz and Ben Campion

Enveloping Algebras (You?)

Daoud Clarke Language as an Algebra

SLIDE 32

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Quotient Algebras

Construct a free algebra (tensor algebra) T(V) = R ⊕ V ⊕ (V ⊗ V) ⊕ (V ⊗ V ⊗ V) ⊕ · · · Choose relations u1 = v1, u2 = v2, . . . we wish to hold Λ = {u1 − v1, u2 − v2, . . .} Construct ideal I generated by Λ Take equivalence classes to get quotient algebra T(V)/I

Daoud Clarke Language as an Algebra

SLIDE 33

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Quotient Algebras

Construct a free algebra (tensor algebra) T(V) = R ⊕ V ⊕ (V ⊗ V) ⊕ (V ⊗ V ⊗ V) ⊕ · · · Choose relations u1 = v1, u2 = v2, . . . we wish to hold Λ = {u1 − v1, u2 − v2, . . .} Construct ideal I generated by Λ Take equivalence classes to get quotient algebra T(V)/I

Daoud Clarke Language as an Algebra

SLIDE 34

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Quotient Algebras

Construct a free algebra (tensor algebra) T(V) = R ⊕ V ⊕ (V ⊗ V) ⊕ (V ⊗ V ⊗ V) ⊕ · · · Choose relations u1 = v1, u2 = v2, . . . we wish to hold Λ = {u1 − v1, u2 − v2, . . .} Construct ideal I generated by Λ Take equivalence classes to get quotient algebra T(V)/I

Daoud Clarke Language as an Algebra

SLIDE 35

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Quotient Algebras

Construct a free algebra (tensor algebra) T(V) = R ⊕ V ⊕ (V ⊗ V) ⊕ (V ⊗ V ⊗ V) ⊕ · · · Choose relations u1 = v1, u2 = v2, . . . we wish to hold Λ = {u1 − v1, u2 − v2, . . .} Construct ideal I generated by Λ Take equivalence classes to get quotient algebra T(V)/I

Daoud Clarke Language as an Algebra

SLIDE 36

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Quotient Algebra: Why?

Vectors that were orthogonal in T(V) can be non-orthogonal in T(V)/I Strings of different lengths can be compared

Daoud Clarke Language as an Algebra

SLIDE 37

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Quotient Algebra: How?

An ideal I of an algebra A is a sub-vector space of A such that xa ∈ I and ax ∈ I for all a ∈ A and all x ∈ I Congruence: x ≡ y if x − y ∈ I Quotient algebra A/I formed from equivalence classes

Daoud Clarke Language as an Algebra

SLIDE 38

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Data-driven Composition

Use a treebank For each rule π : X → X1 . . . Xr with head Xh we add vectors λπ,i = ei − X1 ⊗ . . . ⊗ Xh−1 ⊗ ei ⊗ Xh+1 ⊗ . . . ⊗ Xr for each basis element ei of VXh to the generating set.

X is the sum over all individual vectors of subtrees rooted

with X

Preserve meaning of head

Daoud Clarke Language as an Algebra

SLIDE 39

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Example

Example Corpus see big city modernise city see modern city see red apple buy apple visit big apple read big book throw old small red book buy large new book Grammar: N′ → Adj N′ N′ → N Generating set Λ: λi = ei − Adj ⊗ ei

Adj = 2eapple +6ebook +2ecity

where ei ranges over basis vectors for noun contexts.

Daoud Clarke Language as an Algebra

SLIDE 40

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Example: Cosine Similarities

apple big apple red apple city big city red city book big book red book apple 1.0 0.26 0.24 0.52 0.13 0.12 0.33 0.086 0.080 big apple 1.0 0.33 0.13 0.52 0.17 0.086 0.33 0.11 red apple 1.0 0.12 0.17 0.52 0.080 0.11 0.33 city 1.0 0.26 0.24 0.0 0.0 0.0 big city 1.0 0.33 0.0 0.0 0.0 red city 1.0 0.0 0.0 0.0 book 1.0 0.26 0.24 big book 1.0 0.33 red book 1.0

Daoud Clarke Language as an Algebra

SLIDE 41

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Example: Cosine Similarities

apple big apple red apple city big city red city book big book red book apple 1.0 0.26 0.24 0.52 0.13 0.12 0.33 0.086 0.080 big apple 1.0 0.33 0.13 0.52 0.17 0.086 0.33 0.11 red apple 1.0 0.12 0.17 0.52 0.080 0.11 0.33 city 1.0 0.26 0.24 0.0 0.0 0.0 big city 1.0 0.33 0.0 0.0 0.0 red city 1.0 0.0 0.0 0.0 book 1.0 0.26 0.24 big book 1.0 0.33 red book 1.0

Daoud Clarke Language as an Algebra

SLIDE 42

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Example: Cosine Similarities

apple big apple red apple city big city red city book big book red book apple 1.0 0.26 0.24 0.52 0.13 0.12 0.33 0.086 0.080 big apple 1.0 0.33 0.13 0.52 0.17 0.086 0.33 0.11 red apple 1.0 0.12 0.17 0.52 0.080 0.11 0.33 city 1.0 0.26 0.24 0.0 0.0 0.0 big city 1.0 0.33 0.0 0.0 0.0 red city 1.0 0.0 0.0 0.0 book 1.0 0.26 0.24 big book 1.0 0.33 red book 1.0

Daoud Clarke Language as an Algebra

SLIDE 43

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Example: Cosine Similarities

apple big apple red apple city big city red city book big book red book apple 1.0 0.26 0.24 0.52 0.13 0.12 0.33 0.086 0.080 big apple 1.0 0.33 0.13 0.52 0.17 0.086 0.33 0.11 red apple 1.0 0.12 0.17 0.52 0.080 0.11 0.33 city 1.0 0.26 0.24 0.0 0.0 0.0 big city 1.0 0.33 0.0 0.0 0.0 red city 1.0 0.0 0.0 0.0 book 1.0 0.26 0.24 big book 1.0 0.33 red book 1.0

Daoud Clarke Language as an Algebra

SLIDE 44

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Finite-dimensional Algebras

Fix dimensionality n of the vector space Learn vectors for words and pairs of words using e.g. LSA Find the bilinear product on the vector space which best fits these vectors

Least squares Linear optimisation

Daoud Clarke Language as an Algebra

SLIDE 45

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Finite-dimensional Algebras: Results

Daoud Clarke Language as an Algebra

SLIDE 46

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Algebras from Monoids

Montague semantics with lambda calculus? Cartesian closed categories? Curry-Howard-Lambek correspondence? Enveloping C∗ algebras?

Daoud Clarke Language as an Algebra

SLIDE 47

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Algebras from Monoids: the Idea

Given any monoid S, we can construct an algebra Put complex syntactic and semantic information in S Then “vectorize” it using a standard construction Represent words as weighted sums of elements of S

Daoud Clarke Language as an Algebra

SLIDE 48

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

From Syntax and Semantics to Monoids

Let S be the set of all pairs (s, σ)

s is a syntactic type (e.g. in Lambek calculus) σ is semantics (e.g. a combination of lambda calculus and first order logic)

Multiplication defined by reduction to normal form

Lambek calculus ∼ residuated lattice Lambda calculus is a Cartesian closed category under βη-equivalence (Curry-Howard-Lambek correspondence)

Daoud Clarke Language as an Algebra

SLIDE 49

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

From Syntax and Semantics to Monoids

Let S be the set of all pairs (s, σ)

s is a syntactic type (e.g. in Lambek calculus) σ is semantics (e.g. a combination of lambda calculus and first order logic)

Multiplication defined by reduction to normal form

Lambek calculus ∼ residuated lattice Lambda calculus is a Cartesian closed category under βη-equivalence (Curry-Howard-Lambek correspondence)

Daoud Clarke Language as an Algebra

SLIDE 50

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

From Monoids to Algebras

Define multiplication on L1(S) by convolution: (u · v)(x) =

y,z∈S:yz=x

u(y)v(z) We want lattice properties of S to carry over

C∗ enveloping algebra? Need an involution on S Use right adjoint of Cartesian closed category?

Daoud Clarke Language as an Algebra

SLIDE 51

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

From Monoids to Algebras

Define multiplication on L1(S) by convolution: (u · v)(x) =

y,z∈S:yz=x

u(y)v(z) We want lattice properties of S to carry over

C∗ enveloping algebra? Need an involution on S Use right adjoint of Cartesian closed category?

Daoud Clarke Language as an Algebra

SLIDE 52

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Monoid to Algebra Example

Term Context vector fish (0, 0, 1) big (1, 2, 0) ni = (N, λx nouni(x)) ai = (N/N, λpλy adji(y) ∧ p.y) Define big = a1 + 2a2 and fish = n3, Then big · fish = a1n3 + 2a2n3, where ainj = (N, λx(nounj(x) ∧ adji(x))).

Daoud Clarke Language as an Algebra

SLIDE 53

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Monoid to Algebra Example

Term Context vector fish (0, 0, 1) big (1, 2, 0) ni = (N, λx nouni(x)) ai = (N/N, λpλy adji(y) ∧ p.y) Define big = a1 + 2a2 and fish = n3, Then big · fish = a1n3 + 2a2n3, where ainj = (N, λx(nounj(x) ∧ adji(x))).

Daoud Clarke Language as an Algebra

SLIDE 54

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Monoid to Algebra Example

Term Context vector fish (0, 0, 1) big (1, 2, 0) ni = (N, λx nouni(x)) ai = (N/N, λpλy adji(y) ∧ p.y) Define big = a1 + 2a2 and fish = n3, Then big · fish = a1n3 + 2a2n3, where ainj = (N, λx(nounj(x) ∧ adji(x))).

Daoud Clarke Language as an Algebra

SLIDE 55

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Conclusion

Hypothesis: meanings live in a unital associative real algebra Three ways to construct such algebras:

Quotient algebras — apply relations between vectors Search finite-dimensional algebras Wrap up a monoid in an algebra

Daoud Clarke Language as an Algebra

SLIDE 56

Background Context Theories Tools Quotient Algebras Finite-dimensional Algebras Algebras from Monoids

Conclusion

Hypothesis: meanings live in a unital associative real algebra Three ways to construct such algebras:

Quotient algebras — apply relations between vectors Search finite-dimensional algebras Wrap up a monoid in an algebra

Daoud Clarke Language as an Algebra