SLIDE 1
Natural Language Processing
Historical Document Transcription
Dan Klein — UC Berkeley
Joint work with Taylor Berg-Kirkpatrick and Greg Durrett [ACL 2013]
Historical Document
Natural Language Processing Historical Document Transcription Dan - - PowerPoint PPT Presentation
Natural Language Processing Historical Document Transcription Dan - - PowerPoint PPT Presentation
Natural Language Processing Historical Document Transcription Dan Klein UC Berkeley Joint work with Taylor Berg-Kirkpatrick and Greg Durrett [ACL 2013] Historical Document Historical Document Old Bailey Court Proceedings 1775 Transcription
SLIDE 2
SLIDE 3
Historical Document
Old Bailey Court Proceedings 1775
SLIDE 4
Transcription
Document Image
SLIDE 5
Transcription
Document Image
and Ch’: priftmer anhc bar. Jacob Lazarus and his IHP1 uh: prifoner. were both together when! rcccivcd lhczn. I fold eievén pair of than for xiirce guincas, and dclivcrcd the rcll'l:.in- d:r hack lo :11: prifuner. 1 fold ftvcn pairof filk to Mark Simpcr : nncpuir of mixcd. and. mo pair of Ifircad to lhz: foolnun, and on: pair of zhrzad to lh: barber. ' Q: What is the foolmarfs name? Fraum Mgfzr. I dun’: know. Hairy Hzrvir. l was flandingar the Camp Icr waizin far the thcrrilfs ufliceruo employ in: : Mo 3‘: daughter came for me to 0 am! take the prifoncr. 1 Wm! to |hc Old aailcy
Transcription (Google Tesseract)
SLIDE 6
Pipelined Approach
SLIDE 7
Pipelined Approach
SLIDE 8
Pipelined Approach
SLIDE 9
Pipelined Approach
SLIDE 10
Pipelined Approach
SLIDE 11
Pipelined Approach
m
SLIDE 12
Pipelined Approach
m o
SLIDE 13
Pipelined Approach
m o d
SLIDE 14
Historical Document
SLIDE 15
Unknown Fonts
SLIDE 16
Unknown Fonts
po
SLIDE 17
Unknown Fonts
po
SLIDE 18
Unknown Fonts
po
SLIDE 19
Unknown Fonts
long s glyph
SLIDE 20
Wandering Baseline
SLIDE 21
Wandering Baseline
SLIDE 22
Wandering Baseline
SLIDE 23
Wandering Baseline
SLIDE 24
Uneven Inking
SLIDE 25
Uneven Inking
SLIDE 26
Uneven Inking
SLIDE 27
Uneven Inking
SLIDE 28
Various Historical Documents
1725 1875 1823 1883:
SLIDE 29
Our Approach
SLIDE 30
Our Approach
po
SLIDE 31
Our Approach
po
SLIDE 32
Our Approach
po
SLIDE 33
Generative Model
p r i s o n e r
SLIDE 34
Generative Model
p r i s o n e r
SLIDE 35
Generative Model
p r i s o n e r
SLIDE 36
Generative Model
p r i s o n e r
SLIDE 37
Language Model
Generative Model
p r i s o n e r
SLIDE 38
Typesetting Model
Generative Model
p r i s o n e r
SLIDE 39
Typesetting Model
Generative Model
p r i s o n e r
SLIDE 40
Typesetting Model
Generative Model
p r i s o n e r
SLIDE 41
Typesetting Model
Generative Model
p r i s o n e r
SLIDE 42
Typesetting Model
Generative Model
p r i s o n e r
SLIDE 43
Typesetting Model
Generative Model
p r i s o n e r
SLIDE 44
Typesetting Model
Generative Model
p r i s o n e r
SLIDE 45
Typesetting Model
Generative Model
p r i s o n e r
SLIDE 46
Generative Model
p r i s o n e r
Typesetting Model
SLIDE 47
Generative Model
p r i s o n e r
Typesetting Model
SLIDE 48
Generative Model
p r i s o n e r
Typesetting Model
SLIDE 49
Generative Model
p r i s o n e r
Typesetting Model
SLIDE 50
Generative Model
p r i s o n e r
Typesetting Model
SLIDE 51
Generative Model
p r i s o n e r
Typesetting Model
SLIDE 52
Generative Model
p r i s o n e r
Rendering Model
SLIDE 53
Generative Model
p r i s o n e r
Rendering Model
SLIDE 54
Generative Model
p r i s o n e r
Rendering Model
SLIDE 55
Generative Model
p r i s o n e r
SLIDE 56
Generative Model
Language Model
P(E)
E
p r i s o n e r
SLIDE 57
Generative Model
Language Model
P(E)
E
Typesetting Model
· P(T|E)
T
p r i s o n e r
SLIDE 58
Generative Model
Language Model
P(E)
E
Typesetting Model
· P(T|E)
T X
Rendering Model
P(X|E, T)
p r i s o n e r
SLIDE 59
Generative Model
Language Model
P(E)
E
Typesetting Model
· P(T|E)
T X
Rendering Model
P(X|E, T)
SLIDE 60
Language Model
E
SLIDE 61
Language Model
E
SLIDE 62
Language Model
a
ei
r t
ei−1
ei+1
E
SLIDE 63
Language Model
a
ei
r t
ei−1
ei+1
Kneser-Ney smoothed character 6-gram
E
SLIDE 64
Typesetting Model
a
ei
SLIDE 65
T
Typesetting Model
a
ei
SLIDE 66
T
Typesetting Model
1 5
Left pad width
li
a
ei
SLIDE 67
T
Typesetting Model
1 5
Left pad width
1
30
Glyph box width
gi li
a
ei
SLIDE 68
T
Typesetting Model
1 5
Left pad width
1 5
Right pad width
1
30
Glyph box width
gi li ri
a
ei
SLIDE 69
T
a
Typesetting Model
1 5
Left pad width
1 5
Right pad width
1
30
Glyph box width
gi li ri
a
ei
SLIDE 70
T
a
Typesetting Model
1 5
Left pad width
1 5
Right pad width
1
30
Glyph box width
gi li ri
a
ei
SLIDE 71
T
a
Typesetting Model
vi
1 5
Left pad width
1 5
Right pad width
1
30
Glyph box width
a a a
Vertical offset
gi li ri
a
ei
SLIDE 72
T
a
Typesetting Model
di vi
1 5
Left pad width
1 5
Right pad width
1
30
Glyph box width
a a
a
a a
a
a
a
a
a
a
a a
a
a a
a
a a
a
a
a
a
a
a
a
a
a
a a
a
a
a
a a
a
a
a
Inking level
a a a
Vertical offset
gi li ri
a
ei
SLIDE 73
Rendering Model
SLIDE 74
Rendering Model
SLIDE 75
Rendering Model
Glyph box
SLIDE 76
Rendering Model
gi
Glyph box width Glyph box
SLIDE 77
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
Glyph box
SLIDE 78
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box
SLIDE 79
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box
SLIDE 80
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Glyph shape parameters
SLIDE 81
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Glyph shape parameters
SLIDE 82
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Bernoulli pixel probs Glyph shape parameters
SLIDE 83
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 84
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Glyph shape parameters Sample pixels Bernoulli pixel probs
SLIDE 85
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Glyph shape parameters Sample pixels Bernoulli pixel probs
SLIDE 86
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 87
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 88
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 89
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 90
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 91
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 92
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 93
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 94
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Sample pixels Bernoulli pixel probs Glyph shape parameters
SLIDE 95
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Glyph shape parameters Sample pixels Bernoulli pixel probs
SLIDE 96
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Glyph shape parameters Sample pixels Bernoulli pixel probs
SLIDE 97
X
Rendering Model
gi
Glyph box width
vi
Vertical
- ffset
di
Inking level Glyph box Glyph shape parameters Sample pixels Bernoulli pixel probs
SLIDE 98
Log-linear Interpolation
SLIDE 99
Log-linear Interpolation
Glyph shape parameters φ Bernoulli pixel probs θ
SLIDE 100
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ
SLIDE 101
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ
SLIDE 102
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ
SLIDE 103
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ
SLIDE 104
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ Interpolation weights α
SLIDE 105
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ Interpolation weights α Dot product α>φ
SLIDE 106
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ Interpolation weights α Dot product α>φ Apply logistic
SLIDE 107
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ Interpolation weights α
SLIDE 108
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ Interpolation weights α
SLIDE 109
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ Interpolation weights α
SLIDE 110
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ
j
j
Interpolation weights α
SLIDE 111
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ
j
j
θj ∝
Interpolation weights α
SLIDE 112
Log-linear Interpolation
Bernoulli pixel probs θ Glyph shape parameters φ
j
j
θj ∝ exp[α>
j φ]
Interpolation weights α
SLIDE 113
Learning and Inference
E
X T
SLIDE 114
Learning and Inference
- Learn font parameters using EM
E
X T
SLIDE 115
Learning and Inference
- Learn font parameters using EM
E
X T
1 5 1 5 1 30 SLIDE 116
Learning and Inference
- Learn font parameters using EM
E
X T
1 5 1 5 1 30 SLIDE 117
Learning and Inference
- Learn font parameters using EM
- Initialize font parameters with
mixtures of modern fonts
E
X T
1 5 1 5 1 30 SLIDE 118
Learning and Inference
- Learn font parameters using EM
- Initialize font parameters with
mixtures of modern fonts
- Semi-Markov DP to compute
expectations
E
X T
1 5 1 5 1 30 SLIDE 119
Learning and Inference
- Learn font parameters using EM
- Initialize font parameters with
mixtures of modern fonts
- Semi-Markov DP to compute
expectations
- Efficient inference using a coarse-
to-fine approach
E
X T
1 5 1 5 1 30 SLIDE 120
System Output Example
SLIDE 121
System Output Example
1 5 1 5 1 30 SLIDE 122
System Output Example how the murderers came to
1 5 1 5 1 30 SLIDE 123
System Output Example how the murderers came to
1 5 1 5 1 30 SLIDE 124
System Output Example how the murderers came to
1 5 1 5 1 30 SLIDE 125
System Output Example how the murderers came to
1 5 1 5 1 30 SLIDE 126
System Output Example how the murderers came to
1 5 1 5 1 30 SLIDE 127
System Output Example
SLIDE 128
System Output Example
1 5 1 5 1 30 SLIDE 129
System Output Example
taken ill and taken away -- I remember
1 5 1 5 1 30 SLIDE 130
System Output Example
taken ill and taken away -- I remember
1 5 1 5 1 30 SLIDE 131
System Output Example
taken ill and taken away -- I remember
1 5 1 5 1 30 SLIDE 132
System Output Example
taken ill and taken away -- I remember
1 5 1 5 1 30 SLIDE 133
Experiments
SLIDE 134
Experiments
Test data
SLIDE 135
- Old Bailey (1715-1905)
Experiments
Test data
SLIDE 136
- Old Bailey (1715-1905)
20 images, 30 lines each
Experiments
Test data
SLIDE 137
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
Experiments
Test data
SLIDE 138
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
10 images, 30 lines each
Experiments
Test data
SLIDE 139
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
10 images, 30 lines each
Experiments
Baselines Test data
SLIDE 140
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
10 images, 30 lines each
Experiments
- Google Tesseract
Baselines Test data
SLIDE 141
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
10 images, 30 lines each
Experiments
- Google Tesseract
- ABBYY FineReader 11
Baselines Test data
SLIDE 142
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
10 images, 30 lines each
Experiments
- Google Tesseract
- ABBYY FineReader 11
Baselines Language models Test data
SLIDE 143
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
10 images, 30 lines each
Experiments
- Google Tesseract
- ABBYY FineReader 11
Baselines
- New
York Times
Language models Test data
SLIDE 144
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
10 images, 30 lines each
Experiments
- Google Tesseract
- ABBYY FineReader 11
Baselines
- New
York Times 34M words NYT Gigaword
Language models Test data
SLIDE 145
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
10 images, 30 lines each
Experiments
- Google Tesseract
- ABBYY FineReader 11
Baselines
- New
York Times 34M words NYT Gigaword
- Old Bailey
Language models Test data
SLIDE 146
- Old Bailey (1715-1905)
20 images, 30 lines each
- Trove (1803-1893)
10 images, 30 lines each
Experiments
- Google Tesseract
- ABBYY FineReader 11
Baselines
- New
York Times 34M words NYT Gigaword
- Old Bailey
32M words manually transcribed
Language models Test data
SLIDE 147
Results
10 20 30 40 50 60
Word Error Rate
Old Bailey Court Proceedings (1715-1905)
SLIDE 148
Results
10 20 30 40 50 60
54.8
Word Error Rate
Old Bailey Court Proceedings (1715-1905)
Google Tesseract
SLIDE 149
Results
10 20 30 40 50 60
40.0 54.8
Word Error Rate
Old Bailey Court Proceedings (1715-1905)
Google Tesseract ABBYY FineReader
SLIDE 150
Results
10 20 30 40 50 60
28.1
40.0 54.8
Word Error Rate
Old Bailey Court Proceedings (1715-1905)
Google Tesseract ABBYY FineReader Ocular w/ NYT
[Berg-Kirkpatrick et al. 2013]
SLIDE 151
Results
10 20 30 40 50 60 24.1
28.1
40.0 54.8
Word Error Rate
Old Bailey Court Proceedings (1715-1905)
Google Tesseract ABBYY FineReader Ocular w/ NYT Ocular w/ OB
[Berg-Kirkpatrick et al. 2013]
SLIDE 152
Trove Historical Newspapers (1803-1893)
Results
10 20 30 40 50 60
Word Error Rate
SLIDE 153
Trove Historical Newspapers (1803-1893)
Results
10 20 30 40 50 60
59.3
Word Error Rate
Google Tesseract
SLIDE 154
Trove Historical Newspapers (1803-1893)
Results
10 20 30 40 50 60
49.2 59.3
Word Error Rate
Google Tesseract ABBYY FineReader
SLIDE 155
Trove Historical Newspapers (1803-1893)
Results
10 20 30 40 50 60
33.0 49.2 59.3
Word Error Rate
Google Tesseract ABBYY FineReader Ocular w/ NYT
SLIDE 156
Trove Historical Newspapers (1803-1893)
Results
10 20 30 40 50 60
25.6 49.2 59.3
Word Error Rate
Google Tesseract ABBYY FineReader Ocular w/ NYT
[Berg-Kirkpatrick et al. 2014]
SLIDE 157
Transcription
and Ch’: priftmer anhc bar. Jacob Lazarus and his IHP1 uh: prifoner. were both together when! rcccivcd lhczn. I fold eievén pair of than for xiirce guincas, and dclivcrcd the rcll'l:.in- d:r hack lo :11: prifuner. 1 fold ftvcn pairof filk to Mark Simpcr : nncpuir of mixcd. and. mo pair of Ifircad to lhz: foolnun, and on: pair of zhrzad to lh: barber. ' Q: What is the foolmarfs name? Fraum Mgfzr. I dun’: know. Hairy Hzrvir. l was flandingar the Camp Icr waizin far the thcrrilfs ufliceruo employ in: : Mo 3‘: daughter came for me to 0 am! take the prifoncr. 1 Wm! to |hc Old aailcy
Google Tesseract
SLIDE 158
Transcription
and Ch’: priftmer anhc bar. Jacob Lazarus and his IHP1 uh: prifoner. were both together when! rcccivcd lhczn. I fold eievén pair of than for xiirce guincas, and dclivcrcd the rcll'l:.in- d:r hack lo :11: prifuner. 1 fold ftvcn pairof filk to Mark Simpcr : nncpuir of mixcd. and. mo pair of Ifircad to lhz: foolnun, and on: pair of zhrzad to lh: barber. ' Q: What is the foolmarfs name? Fraum Mgfzr. I dun’: know. Hairy Hzrvir. l was flandingar the Camp Icr waizin far the thcrrilfs ufliceruo employ in: : Mo 3‘: daughter came for me to 0 am! take the prifoncr. 1 Wm! to |hc Old aailcy
Google Tesseract
the prisoner at the bar. Jacob Lazarus and his wife, the prisoners were both together when I received them. I sold eleven pair of them for three guineas, and delivered the remain- der back to the prisoner. I sold, seven pair of silk to Mark Simpert one pair of mixed, and two pair of thread to the footman, and one pair of thread to the barber,
- Ms. What in the footman's name?
Franco Asyut, I don't know- Nearly Norris. I was standing at the Comp- ter waiting for the sherrill's officers to employ me a Moses's daughter came for me to go and take the prisoner. I went to the Old Bailey
Ocular
SLIDE 159
Learned Fonts
Initializer
SLIDE 160
Learned Fonts
Initializer
g
SLIDE 161
Learned Fonts
Initializer
g
SLIDE 162
1700 1740 1780 1820 1860 1900
Learned Fonts
Initializer
SLIDE 163
Unobserved Pixels
SLIDE 164
Unobserved Pixels
SLIDE 165
Unobserved Pixels
SLIDE 166
Unobserved Pixels
SLIDE 167
Conclusion
SLIDE 168
Conclusion
- Unsupervised font learning yields state-of-the-art
results on documents where font is unknown
SLIDE 169
Conclusion
- Unsupervised font learning yields state-of-the-art
results on documents where font is unknown
- Generatively modeling sources of noise specific to
printing-press era documents is effective
SLIDE 170
Conclusion
- Unsupervised font learning yields state-of-the-art
results on documents where font is unknown
- Generatively modeling sources of noise specific to
printing-press era documents is effective
- Ocular available as a downloadable tool:
nlp.cs.berkeley.edu/ocular.shtml
SLIDE 171
Conclusion