Fields for Information Extraction S U N I T A S A R A W A G I A N - - PowerPoint PPT Presentation
Semi-Markov Conditional Random Fields for Information Extraction S U N I T A S A R A W A G I A N D W I L L I A M C O H E N N I P S 2 0 0 4 P R E S E N T E D B Y : D I N E S H K H A N D E L W A L S L I D E S A R E A D O P T E D F R O
S U N I T A S A R A W A G I A N D W I L L I A M C O H E N N I P S 2 0 0 4
S L I D E S A R E A D O P T E D F R O M D A N I E L K H A S H A B I P R E S E N T E D B Y : D I N E S H K H A N D E L W A L
2
3
1 2 3 4 5 6 7 8 9 My review
Fermat’s last theorem by S. Singh
Other Other Other Title Title Title
Author Author
The relational connection occurs in many applications, NLP,
Traditionally in graphical models, Modeling the joint distribution can lead to difficulties rich local features occur in relational data, features may have complex dependencies,
constructing probability distribution over them is difficult
Solution: directly model the conditional, is sufficient for classification! CRF is simply a conditional distribution
𝐲 𝐠(i, 𝐲, 𝐳)
1 𝑎(𝐲) 𝑓𝐗T𝐆(𝐲,𝐳) Vector of local feature functions 𝑔𝐿(𝑗, 𝐲, 𝐳) ∈ 𝑆
Parameters to be estimated, 𝐗
=unobservable =observable
𝑔𝐿 𝑗, 𝐲, 𝐳 = 𝑔′𝐿 𝑗, 𝐲, 𝐳𝒋, 𝐳𝒋−𝟐
Binary values: Think of a feature as being on or off
Values that are relative to an object/class pair
Typically have lots and lots of features (100,000s of
𝑔1(𝑗, 𝐲, 𝐳)= 1 yi = DT and yi−1 = V 0,
𝑔2(𝑗, 𝐲, 𝐳)= 1 xi = the and yi = DT 0,
𝑔3(𝑗, 𝐲, 𝐳)= 1 suffix xi = "ing" and yi = V 0, ,
1 2 3 4 5 6 7 8 9 I went skiing with Fernando Pereira in British Columbia
O O O O I I O I I
Features describe the single word
t1=u1=1 t2=u2=2 t3=u3=3 t4= u4=4
t5=5, u5=6 t6= u6=7 t7=8, u7=9
I went skiing with Fernando Pereira in British Columbia
O O O O I O I
Features describe the segment from 𝑢𝑘 to 𝑣𝑘
𝑔𝐿 𝑗, 𝐲, 𝐳𝒋, 𝐳𝒋−𝟐 𝐿 y𝑘, y𝑘−1, 𝐲, 𝑢𝑘, 𝑣𝑘
𝑻𝟐 𝑻𝒒 s = 𝑡1, … , 𝑡𝑞 denote a segmentation of x Segment 𝑡
𝑘 = 𝑢𝑘, 𝑣𝑘, 𝑧𝑘
consists of a start position 𝑢𝑘, an end position 𝑣𝑘, and a label 𝑧𝑘 1 ≤ 𝑢𝑘 ≤ 𝑣𝑘 ≤ 𝑡 𝑢𝑘+1 = 𝑣𝑘 + 1 and
=unobservable =observable
𝐿 𝑘, 𝐲, 𝐭 = ′𝐿 y𝑘, y𝑘−1, 𝐲, 𝑢𝑘, 𝑣𝑘 P𝑠 𝐭 𝐲, 𝐗 =
1 𝑎(𝐲) 𝑓𝐗T𝐇(𝐲,𝐭)
𝑎(𝐲)= 𝑡′ 𝑓𝐗T𝐇(𝐲,𝐭) G 𝐲, 𝐭 = 𝒋=𝟐
𝐲 𝐡(i, 𝐲, 𝐭)
𝐡 is a vector of segment level feature functions.
𝐡 is a vector of segment level feature functions.
𝐭
𝒌=𝟐 𝐭
𝒛′,𝒆
𝐭′∈𝐭𝐣−𝐞:𝒛′
𝒌=𝟐 𝐭′
𝒛′,𝒆
𝐭′∈𝐭𝐣−𝐞:𝒛′
𝒌=𝟐 𝐭′
𝒛′,𝒆 𝐗T𝐡 𝑧, 𝑧′, 𝐲, 𝑗 − 𝑒, 𝑗
𝐭′∈𝐭𝐣−𝐞:𝒛′ 𝐗T 𝒌=𝟐 𝐭′
𝒛′,𝒆 𝑊 𝑗 − 𝑒, 𝑧′ + 𝐗T𝐡 𝑧, 𝑧′, 𝐲, 𝑗 − 𝑒, 𝑗
𝑊(𝑗, 𝑧) = max
𝒛′,𝒆=𝟐,..𝑴 𝑊 𝑗 − 𝑒, 𝑧′ + 𝐗T𝐡 𝑧, 𝑧′, 𝐲, 𝑗 − 𝑒, 𝑗
−∞ If i >0 If i = 0 If i<0 The optimal label sequence corresponds to path traced by max
𝑧
𝑊 𝑦 , 𝑧 .
Given the training data, we wish to learn parameters of
𝑂
𝑀 𝑋 =
𝑚
log 𝑄(𝒕𝑚|𝐲𝑚, 𝐗) =
𝑚
(𝐗T𝐇(𝐲𝑚, 𝒕𝑚)−log Z𝐗 (𝐲𝑚))
𝛼𝑀 𝑋 =
𝑚
(𝐇 𝐲𝑚, 𝒕𝑚 − 𝐹P𝑠 𝐭′ 𝐲, 𝐗 𝐇(𝐲𝑚, 𝐭′)) Observed feature count Expected feature count
𝛼𝑀 𝑋 =
𝑚
(𝐇 𝐲𝑚, 𝒕𝑚 − 𝐹P𝑠 𝐭′ 𝐲, 𝐗 𝐇(𝐲𝑚, 𝐭′)) 𝛼𝑀 𝑋 =
𝑚
𝐇 𝐲𝑚, 𝒕𝑚 − 𝑡′ 𝐇 𝐲𝑚, 𝑡′ 𝒇𝐗T𝐇(𝐲𝑚,𝑡′) Z𝐗 (𝐲𝑚) Markov property of G and a dynamic programming helps in fast computation of the expected value of the features under the current weight vector
𝐹P𝑠 𝐭′ 𝐲, 𝐗 𝐇(𝐲𝑚, 𝐭′)
𝛽(𝑗, 𝑧) =
𝑡′∈ 𝐭𝐣:𝐳
𝒇𝐗T𝐇(𝐲𝑚,𝑡′) Where 𝐭𝐣:𝐳 denotes all segmentations from 1 to 𝑗 ending at 𝑗 and labeled 𝑧. Z𝐗 (𝐲)=
𝑧
𝛽( 𝐲 , 𝑧)
𝒆=𝟐 𝑴 𝑧′∈𝓩
if i > 0 if i =0 if i < 0 A similar approach can be used to compute the expectation 𝑡′ 𝐇 𝐲𝑚, 𝑡′ 𝒇𝐗T𝐇(𝐲𝑚,𝑡′) 𝜃𝑙 𝑗, 𝑧 = 𝑡′∈ 𝐭𝐣:𝐳 𝐻𝑙 𝐲𝑚, 𝑡′ 𝒇𝐗T𝐇(𝐲𝑚,𝑡′) , restricted to the part of the segmentation ending at position i. 𝜃𝑙 𝑗, 𝑧 = 𝒆=𝟐
𝑴
𝑧′∈𝓩(𝜃𝑙 𝑗 − 𝑒, 𝑧′ + 𝛽 𝑗 − 𝑒, 𝑧′ ′𝐿 𝑧, 𝑧′, 𝐲, 𝑗 − 𝑒, 𝑗 )𝑓𝐗𝑈𝐡 𝑧,𝒛′,𝐲,𝑗−𝑒,𝑗
𝐹P𝑠 𝐭′ 𝐲, 𝐗 𝐇 𝐲, 𝐭′ = 𝟐 Z𝐗 (𝐲)
𝑧
𝜃𝑙( 𝐲 , 𝑧)
Barun,Gagan, Dhruvin,Yashoteja: This idea of reasoning over segments can be extended in the task of image segmentation. Nupur: Introducing constraints in the model to have something similar to CCM as in case of CRF. Happy: Apart from the similarity measures they have used, there is a very good similarity measure called Gower distance, which is primarily used for non- numerical data. I think, we can also use that here. Prachi: Compare SOTA deep learning models and semi-CRFs to building insights
architectures of both the models. Yashoteja: Start with L=1, and quickly filter out the regions of the sequence that we are confident to not contain any named entities. Now we can use L=2 and resegment only those regions where entities might lie. We can then proceed with L=3, etc. Intuition is similar to those in Apriori algorithm.
Baseline algorithms: CRF/1, labels words inside and outside entities with I and O, respectively. CRF/4, replaces the I tag with four tags B, E, C, and U, which depend on where the word appears in an entity. Datasets: The Address corpus contains 4,226 words, and consists of 395 home addresses of
corpus. The Jobs corpus contains 73,330 words, and consists of 300 computer related job postings. Paper considered extraction of company names and job titles. The 18,121-word Email corpus contains 216 email messages taken from the CSPACE email corpus , which is mail associated with a 14-week, 277-person management game. Paper considered extraction of person names.
CRF Features Indicators for specific words at location i, or locations within three words of i. Indicators for capitalization/letter patterns Indicators for the phrase inside a segment and the capitalization pattern inside a segment. Indicators for words and capitalization patterns in 3-word windows before and after the segment. Indicators for each segment length (d = 1, . . . ,L), and combined all word-level features with indicators for the beginning and end of a segment. Semi-CRF Features External dictionary of strings Internal segment dictionary 𝐡𝒕𝒋𝒏,𝑬 (𝐤, 𝐲, 𝐭)=𝑛𝑏𝑦𝑣∈𝐸 sim(𝑦𝑡𝑘, 𝑣) Dictionary based features:
Dhruvin, Prachi, Gagan - Precision/ Recall values not reported. Anshul- Why order-L CRFs perform much worse than semi-CRFs ? Nupur, Haroun- Comparison with only CRF ?