Normalizing tweets with edit scripts and recurrent neural - PowerPoint PPT Presentation

Nov 01, 2023 •167 likes •373 views

Normalizing tweets with edit scripts and recurrent neural embeddings Grzegorz Chrupaa | Tilburg University Normalizing tweets Convert tweets to canonical form easy to understand for downstream applications Examples I will c wat i can do

Normalizing tweets with edit scripts and recurrent neural embeddings Grzegorz Chrupała | Tilburg University
Normalizing tweets
Convert tweets to canonical form easy to understand for downstream applications
Examples I will c wat i can do I will see what I can do imma jus start puttn it out there I'm going to just start putting it out there
Approaches ● Noisy-channel-style ● Finite-state transducers ● Dictionary-based ○ Hand-crafted ○ Automatically constructed
Labeled vs unlabeled data ● Noisy-channel: P(target|source) ∝ P(source|target) × P(target) labeled unlabeled ● Dictionary lookup: ○ Induce dictionary from unlabeled data ○ Labeled data for parameter tuning
Discriminative model target = argmax target P(diff( source, target ) | source ) ● diff(·,·) transforms source to target ● P(·) is a Conditional Random Field
Signal from raw tweets included via learned text representations .
Architecture
Simple Recurrent Networks Elman, J. L. (1990). Finding structure in time. Cognitive science , 14 (2), 179-211.
Recurrent neural embeddings ● SRN trained to predict next character ● Representation: ● Embed string (at each position) in low- dimensional space
Visualizing embeddings String Nearest neighbors in embedding space should h should d will s will m should a @justth @neenu @raven_ @lanae @despic maybe u maybe y cause i wen i when i
diff - Edit script Input c _ w a t diff DEL INS(see) NIL INS(h) NIL Output see_ w ha t Each position in string labeled with edit op
Features ● Baseline n-gram features c _ w a t c_ _w wa at c_w _wa wat c_wa _wat c_wat ● SRN features ○ 400 MB raw Twitter feed ○ 400 hidden units ○ Activations discretized
Dataset ● Han, B., & Baldwin, T. (2011). Lexical normalisation of short text messages: Makn sens a# twitter. In ACL . ● 549 tweets, with normalized versions ● Only lexical normalizations
Results ● No-op make no changes ● Doc train on and label whole tweets ● OOV train on and label OOV-words
Compared to Han & Bo 2012 Method WER (%) No-op 11.2 S-dict 9.7 GHM-dict 7.6 HB-dict 6.6 Dict-combo 4.9 OOV NGRAM+SRN 4.7
Where SRN features helped 9 cont continued 5 gon gonna 4 bro brother 4 congrats congratulations 3 yall you 3 pic picture 2 wuz what’s 2 mins minutes 2 juss just 2 fb facebook
Conclusion ● Supervised discriminative model performs at state-of-the-art with little training data ● Neural text embeddings effectively incorporate signal from raw tweets

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I : Recurrent Neural Networks CHAPTER I Recurrent Neural Networks Introduction In this chapter first the

404 views • 27 slides

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views • 48 slides

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views • 27 slides

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks Stephan Oepen University of Oslo March 24, 2019 Our Roadmap Last Week Language structure: sequences, trees, graphs Recurrent Neural

217 views • 19 slides

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer University of Washington [most slides from Yejin Choi] RECURRENT NEURAL L NE NETWOR WORKS Recurrent Neural Networks (RNNs) Each input

787 views • 29 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

793 views • 21 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

583 views • 47 slides

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen University of Oslo March 10, 2019 Our Roadmap Today Language structure: sequences, trees, graphs Recurrent Neural Networks Different

780 views • 76 slides

Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep Learning textbook Ch. 10 Recurrent Neural Networks Long

667 views • 18 slides

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent Neural Networks Compact schematic drawing of standard multi-layer perceptron (MLP) output hidden input Modeling sequential data So far we

796 views • 50 slides

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova March 9, 2020 Alex Graves, Supervised Sequence Labelling with Recurrent Neural Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/

538 views • 18 slides

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random Variables Text Preprocessing Language Modeling Recurrent Neural Networks (RNN) LSTM Bidirectional RNN Deep RNN d2l.ai

798 views • 55 slides

RNN Input Layer RNN Hidden Layer RNN h t-1 h t x t (Picture adapted from Andrej

Outline Recurrent Neural Networks (RNNs) Neural Machine Transla/on NMT basics (Sutskever et al., 2014) ABenCon mechanism (Bahdanau et al., 2015) Marcello Federico 2016 Recurrent Neural Networks (RNNs) Based on slides kindly provided

437 views • 24 slides

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural Network Traditional RNNs encounter

229 views • 18 slides

Understanding Hidden Memories of Recurrent Neural Networks Yao Ming , Shaozu Cao, Ruixiang Zhang,

Understanding Hidden Memories of Recurrent Neural Networks Yao Ming , Shaozu Cao, Ruixiang Zhang, Zhen Li, Yuanzhe Chen, Yangqiu Song, Huamin Qu. THE HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY What is a Recurrent Neural Network? H K U S T

397 views • 37 slides

Introduction to the course RECURREN T N EURAL N ETW ORK S F OR LAN GUAGE MODELIN G IN P YTH ON

Introduction to the course RECURREN T N EURAL N ETW ORK S F OR LAN GUAGE MODELIN G IN P YTH ON David Cecchini Data Scientist Text data is available online RECURRENT NEURAL NETWORKS FOR LANGUAGE MODELING IN PYTHON Applications of machine

683 views • 33 slides

Recurrent neural network grammars Slide credits: Chris Dyer, Adhiguna Kuncoro Widespread

Recurrent neural network grammars Slide credits: Chris Dyer, Adhiguna Kuncoro Widespread phenomenon: Polarity items can only appear in certain contexts Example: anybody is a polarity item that tends to appear only in specific contexts:

1.31k views • 102 slides

Recurrent Recommendation with Local Coherence Jianling Wang and James Caverlee Dynamics in

Recurrent Recommendation with Local Coherence Jianling Wang and James Caverlee Dynamics in Recommenders Users and Items are constantly in flux. Local Coherence Within a short-term sequence, the neighboring items or users is

432 views • 16 slides

Sparse Attentive Backtracking: Temporal credit assignment through reminding Nan Rosemary Ke 1,2 ,

Sparse Attentive Backtracking: Temporal credit assignment through reminding Nan Rosemary Ke 1,2 , Anirudh Goyal 1 , Olexa Bilaniuk 1 , Jonathan Binas 1 Chris Pal 2,4 , Mike Mozer 3 , Yoshua Bengio 1,5 1 Mila, Universit e de Montr eal 2 Mila,

377 views • 6 slides

Geometry-Aware Deep Visual Learning Katerina Fragkiadaki zebras How this talk fits the workshop

Geometry-Aware Deep Visual Learning Katerina Fragkiadaki zebras How this talk fits the workshop We will discuss new neural architectures for video understanding and feature learning without human annotations We will still use SGD to

784 views • 57 slides

Neural Network Part 4: Recurrent Neural Networks Yingyu Liang Computer Sciences 760 Fall 2017

Neural Network Part 4: Recurrent Neural Networks Yingyu Liang Computer Sciences 760 Fall 2017 http://pages.cs.wisc.edu/~yliang/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed by Mark Craven,

384 views • 37 slides

Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap

Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) CMSC 678 UMBC Recap from last time Feed-Forward Neural Network: Multilayer Perceptron + 0 ) = (

1.04k views • 93 slides