assignment 1 corpus creation
play

Assignment 1 - Corpus Creation Marko Lo zaji c University of T - PowerPoint PPT Presentation

Oct 06, 2022 •209 likes •564 views

Assignment 1 - Corpus Creation Marko Lo zaji c University of T ubingen June 5, 2019 Marko Lo zaji c (University of T ubingen) Assignment 1 June 5, 2019 1 / 15 General remarks Great job! Marko Lo zaji c (University

  1. Assignment 1 - Corpus Creation Marko Loˇ zaji´ c University of T¨ ubingen June 5, 2019 Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 1 / 15

  2. General remarks Great job! Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 2 / 15

  3. General remarks Great job! Please include honor code Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 2 / 15

  4. General remarks Great job! Please include honor code Please do not commit to assignment repository after deadline Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 2 / 15

  5. General remarks Great job! Please include honor code Please do not commit to assignment repository after deadline Reminder: worst lab doesn’t count! Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 2 / 15

  6. General (Python) comment # Recommended way to open a file in Python 3: with open(file, "r") as f: ... # As opposed to: f = open(file, "r") ... f.close() # As well as: import io with io.open(file, "r") as f: ... Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 3 / 15

  7. Recap 1 Construct distinctive word lists for English and German 2 Collect tweets containing words in both languages 3 Have langdetect tell you how you did Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 4 / 15

  8. Part 1 - Getting the most frequent words (sketch) Goal: We want the 5000 most frequent English words and the most frequent 2000 German words, such that no words within the most common 20000 of the other language are selected. import gzip from collections import Counter with gzip.open(corpus, "rt", encoding="utf-8") as f: c = Counter() for line in f: for word in line.split(): if len(word) > 3: c[word] += 1 # or word.lower() # return all words, not just first 20,000! return [word for word, _freq in c.most_common()] Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 5 / 15

  9. Part 1 - Saving word lists (sketch) common_words = set(en_words[:20000]) \ & set(de_words[:20000]) en_wordlist = [] collected = 0 for word in en_words: if word not in common_words: en_wordlist.append(word) collected += 1 if collected == 5000: # 2000 for German ... # write words to file and break Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 6 / 15

  10. Part 1 - Saving word lists (Option 2) en_wordlist = set(en_words[:5000]) de_wordlist = set(de_words[:2000]) intersec = eng_wordlist & de_wordlist words_discarded = 0 while len(intersec) > 0: en_wordlist -= intersec de_wordlist -= intersec en_wordlist.add(set( eng_words[5000 + words_discarded: 5000 + words_discarded + len(intersec)])) de_wordlist.add(set( de_words[2000 + words_discarded: 2000 + words_discarded + len(intersec)])) words_discarded += len(intersec) intersec = en_wordlist & de_wordlist ... # write word lists to files Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 7 / 15

  11. Part 2 - Getting code-switched tweets Search API vs Stream API Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 8 / 15

  12. Part 2 - Getting code-switched tweets Search API vs Stream API Fighting rate limits Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 8 / 15

  13. Part 2 - Getting code-switched tweets Search API vs Stream API Fighting rate limits Fetching unique tweets Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 8 / 15

  14. Part 2 - Getting code-switched tweets Search API vs Stream API Fighting rate limits Fetching unique tweets Speeding up tweet hunt Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 8 / 15

  15. Part 2 - Getting code-switched tweets Some concrete possibilities for the search API: query? ”-filter:retweets”, part of German word list Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 9 / 15

  16. Part 2 - Getting code-switched tweets Some concrete possibilities for the search API: query? ”-filter:retweets”, part of German word list count? 100 Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 9 / 15

  17. Part 2 - Getting code-switched tweets Some concrete possibilities for the search API: query? ”-filter:retweets”, part of German word list count? 100 max id? Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 9 / 15

  18. Part 2 - Getting code-switched tweets Some concrete possibilities for the search API: query? ”-filter:retweets”, part of German word list count? 100 max id? geocode? ”48.6,11.5,400km”, list of countries Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 9 / 15

  19. Part 2 - Getting code-switched tweets Some concrete possibilities for the search API: query? ”-filter:retweets”, part of German word list count? 100 max id? geocode? ”48.6,11.5,400km”, list of countries tweet mode? ”extended” Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 9 / 15

  20. Part 2 - Getting code-switched tweets (sketch) en_wordset = read_words("en.words") de_wordset = read_words("de.words") for tweet in tweets: if len(tweet.full_text < 50): continue en, de = False, False for word in tweet: if word in en_wordset: en = True elif word in de_wordset: de = True if en and de: ... # dump to file, check if 50 tweets reached Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 10 / 15

  21. Gallery Like Italy is a 10 und Schweiz naja 7 oder so. Still high aber like nichts gegen Italien bitte Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 11 / 15

  22. Gallery Like Italy is a 10 und Schweiz naja 7 oder so. Still high aber like nichts gegen Italien bitte Die midlife crisis des millennial startet bei der ¨ Uberlegung sich beim poetry slam anzumelden Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 11 / 15

  23. Gallery Like Italy is a 10 und Schweiz naja 7 oder so. Still high aber like nichts gegen Italien bitte Die midlife crisis des millennial startet bei der ¨ Uberlegung sich beim poetry slam anzumelden Excuse me, wieso steht dieser komplett creepy aussehende B¨ ar da neben dem Altar?! Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 11 / 15

  24. Gallery Like Italy is a 10 und Schweiz naja 7 oder so. Still high aber like nichts gegen Italien bitte Die midlife crisis des millennial startet bei der ¨ Uberlegung sich beim poetry slam anzumelden Excuse me, wieso steht dieser komplett creepy aussehende B¨ ar da neben dem Altar?! Lots of sports ball fans excited for their sports ball and lots of polizei should they get too excited. Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 11 / 15

  25. Gallery...? Erste Vertragsgespr¨ ache werden nach dem Pokalspiel gegen Worms gef¨ uhrt. Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 12 / 15

  26. Gallery...? Erste Vertragsgespr¨ ache werden nach dem Pokalspiel gegen Worms gef¨ uhrt. Ok, dann muss ich halt Bill Gates werden, um von ¨ alteren angesehen zu werden lol Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 12 / 15

  27. Gallery...? Erste Vertragsgespr¨ ache werden nach dem Pokalspiel gegen Worms gef¨ uhrt. Ok, dann muss ich halt Bill Gates werden, um von ¨ alteren angesehen zu werden lol OMG wasn los??? Leider von hier aus keine Hilfe m¨ oglich... Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 12 / 15

  28. Gallery...? Erste Vertragsgespr¨ ache werden nach dem Pokalspiel gegen Worms gef¨ uhrt. Ok, dann muss ich halt Bill Gates werden, um von ¨ alteren angesehen zu werden lol OMG wasn los??? Leider von hier aus keine Hilfe m¨ oglich... Ja abwarten aber das ist der perfekte Mann f¨ ur die defensive gewesen schon mal. Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 12 / 15

  29. Gallery...? Erste Vertragsgespr¨ ache werden nach dem Pokalspiel gegen Worms gef¨ uhrt. Ok, dann muss ich halt Bill Gates werden, um von ¨ alteren angesehen zu werden lol OMG wasn los??? Leider von hier aus keine Hilfe m¨ oglich... Ja abwarten aber das ist der perfekte Mann f¨ ur die defensive gewesen schon mal. I had told Liz Johnston that their tote bag was going places Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 12 / 15

  30. Gallery...? Erste Vertragsgespr¨ ache werden nach dem Pokalspiel gegen Worms gef¨ uhrt. Ok, dann muss ich halt Bill Gates werden, um von ¨ alteren angesehen zu werden lol OMG wasn los??? Leider von hier aus keine Hilfe m¨ oglich... Ja abwarten aber das ist der perfekte Mann f¨ ur die defensive gewesen schon mal. I had told Liz Johnston that their tote bag was going places Bei mir hats auch gedauert.. Marko Loˇ zaji´ c (University of T¨ ubingen) Assignment 1 June 5, 2019 12 / 15

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

CORPUS CREATION FOR NEW GENRES: A Crowdsourced Approach to PP Attachment Mukund Jha, Jacob

CORPUS CREATION FOR NEW GENRES: A Crowdsourced Approach to PP Attachment Mukund Jha, Jacob

CORPUS CREATION FOR NEW GENRES: A Crowdsourced Approach to PP Attachment Mukund Jha, Jacob Andreas, Kapil Thadani, Sara Rosenthal, Kathleen McKeown Background Supervised techniques for text analysis require annotated data LDC

543 views • 37 slides
When? Investigating the temporal entities in a corpus Max Kemman University of Luxembourg

When? Investigating the temporal entities in a corpus Max Kemman University of Luxembourg

When? Investigating the temporal entities in a corpus Max Kemman University of Luxembourg November 8, 2015 Doing Digital History: Introduction to Tools and Technology Recap - Assignment How did the assignment go? What did you think of the

537 views • 36 slides
Who? Investigating the social entities in a corpus Max Kemman University of Luxembourg December

Who? Investigating the social entities in a corpus Max Kemman University of Luxembourg December

Who? Investigating the social entities in a corpus Max Kemman University of Luxembourg December 6, 2016 Doing Digital History: Introduction to Tools and Technology Where assignment How is the assignment going so far? Any questions about the

584 views • 35 slides
Corpus Creation for Disfluency Research Stephanie Strassel Linguistic Data Consortium

Corpus Creation for Disfluency Research Stephanie Strassel Linguistic Data Consortium

Corpus Creation for Disfluency Research Stephanie Strassel Linguistic Data Consortium {strassel@ldc.upenn.edu} DiSS 03 Workshop Introduction The Linguistic Data Consortium supports linguistic research, education and technology

334 views • 22 slides
Corpus Stylistics: Speech, Writing and Thought Presentation in a Corpus of English Writing

Corpus Stylistics: Speech, Writing and Thought Presentation in a Corpus of English Writing

Corpus Stylistics: Speech, Writing and Thought Presentation in a Corpus of English Writing (Routledge Advances in Corpus Linguistics) Elena Semino, Mick Short Click here if your download doesn&quot;t start automatically Corpus Stylistics:

386 views • 5 slides
The need for Corpus Statistics: Corpus analysis and the identification of linguistically relevant

The need for Corpus Statistics: Corpus analysis and the identification of linguistically relevant

The need for Corpus Statistics: Corpus analysis and the identification of linguistically relevant patterns Launching the Corpus Statistics Group 11 th Feb. 2016 University of Birmingham The Corpus Statistics group Core members (not just

460 views • 19 slides
Assignment #3 Which is something you may wish to do since it is Assignment #3 So You Want to

Assignment #3 Which is something you may wish to do since it is Assignment #3 So You Want to

Assignment #3 Which is something you may wish to do since it is Assignment #3 So You Want to Write some Procedural Shaders In fact Assignment #3 Assignments Goal is to be able to produce something nicer than Some advice:

92 views • 5 slides
Who? Investigating the social entities in a corpus Max Kemman University of Luxembourg December

Who? Investigating the social entities in a corpus Max Kemman University of Luxembourg December

Who? Investigating the social entities in a corpus Max Kemman University of Luxembourg December 7, 2015 Doing Digital History: Introduction to Tools and Technology Today Final assignment From Hermeneutics to Data to Networks

632 views • 38 slides
MACAQ : A Multi Annotated Corpus to study how we adapt Answers to various Questions Anne

MACAQ : A Multi Annotated Corpus to study how we adapt Answers to various Questions Anne

Overview Why a corpus of human answers? Corpus constitution Corpus annotation Conclusion MACAQ : A Multi Annotated Corpus to study how we adapt Answers to various Questions Anne Garcia-Fernandez , Sophie Rosset, Anne Vilnat LIMSI-CNRS and

562 views • 35 slides
A mas novas vos torn / Now I take you back Corpus to my tale Structure Corpus Study

A mas novas vos torn / Now I take you back Corpus to my tale Structure Corpus Study

Introduction Parallel Corpora A mas novas vos torn / Now I take you back Corpus to my tale Structure Corpus Study Conclusion The Romance of Flamenca References Olga Scrivner, E.D. Blodgett*, Sandra K ubler, Michael McGuire

526 views • 39 slides
TrustedOut Corpus Intelligence Corpus Intelligence Makes Intelligence Trustworthy. Florent Solt,

TrustedOut Corpus Intelligence Corpus Intelligence Makes Intelligence Trustworthy. Florent Solt,

TrustedOut Corpus Intelligence Corpus Intelligence Makes Intelligence Trustworthy. Florent Solt, CTO &amp; co-founder GESTE, Feb 20 th 2019, Paris. The problem: Distrust in media. TrustedOut Corpus Intelligence ?? ?? The consequence: In

437 views • 12 slides
From the National Corpus of Polish to the Polish Corpus Infrastructure Maciej Ogrodniczuk

From the National Corpus of Polish to the Polish Corpus Infrastructure Maciej Ogrodniczuk

From the National Corpus of Polish to the Polish Corpus Infrastructure Maciej Ogrodniczuk Linguistic Engineering Group Institute of Computer Science Polish Academy of Sciences SLOVKO 2019 Bratislava, 25 October 2019 Agenda Three main

437 views • 39 slides
CORPUS STYLISTICS: SPEECH, WRITING AND THOUGHT PRESENTATION IN A CORPUS OF ENGLISH WRITING

CORPUS STYLISTICS: SPEECH, WRITING AND THOUGHT PRESENTATION IN A CORPUS OF ENGLISH WRITING

CORPUS STYLISTICS: SPEECH, WRITING AND THOUGHT PRESENTATION IN A CORPUS OF ENGLISH WRITING Download Free Author: Elena Semino, Mick Short Number of Pages: 272 pages Published Date: 15 Aug 2014 Publisher: Taylor &amp; Francis Ltd Publication

82 views • 4 slides
Corpus Analysis from a Mathematical Perspective Corpus Statistics Research Group launch event

Corpus Analysis from a Mathematical Perspective Corpus Statistics Research Group launch event

Corpus Analysis from a Mathematical Perspective Corpus Statistics Research Group launch event Birmingham, 11th Feb 2016 Simon Preston (University of Nottingham) Joint work with R. Carrington, A. Hennessey, M. Mahlberg, K. Severn, Y. van Gennip,

510 views • 17 slides
Language Models Assignment of probabilities to sequences of words Can be used incrementally

Language Models Assignment of probabilities to sequences of words Can be used incrementally

N-Grams and Language Models Language Models Assignment of probabilities to sequences of words Can be used incrementally to predict the next word N-gram Sequence of n words (bigram, trigram, . . . ) The size of the corpus

365 views • 10 slides
ERROR ANALYSIS IN A WRITTEN LEARNER CORPUS FROM SPANISH SPEAKERS EFL LEARNERS. A CORPUS BASED

ERROR ANALYSIS IN A WRITTEN LEARNER CORPUS FROM SPANISH SPEAKERS EFL LEARNERS. A CORPUS BASED

ERROR ANALYSIS IN A WRITTEN LEARNER CORPUS FROM SPANISH SPEAKERS EFL LEARNERS. A CORPUS BASED STUDY Mara Victoria Pardo Rodrguez UCREL Session Lancaster University November 30th, 2017 Work plan 1. Problem summary, hypothesis, error

760 views • 28 slides
Modern Internet architecture, technology &amp; philosophy Advanced Internet Services Dept. of

Modern Internet architecture, technology &amp; philosophy Advanced Internet Services Dept. of

1 3/2/15 AIS 2015 Modern Internet architecture, technology &amp; philosophy Advanced Internet Services Dept. of Computer Science Columbia University Henning Schulzrinne Spring 2015 03/02/2015 2 3/2/15 AIS 2015 Objectives Why do

459 views • 41 slides
The Rise and Spread of Psychoheresy That your faith should not stand in the wisdom of men,

The Rise and Spread of Psychoheresy That your faith should not stand in the wisdom of men,

Is the Bible Sufficient? The Rise and Spread of Psychoheresy That your faith should not stand in the wisdom of men, but in the power of God. (1 Corinthians 2:5) So the spiritual battle ground is between the wisdom of God and the

1k views • 69 slides
A Good life For All in Oklahoma Project Goal To build capacity through a community of practice

A Good life For All in Oklahoma Project Goal To build capacity through a community of practice

INNOVATIONS IN SUPPORTING FAMILIES COMMUNITY OF PRACTICE FRAMEWORK FOR SYSTEMS CHANGE WEBINAR SERIES LifeCourse Tools in Action: A Good life For All in Oklahoma Project Goal To build capacity through a community of practice across and

945 views • 40 slides
The problem Consider the following rules specifying robots behavior: CUGS: Logic II 1 if the

The problem Consider the following rules specifying robots behavior: CUGS: Logic II 1 if the

Modeling and reasoning in propositional calculus Modeling and reasoning in propositional calculus Lecture 01 The problem Consider the following rules specifying robots behavior: CUGS: Logic II 1 if the surface is not dry then slow down or

1k views • 84 slides
A Review of Marketing Mix: 4Ps or More? Chai Lee Goi Department of Marketing &amp; Management,

A Review of Marketing Mix: 4Ps or More? Chai Lee Goi Department of Marketing &amp; Management,

Vol. 1, No. 1 International Journal of Marketing Studies A Review of Marketing Mix: 4Ps or More? Chai Lee Goi Department of Marketing &amp; Management, School of Business, Curtin University of Technology CDT 250, 98009 Miri, Sarawak, Malaysia

618 views • 14 slides
Source images - internet

Source images - internet

Source images - internet

666 views • 45 slides
Enterprise Computing: Software Development Professor Stephen Gilmore School of Informatics The

Enterprise Computing: Software Development Professor Stephen Gilmore School of Informatics The

Outline Reflection on the hackathon Suggestions for next time Innovative Learning Week Enterprise Computing: Software Development Professor Stephen Gilmore School of Informatics The University of Edinburgh February 12, 2015 Outline

341 views • 30 slides
PRIVACY IN UBIQUITOUS PRIVACY IN UBIQUITOUS Understanding Privacy Technical Approaches

PRIVACY IN UBIQUITOUS PRIVACY IN UBIQUITOUS Understanding Privacy Technical Approaches

Marc Langheinrich: Privacy in Ubiquitous Computing 5/29/2010 Todays Menu PRIVACY IN UBIQUITOUS PRIVACY IN UBIQUITOUS Understanding Privacy Technical Approaches COMPUTING Definitions Challenges 1. History and legal aspects 1.

264 views • 13 slides

More recommend