a pipeline for scalable text reuse analysis
play

A Pipeline for Scalable Text Reuse Analysis Milad Alshomary - PowerPoint PPT Presentation

Jun 11, 2023 •123 likes •1.01k views

A Pipeline for Scalable Text Reuse Analysis Milad Alshomary Bauhaus Universitt 05.07.2018 Milad Alshomary Pipeline for TR extraction 05.07.2018 1 Overview Motivation A Pipeline for Scalable Text Reuse Extraction Application on

  1. A Pipeline for Scalable Text Reuse Analysis Milad Alshomary Bauhaus Universität 05.07.2018 Milad Alshomary Pipeline for TR extraction 05.07.2018 1

  2. Overview ● Motivation ● A Pipeline for Scalable Text Reuse Extraction ● Application on Wikipedia ● Application on Wikipedia and Common Crawl ● Conclusion Milad Alshomary Pipeline for TR extraction 05.07.2018 2

  3. Motivation Text Reuse (TR) ● Quoting ● Verbatim ● Paraphrasing ● Translation ● Summarization Milad Alshomary Pipeline for TR extraction 05.07.2018 3

  4. Motivation TR Detection Applications Plagiarism detection METER project (Measuring Text Reuse) Milad Alshomary Pipeline for TR extraction 05.07.2018 4

  5. Motivation TR Detection Applications Plagiarism detection METER projet (Measuring Text Reuse) Milad Alshomary Pipeline for TR extraction 05.07.2018 5

  6. Motivation TR Detection Applications Plagiarism detection METER projet (Measuring Text Reuse) Milad Alshomary Pipeline for TR extraction 05.07.2018 6

  7. Motivation Wikipedia vs The World ● Digital Encyclopedia ● Collaborative environment ● Giant public source of information ● Free to use Milad Alshomary Pipeline for TR extraction 05.07.2018 7

  8. Motivation Wikipedia vs The World ● Digital Encyclopedia ● Collaborative environment ● Giant public source of information ● Free to use Milad Alshomary Pipeline for TR extraction 05.07.2018 8

  9. Motivation Wikipedia vs The World ● Digital Encyclopedia ● Collaborative environment ● Giant public source of information ● Free to use Milad Alshomary Pipeline for TR extraction 05.07.2018 9

  10. Motivation Wikipedia vs The World ● Digital Encyclopedia ● Collaborative environment ● Giant public source of information ● Free to use Milad Alshomary Pipeline for TR extraction 05.07.2018 10

  11. Motivation Wikipedia vs The World Quality Flaws Scientific community Milad Alshomary Pipeline for TR extraction 05.07.2018 11

  12. Motivation Wikipedia vs The World - Web pages = Wikipedia text + advertisements Milad Alshomary Pipeline for TR extraction 05.07.2018 12

  13. Motivation Research Questions ➔ What kinds of text reuse occur within Wikipedia? ➔ How much of the web is a copy of Wikipedia content? ➔ How much revenue does this content generate? Milad Alshomary Pipeline for TR extraction 05.07.2018 13

  14. Motivation Research Questions ➔ What kinds of text reuse occur within Wikipedia? ➔ How much of the web is a copy of Wikipedia content? ➔ How much revenue does this content generate? Milad Alshomary Pipeline for TR extraction 05.07.2018 14

  15. Motivation Research Questions ➔ What kinds of text reuse occur within Wikipedia? ➔ How much of the web is a copy of Wikipedia content? ➔ How much revenue does this content generate? Milad Alshomary Pipeline for TR extraction 05.07.2018 15

  16. A Pipeline for Scalable Text Reuse Extraction Milad Alshomary Pipeline for TR extraction 05.07.2018 16

  17. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline D1 TR Pipeline Input: Two datasets ➔ Output: Text reuse ➔ D2 cases Milad Alshomary Pipeline for TR extraction 05.07.2018 17

  18. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline D1 TR Pipeline Input: Two datasets ➔ Output: Text reuse ➔ D2 cases Milad Alshomary Pipeline for TR extraction 05.07.2018 18

  19. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline Text Candidate Text Preprocessing Elimination Alignment Content extraction ➔ Chunking ➔ Feature extraction ➔ Milad Alshomary Pipeline for TR extraction 05.07.2018 19

  20. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline Text Candidate Text Preprocessing Elimination Alignment Content extraction Pairwise scan ➔ ➔ Chunking Text Reuse heuristics ➔ ➔ Feature extraction ➔ Milad Alshomary Pipeline for TR extraction 05.07.2018 20

  21. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline Text Candidate Text Preprocessing Elimination Alignment Content extraction Pairwise scan Detailed scan of text ➔ ➔ ➔ Chunking Text Reuse heuristics reuse ➔ ➔ Feature extraction Picapica framework ➔ ➔ Milad Alshomary Pipeline for TR extraction 05.07.2018 21

  22. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Text Candidate Text Preprocessing Elimination Alignment Keys for scaling-up: Cluster computing ➔ Heuristics based candidate elimination ➔ algorithms Milad Alshomary Pipeline for TR extraction 05.07.2018 22

  23. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Text Candidate Text Preprocessing Elimination Alignment Keys for scaling-up: Cluster computing ➔ Heuristics based candidate elimination ➔ algorithms Milad Alshomary Pipeline for TR extraction 05.07.2018 23

  24. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 1 D 2 For a candidacy function we proposed the following methods: - Cosine similarity of TF-IDF (semantic) - Paragraph embedding (semantic) candidacy(d 11 , d 21 ) → [0, 1] - Stopwords N-grams (structure) - Weighted average of Stopwords Ngrams and d 11 d 21 Paragraph embedding (semantic + structure) d 22 d 12 d 2n d 1n Milad Alshomary Pipeline for TR extraction 05.07.2018 24

  25. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Sample 1k documents Document Generate TR Sample from Wikipedia: Wikipedia Sample - Sample 1k documents from Wikipedia Text alignment using - Using Picapica framework to find picapica framework TR cases TR sample Milad Alshomary Pipeline for TR extraction 05.07.2018 25

  26. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Sample 1k documents Document Generate TR Sample from Wikipedia: Wikipedia Sample - Sample 1k documents from Wikipedia Text alignment using - Using Picapica framework to find picapica framework TR cases TR sample Milad Alshomary Pipeline for TR extraction 05.07.2018 26

  27. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Sample 1k documents Document Generate TR Sample from Wikipedia: Wikipedia Sample - Sample 1k documents from Wikipedia Text alignment using - Using Picapica framework to find picapica framework TR cases TR sample - 232 documents - ~ 90% have < 10 alignements (TR case) Milad Alshomary Pipeline for TR extraction 05.07.2018 27

  28. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Evaluation of “ candidacy” function: TR sample - For each document in TR sample: - Sort all Wikipedia articles according to the proposed “ candidacy” . - Precision/Recall on Thresholds of [1, 101,..,100k] - A True Positive (TP) is a pair of documents that have TR. T1 T2 T3 Milad Alshomary Pipeline for TR extraction 05.07.2018 28

  29. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Evaluation of “ candidacy” function: TR sample p1 p2 r2 r1 T1 T2 T3 Milad Alshomary Pipeline for TR extraction 05.07.2018 29

  30. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Semantic hashing function: - Hashes documents into binary hashes. - Similar documents get similar or exact binary hash. 011001 011001 Milad Alshomary Pipeline for TR extraction 05.07.2018 30

  31. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 2 D 1 Semantic hashing function: - Hashing all documents. - Inverted index. - Hash document’s chunks. - Apply candidacy function only on Inverted index documents that intersect in one 001001 hash at least. 011001 011001 011001 001000 Milad Alshomary Pipeline for TR extraction 05.07.2018 31

  32. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 2 D 1 Semantic hashing function: - Hashing all documents. - Inverted index. - Hash document’s chunks. - Apply candidacy function only on Inverted index documents that intersect in one 001001 hash at least. 011001 011001 011001 001000 Milad Alshomary Pipeline for TR extraction 05.07.2018 32

  33. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 2 D 1 Semantic hashing function: - Hashing all documents. - Inverted index. - Hash document’s chunks. - Apply candidacy function only on Inverted index documents that intersect in one 001001 hash at least. 011001 011001 011001 001000 Milad Alshomary Pipeline for TR extraction 05.07.2018 33

  34. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 2 D 1 Semantic hashing function: - Hashing all documents. - Inverted index. - Hash document’s chunks. - Apply candidacy function only on Inverted index documents that intersect in one 001001 hash at least. 011001 011001 011001 001000 Milad Alshomary Pipeline for TR extraction 05.07.2018 34

  35. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination d i Proposed semantic hashing methods: - Random Projection (data independent) - Variational Deep Semantic Hashing (data dependent) d j Milad Alshomary Pipeline for TR extraction 05.07.2018 35

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

TRACER TUTORIAL: TEXT REUSE DETECTION INTRODUCTION TO HISTORICAL TEXT REUSE DETECTION M arco B

TRACER TUTORIAL: TEXT REUSE DETECTION INTRODUCTION TO HISTORICAL TEXT REUSE DETECTION M arco B

TRACER TUTORIAL: TEXT REUSE DETECTION INTRODUCTION TO HISTORICAL TEXT REUSE DETECTION M arco B uchler, Emily Franzini and Greta Franzini TABLE OF CONTENTS 1. Who am I? 2. What is text reuse? 3. Aspects of text reuse 4. ACID for the Digital

584 views • 34 slides
Orange County Water Reuse Peters Canyon Wash Channel Water Capture and Reuse Pipeline Project

Orange County Water Reuse Peters Canyon Wash Channel Water Capture and Reuse Pipeline Project

Orange County Water Reuse Peters Canyon Wash Channel Water Capture and Reuse Pipeline Project April 20, 2017 Agenda History and Approach Design and Construction Lessons and Challenges Questions 2 History and Approach Jian Peng, PhD

551 views • 26 slides
TRACER TUTORIAL: TEXT REUSE DETECTION RECENT WORK M arco B uchler, Emily Franzini and Greta

TRACER TUTORIAL: TEXT REUSE DETECTION RECENT WORK M arco B uchler, Emily Franzini and Greta

TRACER TUTORIAL: TEXT REUSE DETECTION RECENT WORK M arco B uchler, Emily Franzini and Greta Franzini METHODOLOGY Basic idea: Embed historical text reuse in Shannons N oisy Channel theorem. 2/26 MICROVIEW II Source: Stefan J anicke,

433 views • 26 slides
Text Reuse Detection Using a Composition of Text Similarity Measures Br, Zesch, Gurevych 2012

Text Reuse Detection Using a Composition of Text Similarity Measures Br, Zesch, Gurevych 2012

Text Reuse Detection Using a Composition of Text Similarity Measures Br, Zesch, Gurevych 2012 HS Computational study of Sabrina Galasso linguistic differences sabrina.galasso@student.uni-tuebingen.de December 16th, 2015 HS LingDiff 1 1.

1.46k views • 32 slides
Efficient Analysis of Pipeline Models for WCET Computation Stephan Wilhelm (sw@absint.com)

Efficient Analysis of Pipeline Models for WCET Computation Stephan Wilhelm (sw@absint.com)

A b s I n t Efficient Analysis of Pipeline Models for WCET Computation Stephan Wilhelm (sw@absint.com) AbsInt GmbH and Saarland University Outline 1. aiTs pipeline analysis 3. Why improve its efficiency? 5. BDD based pipeline analysis 7.

300 views • 10 slides
1 Infrastructure Requirements Limit Reuse Planned Indirect Potable Reuse (Purple pipe may be a

1 Infrastructure Requirements Limit Reuse Planned Indirect Potable Reuse (Purple pipe may be a

Overview of Presentation DIRECT POTABLE REUSE: A PATH FORWARD: Take Home Message Reuse Opportunities De Facto and Planned Indirect Potable Reuse 2012 WATER REUSE CONFERENCE Proposed Direct Potable Reuse Strategies Boise, ID

599 views • 8 slides
KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher

KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher

KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher Fondazione Bruno Kessler (FBK) rospocher@fbk.eu :: @marcorospocher KnowledgeStore A scalable, fault-tolerant, and Semantic Web grounded storage

695 views • 23 slides
Scalable Data Analytics Pipeline for Real-Time Attack Detection; Design, Validation, and

Scalable Data Analytics Pipeline for Real-Time Attack Detection; Design, Validation, and

Scalable Data Analytics Pipeline for Real-Time Attack Detection; Design, Validation, and Deployment in a Honeypot Environment Eric Badger Masters Student Computer Engineering 1 Overview Introduction/Motivation Challenges

508 views • 30 slides
ELECTRONIC TEXT REUSE ACQUISITION PROJECT INTRODUCTION &amp; MOTIVATION M arco Bchler TABLE OF

ELECTRONIC TEXT REUSE ACQUISITION PROJECT INTRODUCTION &amp; MOTIVATION M arco Bchler TABLE OF

ELECTRONIC TEXT REUSE ACQUISITION PROJECT INTRODUCTION &amp; MOTIVATION M arco Bchler TABLE OF CONTENTS 2/100 WHO AM I? WHO AM I? 2001-2002: Head of Quality Assurance department in a software company; 2006: Diploma in Computer

494 views • 23 slides
Social Media &amp; Text Analysis lecture 5 - POS/NE Tagging CSE 5539-0010 Ohio State University

Social Media &amp; Text Analysis lecture 5 - POS/NE Tagging CSE 5539-0010 Ohio State University

Social Media &amp; Text Analysis lecture 5 - POS/NE Tagging CSE 5539-0010 Ohio State University Instructor: Alan Ritter Website: socialmedia-class.org NLP Pipeline (summary so far) classification Regular (Nave Bayes) Expression Part-of-

1.17k views • 60 slides
Scalable Full-Text Search for Petascale File Systems Andrew W. Leung Ethan L. Miller

Scalable Full-Text Search for Petascale File Systems Andrew W. Leung Ethan L. Miller

Scalable Full-Text Search for Petascale File Systems Andrew W. Leung Ethan L. Miller University of California, Santa Cruz 3 rd Petascale Data Storage Workshop (PDSW 08) November 17th, 2008 Need scalable file management Todays

307 views • 12 slides
KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher,

KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher,

KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher, Bernardo Magnini, Luciano Serafini Fondazione Bruno Kessler (FBK) INTRODUCTION Information is typically available both in unstructured and

707 views • 43 slides
Scalable Data Analytics Pipeline for Validation of Real-Time Attack Detection Eric Badger , Phuong

Scalable Data Analytics Pipeline for Validation of Real-Time Attack Detection Eric Badger , Phuong

Scalable Data Analytics Pipeline for Validation of Real-Time Attack Detection Eric Badger , Phuong Cao, Alex Withers, Adam Slagell, Zbigniew Kalbarczyk, Ravi Iyer University of Illinois Urbana-Champaign 1 Overview Introduction/Motivation

283 views • 24 slides
Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav Tkachenko Senior Data Engineer at Activision 1+ Data lake size (AWS S3) PB Number of topics in the biggest cluster 500+ (Apache Kafka)

548 views • 44 slides
Text Mining in Hebrew Impact of Morphology Analysis on Topic Analysis and on Search Quality

Text Mining in Hebrew Impact of Morphology Analysis on Topic Analysis and on Search Quality

Text Mining in Hebrew Impact of Morphology Analysis on Topic Analysis and on Search Quality Michael Elhadad, Meni Adler, Yoav Goldberg, Dudi Gabay and Yael Netzer Text in Hebrew Extract information from text in Hebrew Major immediate

1k views • 88 slides
Introduction to Historical Text Reuse Detection Marco Bchler, Emily Franzini, Greta Franzini,

Introduction to Historical Text Reuse Detection Marco Bchler, Emily Franzini, Greta Franzini,

Introduction to Historical Text Reuse Detection Marco Bchler, Emily Franzini, Greta Franzini, Maria Moritz eTRAP Research Group Gttingen Centre for Digital Humanities Institute of Computer Science Georg August University Gttingen, Germany

714 views • 25 slides
Standardization of procedures and reporting in cryopreservation kos Horvth DEPARTMENT OF

Standardization of procedures and reporting in cryopreservation kos Horvth DEPARTMENT OF

Standardization of procedures and reporting in cryopreservation kos Horvth DEPARTMENT OF AQUACULTURE Overview Why are we cryopreserving sperm? Identification of the problem Possible solutions to the problem Does

398 views • 25 slides
Committee November 17, 2017 HEALTH POLICY &amp; ANALYTICS Office of Health Analytics Consent

Committee November 17, 2017 HEALTH POLICY &amp; ANALYTICS Office of Health Analytics Consent

Metrics &amp; Scoring Committee November 17, 2017 HEALTH POLICY &amp; ANALYTICS Office of Health Analytics Consent agenda Review todays agenda HPQMC debrief Update on work related to development of health aspects of kindergarten

364 views • 33 slides
Intonational sentence-type conventions for perlocutionary effects: an experimental investigation

Intonational sentence-type conventions for perlocutionary effects: an experimental investigation

Intonational sentence-type conventions for perlocutionary effects: an experimental investigation link to paper https://github.com/sunwooj/perlocution Sunwoo Jeong &amp; Christopher Potts Department of Linguistics, Stanford University A classic

702 views • 40 slides
Specific Heats of Ideal Gases Assume that a pure, ideal gas is made of tiny particles that bounce

Specific Heats of Ideal Gases Assume that a pure, ideal gas is made of tiny particles that bounce

Specific Heats of Ideal Gases Assume that a pure, ideal gas is made of tiny particles that bounce into each other and the walls of their cubic container of side . Show the average pressure P exerted by this gas is P = 1 N 35 (J/K-mole) V mv

488 views • 22 slides
Abstract: Permian basin and its impact on Americas supply equation Speaker Bio .: David

Abstract: Permian basin and its impact on Americas supply equation Speaker Bio .: David

Abstract: Permian basin and its impact on Americas supply equation Speaker Bio .: David Ramsden-Wood serves as Chief Operating Officer of OneEnergy Partners. Prior to OneEnergy was VP of Reservoir Engineering and Business Development at

569 views • 29 slides
Paleozoic era (570-225 mya) -first vertebrates emerged -first mammal-like reptiles -Pangaea

Paleozoic era (570-225 mya) -first vertebrates emerged -first mammal-like reptiles -Pangaea

Paleozoic era (570-225 mya) -first vertebrates emerged -first mammal-like reptiles -Pangaea formed Mesozoic era (225-65 mya) -Age of dinosaurs Cenozoic era (65-0 mya) -Age of mammals 1 Continental drift: movement of continents on Earth's

341 views • 12 slides
Ch. 6 Primate homologies 1 Overview of the primates -Found in tropics/semi-tropics 2 TL;DR

Ch. 6 Primate homologies 1 Overview of the primates -Found in tropics/semi-tropics 2 TL;DR

Ch. 6 Primate homologies 1 Overview of the primates -Found in tropics/semi-tropics 2 TL;DR Primate shared derived traits -Primates are arboreal = adapted for living in the trees -Flexible limbs and prehensile hands/feet -Generalized dentition

480 views • 13 slides
INDUSTRIES GROUP, INC. A BOUT U S Unit Industries Group, Inc. has been dedicated to

INDUSTRIES GROUP, INC. A BOUT U S Unit Industries Group, Inc. has been dedicated to

UNIT INDUSTRIES GROUP, INC. A BOUT U S Unit Industries Group, Inc. has been dedicated to developing innovative manufacturing solutions with a core of professionals who have over a century of combined experience in integrated connectors,

301 views • 27 slides

More recommend