plagiarism detection system Andrzej Sobecki, Marcin Kpa IKC 2017 - - PowerPoint PPT Presentation

Improving performance of a plagiarism detection system

Andrzej Sobecki, Marcin Kępa IKC 2017

Plagiarism detection problem

• Text documents – unstructured form
• Finding a potential source documents based on the suspected

document

• Searching in many repositories
• Process must be short and accuracy

How we do that

Parsing Hashing Filtering Calculating similarities

The crucial stage for performance The important stage for accuracy

Filtering – actual solution

Doc Doc profile Hash function h(x) One hash – One sentence Suspected doc Suspected doc profile Repository Repositories Doc profile Doc profile Doc profile Available documents profiles Count identical hash values

Filtering – possible solutions

• Algorithms dedicated for digital libraries,
• Available search engines e.g., the elastic search,
• Components of the hadoop ecosystem,
• What is an effect of precision and recall values for performance and

accuracy of the plagiarism detection process?

Class of problem detecting similarities

• Unstructured text documents,
• Is required to analyzing most of the documents available in the

repositories,

• New documents are continuously add to the repositories,
• Effective filtering with high values of recall and precision,
• Finding similar sentences are more important than keywords.

Models described in the article

• KASKADA HashMap,
• HDFS,
• HDFS HashMap,
• Hbase,

Results – documents with fixed length

Results – documents with different lengths

Results – parallel tasks

Results - scalability

Results— cost of preparing structures

Summary

• Have you any questions?