Discovering Information Explaining API Types Using Text Course - - PowerPoint PPT Presentation

Discovering Information Explaining API Types Using Text Classification

Presented by: Sunyam Bagga Course Instructor:

• Dr. Jin Guo

TEXT CLASSIFICATION

Source: https://www.python-course.eu/text_classification_introduction.php

Relevant/Irrelevant [API type, Section fragment]

Technical Concepts

• 1. Recodoc tool
• 2. LOOCV
• 3. Maximum Entropy
• 4. Cosine similarity with tf-idf weighting
• 5. KAPPA

RecoDoc

“Recovering Traceability Links between an API and Its Learning Resources”

1

Aim:

• Find API types referenced in a tutorial:
• Identifies CLTs
• Links these CLTs to exact API type

“DateTime….such as year() or monthOfYear().”

• Precisely link code-like terms (e.g., year()) to

specific code elements (e.g., DateTime.year())

Ambiguity

▪ Declaration Ambiguity: CLTs are rarely fully qualified. ▪ Overload Ambiguity: CLTs do not indicate the number/type of parameters (method is

• verloaded).

▪ External Reference Ambiguity: May refer to code elements in external libraries. ▪ Language Ambiguity: Human errors: typographical (HtttpClient), case errors, forgetting parameters etc.

Parsing Artifacts and Recovering Traceability Links

• Linking Types: Given a CLT, they find all types in the

codebase whose name matches the term.

• Disambiguate and filter.

LOOCV

“Evaluating a classifier’s performance”

2

Leave-one-out Cross Validation

Source: https://towardsdatascience.com/train-test-split-and-cross-validation-in-python-80b61beca4b6

MaxEnt Classifier

“Using Maximum Entropy for Text Classification” by Nigam et al.

3

Maximum Entropy:

• Technique for estimating probability distributions

from data

• Principle: Without external knowledge, pick the

distribution that has the maximum entropy (most-uniform).

• Labeled training data helps put constraints on the

distribution

Example

Source: NLP by Dan Jurafsky and Chris Manning

Add Noun feature: f1 = {NN, NNS, NNP, NNPS} Add Proper Noun feature: f2 = {NNP, NNPS}

Source: NLP by Dan Jurafsky and Chris Manning

Constraints and Features

• Restricts the model distribution to have the same

expected value for a feature as seen in training data, D:

• Features for text classification:

Cosine Similarity with tf-idf

“Comparison with Information Retrieval”

4

Tf-Idf

• Technique to vectorise text data
• Term Frequency is a simple frequency count
• f a term in a document
• Inverse Document Frequency gives more

weight rare words.

Cosine Similarity

• Measures the cosine of the angle between

the vectors:

• They consider a section relevant if the

similarity value is higher than a certain threshold.

KAPPA score

“Annotating the Experimental Corpus”

5

Kappa formula

• Measures inter-annotator agreement.

▪ Po: observed agreement among annotators ▪ Pe: hypothetical probability of chance agreement ▪ More robust than simple percent agreement calculation

Kappa Example:

▪ Po = (20+15) / 50 = 0.7 ▪ P(Yes) = 0.5*0.6 = 0.3 ▪ P(No) = 0.5*0.4 = 0.2 ▪ Pe = P(Yes) + P(No) = 0.5 Kappa = (0.7 - 0.5) / (1 - 0.5) = 0.4

Source: https://en.wikipedia.org/wiki/Cohen%27s_kappa

Thanks!

Any questions?