[PPT] - Bestoun S. Ahmed, Ph.D. Department of Computer Science and PowerPoint Presentation

SLIDE 1

Academia-To-Industry Transition Of Search And Learning- Based Software Engineering: Opportunities And Challenges

Bestoun S. Ahmed, Ph.D.

Department of Computer Science and Engineering Faculty of Electrical Engineering Czech Technical University in Prague Karlovo náměstí 13, 121 35 Praha 2

@bestoon82

albeybes@fel.cvut.cz

www.bestoun.net

SLIDE 2

The Two Cultures

1959- the clash of "the two

cultures”

The humanities and the

sciences.

A similar cleavage between the

academy and industry.

SLIDE 3

–But why not most of them were not used by industry?

“We are publishing many great solutions for nowadays’ problems”

SLIDE 4

Academia-Industry Collaboration

TRANSACTIONS OF THE AMERICAN CLINICAL AND CLIMATOLOGICAL ASSOCIATION, VOL. 113, 2002

ACAIDEMIC-INDUSTRIAL COLLABORATION: THE GOOD,

THE BAD, AND THE UGLY

JOSEPH B. MARTIN

BOSTON, MA

ABSTRACT

Academic-industrial collaborations and technology transfer have

ver the past 50 years played an increasingly prominent role in the

biomedical sciences. University partnerships with industry can expe-

dite the availability of innovative drugs and other medical technolo- gies, bringing both important public health benefits and a source of

income for universities and their faculty through a variety of financial

arrangements. However, these relationships raise ethical concerns,

particularly when research involves human subjects in clinical trials.

Lapses in oversight of industry-sponsored clinical trials at universi-

ties, and especially patient deaths in a number of trials, have brought

these issues into the public spotlight and have led the federal govern-

ment to intensify its oversight of clinical research. The leadership of Harvard Medical School convened a group of leaders in academic

medicine to formulate guidelines on individual financial conflicts of

interest. They and other groups are working to formulate a national

consensus on this issue.

BREACHES IN THE ACADEMIC-INDUSTRIAL WALL: A BRIEF HISTORY

When C.P. Snow in 1959 referred to the clash of "the two cultures,"

he was, of course, referring to the humanities and the sciences. We have over many decades observed a similar cleavage between the

academy and industry. These are two very different cultures with two

very different missions. On the one hand, the academic mission is education and discovery driven by intellectual curiosity-what we in

academia like to regard as "pure motives." In industry on the other

hand, the mission is translational research, commercialization, and

profit making (Figure 1).

Over the course of the 20th century a series of breaches arose in the

Joseph B. Martin, M.D., Ph.D., Harvard Medical School, Office ofthe Dean, 25 Shattuck St.,

Rm. 111, Boston, MA 02115; Phone: 617-432-1501, Fax: 617-432-3907; E-mail: joseph_mar-

tin@ hms.harvard.edu.

227

SLIDE 5

Academia-Industry: Two Different Missions

Academic mission:
Education and discovery driven

by intellectual curiosity-what we in academia like to regard as "pure motives.”

Industry mission:
Translational research,

commercialization, and profit making.

Breaches between the two missions?

SLIDE 6

Breach The Wall

Science, Technology, Engineering, Computer Science (19th century onwards)

Patents > Licensing > Royalties

Medical Devices and Biotechnology (1950 onwards)
Basic science support from industry (1980 onwards)

ACADEMIC-INDUSTRIAL COLLABORATION

Academia

Industry

Misi9n

Mission

Education, discovery

Translational research, driven by intellectual

commercialization,

curiosity: "pure motives" profit mailng

FIG. 1. The two cultures.

wall separating these two activities, rendering an increasingly porous

interface, admired by many and abhorred by some. The first breach in

the wall developed around technology, engineering, and computer sci-

ence, which led to a very deliberate process of patenting, licensing, and royalty income by major research universities engaged in the funda-

mental sciences (Table 1). During the last 50 years, with the National

TABLE 1

Breaches in the Wall

1. Science, Technology, Engineering, Computer Science (19th century onwards)

Patents

> Licensing > Royalties

2. Medical Devices and Biotechnology (1950-2000): same process

New ethical issue: agents or devices to be used in humans

3. 20% Rule: 1 day a week
4. Basic science support from industry: Monsanto, Hoechst,

Novartis, etc. (1980-2000)

5. Institutional equity (1995-2000)
6. Clinical research support
7. Clinical research organizations formed (1995-2000)
IRB workload increases
Informed consent guidelines challenged
8. Gene therapy: introduction of biologicals into humans (1990-2001)

228

SLIDE 7

Information and Software Technology 79 (2016) 106–127

Contents lists available at ScienceDirect

Information and Software Technology

journal homepage: www.elsevier.com/locate/infsof

Challenges and best practices in industry-academia collaborations in software engineering: A systematic literature review

Vahid Garousi

a , b , ∗, Kai Petersen c

, Baris Ozkan

d

a

Software Engineering Research Group, Department of Computer Engineering, Hacettepe University, Ankara, Turkey

b

Maral Software Engineering Consulting Corporation, Calgary, Canada

c

Department of Software Engineering, School of Engineering, Blekinge Institute of Technology, Sweden

d

Department of Information Systems Engineering, Atilim University, Ankara, Turkey

a r t i c l e i n f o

Article history: Received 31 December 2015 Revised 5 May 2016 Accepted 23 July 2016 Available online 30 July 2016 Keywords: Software engineering Industry-academia collaborations Industry Universities Challenges

a b s t r a c t

Context: The global software industry and the software engineering (SE) academia are two large commu-

nities. However, unfortunately, the level of joint industry-academia collaborations in SE is still relatively

very low, compared to the amount of activity in each of the two communities. It seems that the two ’camps’ show only limited interest/motivation to collaborate with one other. Many researchers and practitioners have written about the challenges, success patterns (what to do, i.e., how to collaborate) and anti-patterns (what not do do) for industry-academia collaborations. Objective: To identify (a) the challenges to avoid risks to the collaboration by being aware of the challenges, (b) the best practices to provide an inventory of practices (patterns) allowing for an informed choice of practices to use when planning and conducting collaborative projects.

*Study period 2003-2016

SLIDE 8

The ratio of authors from academia, industry, and joint authorships (Published Research Papers) SE topic areas of the projects studied Types of contributions Research types

SLIDE 9

–Some companies have NDA and it is hard to convince them to publish papers

“Keep in mind !”

SLIDE 10

The Relationship: Experience

Universities are changing the management vision to funding-oriented

management.

Many requests for collaboration from academia.
Less Response from industry.

Academia Industry

Many Collaboration Requests Less Collaboration Requests

SLIDE 11

Theory versus practice Industry versus academe

SLIDE 12

Challenges To Collaborate In Soft Eng.

There are many challenges mentioned by Garousi et. al*
Most relevant to us:
Results produced through research are not relevant for practice
Researchers do not understand the relevant problems from an

industry point of view

Different interests and objectives
Different reward systems
Lack of prior relationships between a company and academia
Lack of resources due to high investment in terms of resources
Licensing restrictions on tools

*V. Garousi, K. Petersen, and B. Ozkan, ‘‘Challenges and best practices in industry-academia collaborations in software engineering: A systematic literature review,’’

Inf. Softw. Technol., vol. 79, pp. 106–127, Nov. 2016

SLIDE 13

Barriers: Our Experience

Companies are interested in fast output.
Academia is infested in publication, which is not preferred

generally by industry (information disclosing avoidance).

Bureaucracy from the university side (especially the lawyers).
Bureaucracy from the industry side, especially to find the

contact person.

Size of the company is important.
Collaboration and finding goes inversely with the company size.

SLIDE 14

Another chance to collaborate ?

“Search and Learning-based Software Engineering”

SLIDE 15

One Of My Meetings With Industry

Look, I don’t think search or learning-based algorithms will contribute to our work.
But, let us do the meeting.
That is basically the end of the meeting from the beginning.

SLIDE 16

SBSE “The application of metaheuristic search-based

ptimization techniques to find near-optimal solutions

in software engineering problems”

SLIDE 17

The term SBSE was first used in 2001 by Harman and Jones.
However, optimization to a software engineering problem was reported

by Webb Miller and David Spooner in 1976 in the area of software testing.

SLIDE 18

1/18/2018

SBSE Repository http://crestweb.cs.ucl.ac.uk/resources/sbse_repository/ 2/2

The number of publications in the year from 1976 t0 2012

*Y. J. M. Harman and
Y. Zhang, “Achievements, open problems and challenges for search based software testing,” in Proc. 8th IEEE Int. Conf. Softw.

Testing, Verification Validation, Apr. 2015, pp. 1–12.

SLIDE 19

How It Works? Simply

There are few general steps that are common among all

the tools and algorithms:

1. Choose a search-based optimization technique.
2. Decide on an objective function (fitness function)

that will be a benchmark for the optimization process.

3. Search for the best candidate among a list of

candidates.

4. Based on the chosen optimization technique, update

the list of candidates.

5. Iterate until no better candidates found.

SLIDE 20

PSO As An Example

PSO = Particle Swarm Optimization
"Particles" fly in this hyperspace and try to find the global minima/maxima, their movement

being governed by a simple mathematical equation.

( )

1 1 2 , 3 , 1 1 t t i t t g t t t t t

v c v c p x c p x x x v

+ + +

ì = +

+
ï

í = + ï î

pi vt xt pg xt+1

particle’s itself particle’s personal best particle’s neighbours best

, , 1 2 3

: velocity at time step : position at time step : best previous position, at time step : best previous best, at time step , , , : co neighbour' gnitive/social s

t t i t g t

v t x t p t p t c c c = = = = = confidence coefficients

SLIDE 21

Your problem must be a non-deterministic problem!

“Remember”

Even for the same input, can exhibit different behaviors on different runs, as
pposed to a deterministic algorithm.
There should not be an exact algorithm to solve your problem.
Many algorithms using Search-based algorithms for deterministic problems

nowadays just to catch the attention and use some fancy representations.

But ………. They are wrong algorithms

SLIDE 22

1- The problem representation

“Two important components”

2- The Fitness Function

SLIDE 23

Fitness Function: The Key Ingredient

In Software Engineering: the fitness function converts some

requirement of the system into some measurable number.

Requirements Functional

Non-functional

Like the coverage

f some attribute
f the system
Security
Usability
Safety
Efficiency
Energy Consumption
Scalability
Availability

SLIDE 24

Some Functional Applications

SLIDE 25

Test Case Generation And Minimization

34 switches = 234 = 1.7 x 1010 possible inputs = 1.7 x 1010 tests?
Every possible combination of inputs? Which interactions cause faults?
How many test cases

we need to test?

SLIDE 26

The combinatorial interaction approach

SLIDE 27

Payment Server Smart Phone Web Server User Browser Business Database Master iPhone iPlanet Chrome SQL Visa Blackberry Apache Explorer Oracle Firefox Access

SLIDE 28

Test Case Minimization Approach

Red color tuples 25% of the t-tuples Green color tuples 50% of the t-tuples Blue color tuples 75% of the t-tuples Brown color tuples 100% of the t-tuples

SLIDE 29

How To Use The Fitness Function?

PSTG Strategy - How to use fitness function to choose best test case that cover most

f the interactions

SLIDE 30

Some applications of combinatorial interaction

SLIDE 31

GUI Testing

The use of event based modeling in GUI testing

SLIDE 32

Configuration Generation

Table 3 Summary of the input factors and levels for the case study program. No. Factors Levels 1 Degree [No Degree, Primary, Secondary, Diploma, Bachelor, Master, Doctor] 2 Children [Non, 1, 2, 3, 4, More_than_4] 3 Read [Checked, unchecked] 4 Write [Checked, unchecked] 5 Speak [Checked, unchecked] 6 Understand [Checked, unchecked] 7 New graduate [Checked, unchecked] 8 Experience [Checked, unchecked] 9 English [Checked, unchecked] 10 Disability [Checked, unchecked] 11 Marital status [Single, Married, Widow] 12 Resident [Local, Outsider, Foreigner]

SLIDE 33

Another Application

SLIDE 34

Control Engineering

Application of combinatorial test design

Application of Combinatorial Interaction PID

Controller Tuning.

SLIDE 35

Generating Configuration Tests For SPL

SLIDE 36

Avocado tool + CIT plugin

The goal is to add the CIT plugin to the other plugin set

Development team Avocado + CIT plugin Quality Engineering team

CIT

Test cases Problem specification

SLIDE 37

CIT

Context and Structure

"Varianter" object (all combinations) "Varianter" object (resulting combinations)

CIT is hooked in the Avocado Job, after

the action of the YAML_TO_MUX plugin, before the call to the Avocado Runner.

The hook is triggered by the command

line option "--combinatorial N".

CIT plugin receives the Varianter object,

serializes it, filters out the unneeded variants, creates a new Varianter object

ut of the remaining variants and returns

the object to the Avocado Job.

SLIDE 38

Avocado Multiplexer

SLIDE 39

Avocado Multiplexer + CIT : Benefits

Consider the example in the left. There are

three branches, machine(5 leaves), image(6 leaves) and architecture(29 leaves).

Basically,

with the current Avocado approach, there will be (5x6x29= 870) test cases.

Using our CIT algorithm, we can reduce the

number of test cases to 175 while keeping the test coverage on very similar level.

By testing the effectiveness of CIT, we found

that the reduced test set is as effective as the exhaustive test set (i.e., 870 test cases)

SLIDE 40

Input Parameter Analysis

*D. Richard Kuhn , Raghu N. Kacker , Yu Lei, Introduction to Combinatorial Testing, Chapman & Hall/CRC, 2013

SLIDE 41

Other Applications

Input testing: A systematic sampling of the input

parameters for the system-under-test instead of random selection.

Model-based testing: Generating test cases for

many models, like class diagrams, state charts.

Test suite prioritization: Arrange the test cases in

the most effective way.

SLIDE 42

Software Module Clustering

SLIDE 43

Preamble

In line with the customer demands for new functionalities,

software line of codes (LOCs) increases tremendously in the last 10 years.

From ->kilobytes -> Megabytes -> Gigabytes-> Terabytes

…………

SLIDE 44

Software Becomes More Complex

How to do maintenance?
Test engineers need to test more and more codes!!

SLIDE 45

Clustering

Clustering = the process of organizing objects into groups

whose members are similar in some way.

A cluster is therefore a collection of objects which are “similar” between

them and are “dissimilar” to the objects belonging to other clusters.

SLIDE 46

Clustering And Impact Analysis

SLIDE 47

Objectives

Maximizing cohesion
Describe functional strength
f a module i.e. similar

functions are grouped into a specific module.

Cohesion is a desirable design

component attribute as when a change has to be made, it is localized in a single component.

Minimizing coupling
Describe interdependencies among

modules

Loose coupling means component

changes are unlikely to affect other components

Shared variables or control information

exchange lead to tight coupling

Loose coupling can be achieved by

state decentralization (as in objects) and component communication via parameters or message passing

SLIDE 48

Problems At Hand

SLIDE 49

Module Dependency Graph (Mdg)

SLIDE 50

Modularization Quality And Modularization Factor

SLIDE 51

SLIDE 52

Non-Functional Properties

SLIDE 53

Energy Optimization

Automatically transforming the color scheme of a mobile web application by rewrites the

server side code and templates of a web application so that the resulting web application generates pages that are more energy efficient when displayed on a smartphone.

*Making Web Applications More Energy Efficient for OLED Smartphones , Ding Li, Angelica Huyen Tran, William G.J. Halfond, ICSE 2014,

HTML Output Graph (HOG) HTML Adjacency Relationship Graph (HARG) Color Conflict Graph (CCG) Color Transformation Scheme (CTS)

SLIDE 54

Learning-Based Software Engineering

The use of machine learning technique to learn from

some set of data or information retrieval system.

A Learning-Based Software Engineering Environment

Sidney C. Bailin, Robert H. Gattis, and Walt Truszkowski* Abstract

We describe an initial prototype of a software engineering environment that combines case-based reasoning (CBR) and explanation-based learning (EBL) functions. CBR and EBL are used to evolve the environment's understanding o f software principles as it is

used. The case base serves as a repository for reusable

solutions to software engineering problems. New solutions are synthesized from the case base through a process of adaptation, evaluation, and repair. When a new solution is returned, the user has the option of rnodioing it through a series o

f primitive edit operations. The

environment is capable of abstracting from these

perations using explanation-based generalization, and

synthesizing a new repair rule on the basis o f the

abstraction. We have successfully taught the environment

a non-trivial design repair rule by means of a single example, and have observed the environment apply this learned rule to the solution of a new input problem.

1 Introduction

A Knowledge-Based Software Engineering Environ- ment (KBSEE) is being developed as part of the NASA Initiative on Software Engineering. A key subsystem of the KBSEE is the LEARN environment (LEARN stands for Learning Enhanced Automation of Reuse eNgineering), so named because of its ability to learn new methods and techniques. The requirement for a learning capability follows from the extraordinary complexity of software engineering, as contrasted with other domains to which knowledge-based techniques have been applied, and from the fact that the discipline is only incompletely understood by its human practitioners. Moreover, the discipline is evolving: as application requirements become

more demanding and

varied, new approaches to software development are continually being devised. In such a field, it is unreasonable to expect to be able to encode sufficient knowledge directly into a computer system for it to be able solve the problems that humans tackle. Human

*Sidney C. Bailin and Robert H. Gattis are with CTA INCORPORATED, 61 16 Executive Boulevard, Suite 800, Rockville, MD 20852. Walt Truszkowski is with the Data Systems Technology Division of NASA/Goddard Space Flight Center, Greenbelt, MD 20771.

expertise

n engineering

software grows and develops over

time. A knowledge-based computer system capable of
ffloading non-trivial engineering tasks from the human

must also possess such an ability. If a KBSEE is ever to be able to move beyond "toy" problems, it must be able to learn. Our analysis of alternative machine learning approaches led us to propose a combination of two techniques: case-based reasoning (CBR) and explanation-

based learning (EBL). Case-based reasoning is, in

essence, an approach to reusing previously acquired

knowledge. New situtations are recognized as having

characteristics in common with situations previously

encountered. Whatever has been learned through the

previous encounter is then applied, with some adaptation

if necessary, to the new situation. In the KBSEE, these

situations (cases) are software engineering problems (for example, a set of requirements), and the reusable knowledge consists of solutions to such problems (e.g., a design meeting the requirements). Explanation-based learning is an approach to learning by abstracting key characteristics from an example provided by the user. In the KBSEE, the user is the software engineer, and the examples represent recommended ways of solving a problem. In every activity that the KBSEE performs, there is an opportunity to learn how to do it better. For example, in matching a new set of requirements to requirements that have been previously satisfied, some judgment of what constitutes a best match is required. This judgment is not easily formalized in a set of rules, but it can be learned piecemeal

ver time as the user corrects the actions taken by the
KBSEE. The same is true of adapting a previous solution

to new requirements, checking the new solution for required quality attributes, and refining the solution if the quality assessment fails. In all of these activities, the

KBSEE can learn from the user how to improve the

process. The KBSEE is thus playing the role of an

apprentice who learns by doing, under the supervision of an expert (the user). Eventually, the apprentice should acquire sufficient knowledge to perform non-trivial tasks independently. This paper describes the fist prototype of LEARN, and walks through an example of the KBSEE learning a new design rule. The prototype consists of a case-based reasoner, into which we have integrated an explanation-

198

0-8186-2605-4/91$1.00 Q 1991 IEEE

SLIDE 55

Two Main Applications

Classification learning
Learning from crowdsourcing
Adaptive feedback system learning

SUT Algo Learn Ver.

SLIDE 56

Applied

Soft Computing 62 (2018) 579–591

Contents

lists available at ScienceDirect

Applied

Soft Computing

j

urna

l h

mepage:

www.elsevier.com/locate/asoc

Learning

to classify software defects from crowds: A novel approach

Jerónimo

Hernández-González a,∗, Daniel Rodriguez c, I˜

naki

Inza a, Rachel Harrison d,

Jose

A. Lozano a,b

a Department

f Computer

Science and Artificial Intelligence, University

f

the Basque Country UPV/EHU, Donostia, Spain

b Basque Center

for Applied Mathematics BCAM, Bilbao, Spain

c Department

f

Computer Science, University

f

Alcala, Madrid, Spain

d Department

f

Computing, Oxford Brookes University, Oxford, UK

a

r t i c l e

i

n f

Article

history:

Received

1 April 2017

Received

in revised form 27 October 2017

Accepted

31 October 2017

Available

nline

7 November 2017

Keywords: Learning

from crowds

Orthogonal

defect classification

Missing

ground truth

Bayesian

network classifiers

a

b s t r a c t

In

software engineering, associating each reported defect with a category allows, among many

ther

things,

for the appropriate allocation

f resources.

Although this classification task can be automated

using

standard machine learning techniques, the categorization

f defects

for model training requires

expert

knowledge, which is not always available. To circumvent this dependency, we propose to apply

the

learning from crowds paradigm, where training categories are obtained from multiple non-expert

annotators

(and so may be incomplete, noisy

r

erroneous) and, dealing with this subjective class infor-

mation,

classifiers are efficiently learnt. To illustrate

ur proposal, we

present two real applications

f

the

IBM’s

rthogonal

defect classification working

n the

issue tracking systems from two different real

domains.

Bayesian network classifiers learnt using two state-of-the-art methodologies from data labeled

by

a crowd of annotators are used to predict the category (impact)

f reported

software defects. The

considered

methodologies show enhanced performance regarding the straightforward solution (major-

ity

voting) according to different

metrics. This

shows the possibilities

f using non-expert

knowledge

aggregation

techniques when expert knowledge is unavailable.

SLIDE 57

Table 3 Examples

f

defects and the corresponding labelings provided by the annotators.

Summary Description L1 L2 L3 L4 L5 Compendium

dataset

Error

Launching

Compendium

LD after

install Hi

team, Error message launching

Compendium

LD after initial

install:

Java Virtual Machine

Launcher

Could not find the main

class: com.compendium.ProjectCompendium. Program

will exit. I have run

through

the suggestion

n

the

forums

f

adding the path to javaw

in

the.bat, and verified the path

through

a command prompt is

successful,

same error. Any

ther

tips?

Regards, Eric

Installability Other Installability Installability Installability Spell

Checker

Add

a spelling checker to

Compendium

with the ability to

switch

n

and

ff

auto-spell

checking. Requirements Requirements Requirements Requirements Requirements Can

small icons also work

with

images? Make small

images? Right

now when you choose small

icons,

it shrinks the normal

Compendium

icons but not any

reference

node images, so they

stay

really big. Can we add an

ption

to shrink those

proportionally

as well?

Requirements Requirements Usability Requirements Other

SLIDE 58

SB And LB Combined

SLIDE 59

Learn from your testing output !

What is next ?

SUT

Search&Learn-based Testing Strategy System’s input Test output

SLIDE 60

On The Challenges Of Search Algorithm

Changing the optimization algorithm will not necessary

change the output to better. (My Experience)

It is the coverage criteria and the finest function that lead

to better results. (My Experience)

SLIDE 61

SBSE And LBSE: Challenges With The Industry

The non-deterministic output due to the randomness.
Expensive computation.
Modeling the system or the problem space.
Realize that you need one of them.