In Engineering Classes, How to Assign Possible Scenarios Partial - - PowerPoint PPT Presentation

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 1 of 23 Go Back Full Screen Close Quit

In Engineering Classes, How to Assign Partial Credit: From Current Subjective Practice to Exact Formulas (Based on Computational Intelligence Ideas)

Joe Lorkowski1, Vladik Kreinovich1 and Olga Kosheleva2

1Department of Computer Science 2Department of Teacher Education

University of Texas at El Paso El Paso, TX 79968, USA lorkowski@computer.org, vladik@utep.edu,

• lgak@utep.edu

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 2 of 23 Go Back Full Screen Close Quit

1. Need to Assign Partial Credit

• If on a test, a problem is solved correctly, then the

student gets full credit.

• If the student did not solve this problem at all, the

student gets no credit for this problem.

• In many cases:

– the student correctly performed some steps that leads to the solution, – but, due to missing steps, still did not get the so- lution.

• The student is usually given partial credit for this prob-

lem.

• The more steps the student performed, the more par-

tial credit this student gets.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 3 of 23 Go Back Full Screen Close Quit

2. How Partial Credit is Usually Assigned

• Usually, partial credit is assigned in a very straightfor-

ward way: – if the solution to a problem j requires nj steps, – and kj < nj of these steps have been correctly per- formed, – then we assign the fraction kj nj

• f the full grade.
• For example:

– if a 10-step problem is worth 100 points – and a student performed 9 steps correctly, – then this student gets 90 points for this problem.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 4 of 23 Go Back Full Screen Close Quit

3. For Engineering Education, This Usual Prac- tice is Sometimes a Problem

• Sometimes, our objective is simply to check intellectual

progress of a student.

• In this case, the usual practice of assigning partial

credit makes perfect sense.

• However, in engineering education, we want to check

how well a student can handle real engineering tasks.

• Let us consider the case when:

– in each of the 10 problems on the test, – the student performed 9 stages out of 10.

• Then, none of the answers is correct, but the student

gets 10 · 9 = 90 points: “excellent”.

• This student gets as many points as another students

who correctly solved 9 problems.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 5 of 23 Go Back Full Screen Close Quit

4. Analysis of the Problem

• Let us imagine that this student has graduated and is

working for an engineering company.

• To gauge the student’s skills, let’s estimate the benefit

that he/she brings to this company.

• Let us start with considering a single problem j whose

solution consists of nj stages.

• Let us assume that the newly hired student can cor-

rectly perform kj out of these nj stages.

• This is not a test, this is a real engineering problem, it

needs to be solved.

• So someone else must help to solve the remaining nj−kj

stages.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 6 of 23 Go Back Full Screen Close Quit

5. Possible Scenarios

• Sometimes, in the company, there are other specialists

who can perform the remaining nj − kj stages.

• In this case, the internal cost of this additional (un-

planned) help is proportional to the number of stages.

• Otherwise, we need to hire outside help.
• In such a situation, the main part of the cost is usually

the hiring itself: – the cost of bringing in the consultant is usually much higher than – the specific cost of the consultant performing the corresponding tasks.

• Thus, in the first approximation, the cost of using a

consultant does not depend on the number of stages.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 7 of 23 Go Back Full Screen Close Quit

6. Estimating Expected Loss to a Company: Case

• f a Single Problem
• Let ci (i for “in-house”) denote the cost of performing
• ne stage in-house.
• The resulting in-house cost is equal to ci · (nj − kj).
• Let ch denote the cost of hiring a consultant.
• Let p denote the probability that there is an in-house

specialist who can perform a given stage.

• It is reasonable to assume that different stages are in-

dependent.

• So the probability that we can find inside help for all

nj − kj stages is equal to pnj−kj.

• Thus, the expected loss is equal to

ci · pnj−kj · (nj − kj) + ch ·

• 1 − pnj−kj

.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 8 of 23 Go Back Full Screen Close Quit

7. Estimating Expected Loss to a Company: Case

• f Several Problems
• Several problems on a test usually simulate different

engineering situations that may occur in real life.

• We can use the above formula to estimate the loss

caused by each of the problems: ci · pnj−kj · (nj − kj) + ch ·

• 1 − pnj−kj

.

• The overall expected loss L can be then computed as

the sum of the costs corresponding to all J problems: L =

J

• j=1
• ci · pnj−kj · (nj − kj) + ch ·
• 1 − pnj−kj

.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 9 of 23 Go Back Full Screen Close Quit

8. So How Should We Assign Partial Credit

• Usually, the grade g is counted in such as way that:

– complete knowledge corresponds to 100 points, and – a complete lack of knowledge – to 0 points.

• Let L be the loss caused by the student’s lack of skills.
• Then, we should take g = 100 − c · L for some c.
• The complete lack of knowledge is kj = 0 for all j:

L =

J

• j=1

(ci · pnj · n + ch · (1 − pnj)).

• We want to select c so that 100−c·L = 100 = 0, thus:

g = 100−100·

J

• j=1
• ci · pnj−kj · (nj − kj) + ch ·
• 1 − pnj−kj

J

• j=1

(ci · pnj · nj + ch · (1 − pnj)) .

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 10 of 23 Go Back Full Screen Close Quit

9. How to Assign Partial Credit: the Resulting Formula

• If we divide numerator and denominator by ch, we get

a simpler formula in terms of c′

i def

= ci ch .

• A student who, for each j-th problems out of J, per-

formed kj out of nj stages, get the grade g = 100 − 100 ·

J

• j=1
• c′

i · pnj−kj · (nj − kj) +

• 1 − pnj−kj

J

• j=1

(ci · pnj · nj + ch · (1 − pnj)) .

• Here c′

i is the ratio of an in-house cost of performing a

stage to the cost of hiring an outside consultant.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 11 of 23 Go Back Full Screen Close Quit

10. Let Us Analyze the Resulting Formula for As- signing Partial Credit

• The above formula uses the probability p that it is

possible to perform a stage in-house.

• The value p depends on the company size:

– In a very big company, with many engineers of dif- ferent type, this probability is close to 1. – On the other hand, in a small company this prob- ability is very small.

• For a very big company, p ≈ 1, so

g = 100 − 100 ·

J

• j=1

c′

i · (nj − kj) J

• j=1

c′

i · nj

= 100 ·

J

• j=1

kj

J

• j=1

nj .

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 12 of 23 Go Back Full Screen Close Quit

11. Resulting Formula (cont-d)

• This is the usual formula for assigning partial credit:

the same number of points for each step.

• Thus, this usual formula corresponds to the case when

a student is hired by a very big company.

• For a very small company, p ≈ 0, so the student gets

no partial credit at all.

• In this case, if the answer is not correct, there are 0

points assigned to this student.

• This situation is similar to how grades are estimated
• n the final exams in medicine: there,

– an “almost correct” (but wrong) answer can kill the patient, so – such “almost correct” answers are not valued at all.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 13 of 23 Go Back Full Screen Close Quit

12. Intermediate Case: General Description

• In the intermediate cases, we do assign some partial

credit.

• However, this credit is much smaller than in the tradi-

tional assignment.

• Usually, ci ≪ ch, so c′

i ≪ 1 and

g = 100 ·

J

• j=1
• pnj−kj − pnj

J

• j=1

(1 − pnj) .

• Usually, the number of stages nj is reasonably large,

so pnj ≈ 0, and g = 100 · 1 J ·

J

• j=1

gj, where gj ≈ pnj−kj.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 14 of 23 Go Back Full Screen Close Quit

13. So What Do We Propose

• Our analysis shows that, depending on the size of com-

pany, we should assign partial credit differently.

• So maybe this is a way to go:

– instead of trying to describe the student’s knowl- edge by a single number, – we use different numbers corresponding to several different values of the parameter p.

• For example, we can select values p = 0, p = 0.1, . . . ,

p = 0.9, p = 1 (or, as a start, just p = 0 and p = 1).

• We recommend that all these grades should be listed

in the transcript.

• Then, each company can look into the grade that is

the best fit for their perceived value p.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 15 of 23 Go Back Full Screen Close Quit

14. The Resulting Formulas Are in Good Accor- dance with Computational Intelligence

• Our formula for partial credit comes from a simplified

– but still rather complicated – mathematical model.

• Since the model is simplified, we are not 100% sure

that this is the right formula.

• We thus need to supplement the mathematical deriva-

tion with a more intuitive explanation.

• The grade for a problem is the instructor’s degree of

confidence that the student knows the material.

• The only information that we can use to compute this

degree is that: – the student successfully performed kj stages, and – the student did not perform nj − kj stages.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 16 of 23 Go Back Full Screen Close Quit

15. Computational Intelligence (cont-d)

• For each stage, our degree of confidence is D if the

students did it, and d < D if he/she didn’t.

• In general:

– each of the kj successful stages adds a confidence D, and – each of nj−kj unsuccessful stages adds a confidence degree d.

• “And”-combinations of degrees of confidence is well-

studied in fuzzy logic.

• The two simplest (and most widely used) “and”-
• perations are min and product.
• If we use min, and the student did not succeed in at

least one stage, then the grade is min(D, . . . , D, d, . . . , d) = d.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 17 of 23 Go Back Full Screen Close Quit

16. min and Product “And”-Operators (cont-d)

• The two simplest (and most widely used) “and”-
• perations are min and product.
• If we use min, and the student did not succeed in at

least one stage, then the grade is min(D, . . . , D, d, . . . , d) = d.

• So, for min, there’s no partial credit: full credit or 0.
• If we use the product “and”-operation, then

µ = D·. . .·D (kj times)·d·. . .·d (nj−kj times) = Dkj·dnj−kj.

• Thus, we get, in effect, the above formula for partial

credit.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 18 of 23 Go Back Full Screen Close Quit

17. Commonsense Interpretation of Our Partial Credit Formula

• The above justification is based on the general descrip-

tion of “and”-operations.

• These general descriptions have nothing to do with the

specific problem of assigning partial credit.

• It is therefore desirable:

– to supplement that general justification – with a commonsense interpretation directly related to the problem of assigning partial credit.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 19 of 23 Go Back Full Screen Close Quit

18. Main Idea Behind Our Commonsense Justifi- cation

• Knowledge about each subject can be viewed as an

ever-growing tree.

• We start with some very basic facts.
• We can say that these facts form the basic (first) level
• f the knowledge tree.
• Then, based on these very basic facts, we develop some

concepts of the second level.

• For example, in Introduction to Computer Science,

– once the students understood the main idea of an algorithm, – they start describing different types of variables: integers, real numbers, characters, strings, etc.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 20 of 23 Go Back Full Screen Close Quit

19. Main Idea (cont-d)

• Based on the knowledge of the second level, we then

further branch out into concepts of the third level, etc.

• In the first approximation, we can assume that the

branching b is the same on each level; so, we have: – 1 cluster of concepts on Level 1, – b clusters of concepts on Level 2, – b2 clusters of concepts on Level 3, . . . – bn−1 clusters of concepts on the highest level n.

• The overall number of concepts of all levels from 1 to

n is 1 + b + . . . + bn−1 = bn − 1 b − 1 .

• For each problem, usually, each of n stages corresponds

to the corresponding level.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 21 of 23 Go Back Full Screen Close Quit

20. Main Idea (cont-d)

• From this viewpoint:

– when a student was able to only successfully per- form k stages, – this means that the student has mastered only the concepts of the first k levels.

• So, of bn − 1

b − 1 concepts, the student has mastered only 1 + b + . . . + bk−1 = bk − 1 b − 1 .

• The proportion of mastered concepts is (bk−1)/(bn−1).
• Usually, b ≫ 1, bn ≫ 1, bk ≫ 1, so this proportion is

approximately bk/bn = pn−k, where p

def

= 1 b.

• So, we get the exact same formula for partial credit.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 22 of 23 Go Back Full Screen Close Quit

21. Conclusion

• When a student correctly performs only some of the

steps, this student gets partial credit.

• Partial credit is usually proportional to the number of

steps that the student performed correctly.

• As a result, we encounter a paradoxical situation:

– if in each of 10 test problems, the student correctly performed 9 out of 10 steps, – then this students gets 90 points out of 100, and an “excellent” (A) grade, – but the student’s ten answers are all wrong!

• If this student graduates and starts working for an en-

gineering company, his performance will be a disaster.

• We use computational intelligence ideas to come up

with a better way of assigning partial credit.

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem Possible Scenarios Estimating Expected . . . Estimating Expected . . . So How Should We . . . Commonsense . . . Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 23 of 23 Go Back Full Screen Close Quit

22. Acknowledgment

• This work was supported in part by the National Sci-

ence Foundation grants – HRD-0734825 and HRD-1242122 (Cyber-ShARE Center of Excellence) and – DUE-0926721.

• The authors are thankful to Dr. Nigel Ward for valu-

able discussions.