SLIDE 1
Measuring the Quality of Credit Scoring Models
Martin Řezáč
- Dept. of Mathematics and Statistics, Faculty of Science,
2/30
Measuring the Quality of Credit Scoring Models Martin ez Dept. of - - PowerPoint PPT Presentation
Measuring the Quality of Credit Scoring Models Martin ez Dept. of - - PowerPoint PPT Presentation
Measuring the Quality of Credit Scoring Models Martin ez Dept. of Mathematics and Statistics, Faculty of Science, Masaryk University CSCC XI, Edinburgh August 2009 Content 1. Introduction 3 2. Good/bad client definition 4 3.
SLIDE 2
SLIDE 3
3/30
Introduction
It is impossible to use scoring model effectively without knowing how good it is. Usually one has several scoring models and needs to select just one. The best one. Before measuring the quality of models one should know (among other things):
good/bad definition expected reject rate
SLIDE 4
4/30
Good/bad client definition
Good Bad Indeterminate Insufficient Excluded Rejected.
Good definition is the basic condition of effective scoring model. The definition usually depends on: Generally we consider following types of client:
days past due (DPD) amount past due time horizon
SLIDE 5
5/30
Customer Default
(60 or 90 DPD)
Not default Fraud
(first delayed payment, 90 DPD)
Early default
(2-4 delayed payment, 60 DPD)
Late default
(5+ delayed payment, 60 DPD)
Good/bad client definition
Rejected Accepted Insufficient
GOOD BAD INDETERMINATE
SLIDE 6
6/30
Measuring the quality
Once the definition of good / bad client and client's score is available, it is possible to evaluate the quality of this score. If the score is an output of a predictive model (scoring function), then we evaluate the quality of this
- model. We can consider two basic types of quality indexes. First, indexes
based on cumulative distribution function like The second, indexes based on likelihood density function like
Kolmogorov-Smirnov statistics (KS) Gini index C-statistics Lift. Mean difference (Mahalanobis distance) Informational statistics/value (IVal).
SLIDE 7
7/30
Indexes based on distribution function
= . , , 1
- therwise
good is client DK
( )
1 1 ) (
1 .
= ∧ ≤ = ∑
= K i n i GOOD n
D a s I n a F
( )
1 ) (
1 .
= ∧ ≤ =
∑
= K i m i BAD m
D a s I m a F
[ ]
H L a , ∈
( )
a s I N a F
i N i ALL N
≤ =
∑
=1 .
1 ) ( Empirical distribution functions:
( )
=
- therwise
true is A A I 1 m n m pB + = , m n n pG + = Number of good clients: Number of bad clients: Proportions of good/bad clients:
n m
Kolmogorov-Smirnov statistics (KS)
[ ]
) ( ) ( max
, , ,
a F a F KS
GOOD n BAD m H L a
− =
∈
SLIDE 8
8/30
Lorenz curve (LC) Gini index
A B A A Gini 2 = + =
[ ].
, ), ( ) (
. .
H L a a F y a F x
GOOD n BAD m
∈ = =
( ) (
)
∑
+ = − −
+ ⋅ − − =
m n k k GOOD n GOOD n k BAD m k BAD m
F F F F Gini
k
2 1 . . 1 . .
1
k BAD m
F .
k
GOOD n
F .
where ( ) is k-th vector value of empirical distribution function of bad (good) clients
Indexes based on distribution function
SLIDE 9
9/30
2 1 Gini C A stat c + = + = −
( )
1
2 1
2 1
= ∧ = ≥ = −
K K
D D s s P stat c
It represents the likelihood that randomly selected good client has higher score than randomly selected bad client, i.e.
c
Indexes based on distribution function
C-statistics:
SLIDE 10
10/30
Another possible indicator of the quality of scoring model can be cumulative Lift, which says, how many times, at a given level of rejection, is the scoring model better than random selection (random model). More precisely, the ratio indicates the proportion of bad clients with less than a score a, , to the proportion of bad clients in the general population. Formally, it can be expressed by:
[ ]
H L a , ∈
( ) ( ) ( ) ( ) ( ) ( )
N n a s I Y a s I Y Y I Y I a s I Y a s I BadRate a CumBadRate a Lift
i m n i i m n i m n i m n i i m n i i m n i
≤ = ∧ ≤ = = ∨ = = ≤ = ∧ ≤ = =
∑ ∑ ∑ ∑ ∑ ∑
+ = + = + = + = + = + = 1 1 1 1 1 1
1 ) ( ) (
Indexes based on distribution function
BadRate a BadRate a absLift ) ( ) ( =
SLIDE 11
11/30
# bad clients Bad rate
- abs. Lift
# bad clients Bad rate
- cum. Lift
1 100 16 16,0% 3,20 16 16,0% 3,20 2 100 12 12,0% 2,40 28 14,0% 2,80 3 100 8 8,0% 1,60 36 12,0% 2,40 4 100 5 5,0% 1,00 41 10,3% 2,05 5 100 3 3,0% 0,60 44 8,8% 1,76 6 100 2 2,0% 0,40 46 7,7% 1,53 7 100 1 1,0% 0,20 47 6,7% 1,34 8 100 1 1,0% 0,20 48 6,0% 1,20 9 100 1 1,0% 0,20 49 5,4% 1,09 10 100 1 1,0% 0,20 50 5,0% 1,00 All 1000 50 5,0%
absolutely cumulatively decile # cleints
Indexes based on distribution function
- 0,50
1,00 1,50 2,00 2,50 3,00 3,50 1 2 3 4 5 6 7 8 9 10 decile Lift value
- abs. Lift
- cum. Lift
Gini=0,55
0,2 0,4 0,6 0,8 1 0,2 0,4 0,6 0,8 1 Lornz curve Base line
Usually it is computed using table with numbers of all and bad clients in some bands (deciles).
SLIDE 12
12/30
# bad clients Bad rate
- abs. Lift
# bad clients Bad rate
- cum. Lift
1 100 8 8,0% 1,60 8 8,0% 1,60 2 100 12 12,0% 2,40 20 10,0% 2,00 3 100 16 16,0% 3,20 36 12,0% 2,40 4 100 5 5,0% 1,00 41 10,3% 2,05 5 100 3 3,0% 0,60 44 8,8% 1,76 6 100 2 2,0% 0,40 46 7,7% 1,53 7 100 1 1,0% 0,20 47 6,7% 1,34 8 100 1 1,0% 0,20 48 6,0% 1,20 9 100 1 1,0% 0,20 49 5,4% 1,09 10 100 1 1,0% 0,20 50 5,0% 1,00 All 1000 50 5,0%
decile # cleints absolutely cumulatively
Indexes based on distribution function
- 0,50
1,00 1,50 2,00 2,50 3,00 3,50 1 2 3 4 5 6 7 8 9 10 decile Lift value
- abs. Lift
- cum. Lift
Gini=0,48
0,2 0,4 0,6 0,8 1 0,2 0,4 0,6 0,8 1 Lornz curve Base line
When bad rates are not monotone:
LC looks fine Gini is slightly lowered Lift looks strange
SLIDE 13
13/30
# bad clients Bad rate
- abs. Lift
# bad clients Bad rate
- cum. Lift
1 100 16 16,0% 3,20 16 16,0% 3,20 2 100 12 12,0% 2,40 28 14,0% 2,80 3 100 8 8,0% 1,60 36 12,0% 2,40 4 100 5 5,0% 1,00 41 10,3% 2,05 5 100 3 3,0% 0,60 44 8,8% 1,76 6 100 2 2,0% 0,40 46 7,7% 1,53 7 100 1 1,0% 0,20 47 6,7% 1,34 8 100 1 1,0% 0,20 48 6,0% 1,20 9 100 1 1,0% 0,20 49 5,4% 1,09 10 100 1 1,0% 0,20 50 5,0% 1,00 All 1000 50 5,0%
absolutely cumulatively decile # cleints
Indexes based on distribution function
# bad clients Bad rate
- abs. Lift
# bad clients Bad rate
- cum. Lift
1 100 1 1,0% 0,20 1 1,0% 0,20 2 100 1 1,0% 0,20 2 1,0% 0,20 3 100 1 1,0% 0,20 3 1,0% 0,20 4 100 1 1,0% 0,20 4 1,0% 0,20 5 100 2 2,0% 0,40 6 1,2% 0,24 6 100 3 3,0% 0,60 9 1,5% 0,30 7 100 5 5,0% 1,00 14 2,0% 0,40 8 100 8 8,0% 1,60 22 2,8% 0,55 9 100 12 12,0% 2,40 34 3,8% 0,76 10 100 16 16,0% 3,20 50 5,0% 1,00 All 1000 50 5,0%
absolutely cumulatively decile # cleints
Gini= - 0,55
0,2 0,4 0,6 0,8 1 0,2 0,4 0,6 0,8 1 Lornz curve Base line
- 0,50
1,00 1,50 2,00 2,50 3,00 3,50 1 2 3 4 5 6 7 8 9 10 decile Lift value
- abs. Lift
- cum. Lift
When score is reversed, we obtain reversed figures.
SLIDE 14
14/30
Gini = 0.42
0,2 0,4 0,6 0,8 1 0,2 0,4 0,6 0,8 1 Lornz curve Base line
Indexes based on distribution function
# bad clients Bad rate
1 100 35 35,0% 2 100 16 16,0% 3 100 8 8,0% 4 100 8 8,0% 5 100 7 7,0% 6 100 6 6,0% 7 100 6 6,0% 8 100 5 5,0% 9 100 5 5,0% 10 100 4 4,0% All 1000 100 10,0%
decile # cleints # bad clients Bad rate
1 100 20 20,0% 2 100 18 18,0% 3 100 17 17,0% 4 100 15 15,0% 5 100 12 12,0% 6 100 6 6,0% 7 100 4 4,0% 8 100 3 3,0% 9 100 3 3,0% 10 100 2 2,0% All 1000 100 10,0%
decile # cleints
SC 1:
K-S = 0.36
0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1
good bad
K-S = 0.34
0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 good bad
Gini= 0,42
0,2 0,4 0,6 0,8 1 0,2 0,4 0,6 0,8 1 Lornz curve Base line
SC 2:
The Gini is not enough!!!
SLIDE 15
15/30
Indexes based on distribution function
- 0,50
1,00 1,50 2,00 2,50 3,00 3,50 4,00 1 2 3 4 5 6 7 8 9 10 decile Lift value
- abs. Lift
- cum. Lift
- 0,50
1,00 1,50 2,00 2,50 1 2 3 4 5 6 7 8 9 10 decile Lift value
- abs. Lift
- cum. Lift
SC 1: SC 2:
Lift20% = 2.55 > Lift50% = 1.48 < Lift20% = 1.90 Lift50% = 1.64
SC 2 is better if reject rate is expected around 50%. SC 1 is much more better if reject rate is expected by 20%.
SLIDE 16
16/30
) ( ) ( ) (
. .
a F a F a Lift
ALL N BAD n
=
[ ]
H L a , ∈
( )
) ( 1 )) ( ( )) ( (
1 . . 1 . . 1 . .
q F F q q F F q F F Lift
ALL N BAD n ALL N ALL N ALL N BAD n q − − −
= =
{ }
q a F H L a q F
ALL N ALL N
≥ ∈ =
−
) ( ], , [ min ) (
. 1 .
( ).
) 1 . ( 10
1 . . % 10 −
⋅ =
ALL N BAD n
F F Lift
Indexes based on distribution function
Lift can be expressed and computed by formulae:
SLIDE 17
17/30
Indexes based on density function
) (x f GOOD ) (x f BAD
( )
∑
= =
− =
n D i i h GOOD
K
s x K n h x f
1 , 1
1 ) , ( ~
( )
∑
= =
− =
m D i i h BAD
k
s x K m h x f
1
1 ) , ( ~
1 2 1 1 2 1 2 3 ,
~ )! 3 2 ( ) 5 2 ( )! 1 2 (
+ + +
⋅ ⋅ + + + =
k k k k OS
n k k k k h σ Likelihood density functions: Kernel estimates: Optimal bandwidth (maximal smoothing): where: is the order of kernel function (e.g. 2 for Epanechnikov kernel) is number of actual cases is an estimate of standard deviation k n
σ ~
SLIDE 18
18/30
Mean difference (Mahalanobis distance):
2 1 2 2
+ + = m n mS nS S
b g
S M M D
b g −
=
Indexes based on density function
where S is pooled standard deviation:
g
M
b
M
g
S
b
S
, are standard deviations of good (bad) clients , are means of good (bad) clients
SLIDE 19
19/30
Information value (Ival) – continuous case (Divergence):
( )
dx x f x f x f x f I
BAD GOOD BAD GOOD val
− = ∫
∞ ∞ −
) ( ) ( ln ) ( ) ( ) ( ) ( ) ( x f x f x f
BAD GOOD diff
− =
= ) ( ) ( ln ) ( x f x f x f
BAD GOOD LR
Indexes based on density function
SLIDE 20
20/30
- Replace density functions by their kernel estimates and compute integral
numerically (e.g. by composite trapezoidal rule).
- Using Epanechnikov kernel, given by
and optimal bandwidth we have
- For given M+1 points
we obtain
Information value (Ival) – discretized continuous case:
( )
[ ] ( )
1 , 1 1 4 3 ) (
2
− ∈ ⋅ − = x I x x K
+ + − =
∑
− = 1 1
) ( ~ ) ( ~ 2 ) ( ~ 2
M i M IV i IV IV M val
x f x f x f M x x I ( )
− = ) , ( ~ ) , ( ~ ln ) , ( ~ ) , ( ~ ) ( ~
2 , 2 , 2 , 2 , OS BAD OS GOOD OS BAD OS GOOD IV
h x f h x f h x f h x f x f
Indexes based on density function
k OS
h
, M
x x , ,
0 K
x
M
x
SLIDE 21
21/30
- Create intervals of score – typically deciles. Number of goods (bads) in i-th
interval is marked by .
- It must holds
- Then we have
∑
− =
i i i i i val
n b m g m b n g I ln
Indexes based on density function
score int. # bad clients #good clients % bad [1] % good [2] [3] = [2] - [1] [4] = [2] / [1] [5] = ln[4] [6] = [3] * [5]
1 1 10 2,0% 1,1%
- 0,01
0,53
- 0,64
0,01 2 2 15 4,0% 1,6%
- 0,02
0,39
- 0,93
0,02 3 8 52 16,0% 5,5%
- 0,11
0,34
- 1,07
0,11 4 14 93 28,0% 9,8%
- 0,18
0,35
- 1,05
0,19 5 10 146 20,0% 15,4%
- 0,05
0,77
- 0,26
0,01 6 6 247 12,0% 26,0% 0,14 2,17 0,77 0,11 7 4 137 8,0% 14,4% 0,06 1,80 0,59 0,04 8 3 105 6,0% 11,1% 0,05 1,84 0,61 0,03 9 1 97 2,0% 10,2% 0,08 5,11 1,63 0,13 10 1 48 2,0% 5,1% 0,03 2,53 0,93 0,03 All 50 950
- Info. Value
0,68
Information statistics/value (Ival) – discrete case:
( )
i i b
g
i b g
i i
∀ > > ,
SLIDE 22
22/30
Information value for our example of two scorecards:
Indexes based on density function
SC 1: SC 2:
decile # cleints # bad clients #good % bad [1] % good [2] [3] = [2] - [1] [4] = [2] / [1] [5] = ln[4] [6] = [3] * [5] cum. [6]
1 100 35 65 35,0% 7,2%
- 0,28
0,21
- 1,58
0,44 0,44 2 100 16 84 16,0% 9,3%
- 0,07
0,58
- 0,54
0,04 0,47 3 100 8 92 8,0% 10,2% 0,02 1,28 0,25 0,01 0,48 4 100 8 92 8,0% 10,2% 0,02 1,28 0,25 0,01 0,49 5 100 7 93 7,0% 10,3% 0,03 1,48 0,39 0,01 0,50 6 100 6 94 6,0% 10,4% 0,04 1,74 0,55 0,02 0,52 7 100 6 94 6,0% 10,4% 0,04 1,74 0,55 0,02 0,55 8 100 5 95 5,0% 10,6% 0,06 2,11 0,75 0,04 0,59 9 100 5 95 5,0% 10,6% 0,06 2,11 0,75 0,04 0,63 10 100 4 96 4,0% 10,7% 0,07 2,67 0,98 0,07 0,70 All 1000 100 900
- Info. Value
0,70
decile # cleints # bad clients #good % bad [1] % good [2] [3] = [2] - [1] [4] = [2] / [1] [5] = ln[4] [6] = [3] * [5] cum. [6]
1 100 20 80 20,0% 8,9%
- 0,11
0,44
- 0,81
0,09 0,09 2 100 18 82 18,0% 9,1%
- 0,09
0,51
- 0,68
0,06 0,15 3 100 17 83 17,0% 9,2%
- 0,08
0,54
- 0,61
0,05 0,20 4 100 15 85 15,0% 9,4%
- 0,06
0,63
- 0,46
0,03 0,22 5 100 12 88 12,0% 9,8%
- 0,02
0,81
- 0,20
0,00 0,23 6 100 6 94 6,0% 10,4% 0,04 1,74 0,55 0,02 0,25 7 100 4 96 4,0% 10,7% 0,07 2,67 0,98 0,07 0,32 8 100 3 97 3,0% 10,8% 0,08 3,59 1,28 0,10 0,42 9 100 3 97 3,0% 10,8% 0,08 3,59 1,28 0,10 0,52 10 100 2 98 2,0% 10,9% 0,09 5,44 1,69 0,15 0,67 All 1000 100 900
- Info. Value
0,67
SLIDE 23
23/30
Using markings we have:
− = m b n g I
i i diffi
Indexes based on density function
= n b m g I
i i LRi
ln
SC 1: SC 2:
- 0,30
- 0,25
- 0,20
- 0,15
- 0,10
- 0,05
0,00 0,05 0,10
1 2 3 4 5 6 7 8 9 10
- 2,00
- 1,50
- 1,00
- 0,50
0,00 0,50 1,00 1,50 I_diff I_LR 0,00 0,05 0,10 0,15 0,20 0,25 0,30 0,35 0,40 0,45 0,50
1 2 3 4 5 6 7 8 9 10
0,00 0,10 0,20 0,30 0,40 0,50 0,60 0,70 0,80 I_diif * I_LR
- cum. I_diff * I_LR
- 0,15
- 0,10
- 0,05
0,00 0,05 0,10
1 2 3 4 5 6 7 8 9 10
- 1,00
- 0,50
0,00 0,50 1,00 1,50 2,00 I_diff I_LR 0,00 0,02 0,04 0,06 0,08 0,10 0,12 0,14 0,16
1 2 3 4 5 6 7 8 9 10
0,00 0,10 0,20 0,30 0,40 0,50 0,60 0,70 0,80 I_diif * I_LR
- cum. I_diff * I_LR
K-S = 0.34 Gini = 0.42 Lift20% = 2.55 Lift50% = 1.48 Ival = 0.70 Ival20% = 0.47 Ival50% = 0.50 K-S = 0.36 Gini = 0.42 Lift20% = 1.90 Lift50% = 1.64 Ival = 0.67 Ival20% = 0.15 Ival50% = 0.23
SLIDE 24
24/30
Assume that the scores of good and bad clients are normally distributed, i.e. we can write their densities as Estimates of parameters and : Pooled standard deviation: Estimates of mean and standard dev. of scores for all clients :
Some results for normally distributed scores
( )
2 2
2
2 1 ) (
g g
x g GOOD
e x f
σ µ
π σ
− −
=
( )
2 2
2
2 1 ) (
b b
x b BAD
e x f
σ µ
π σ
− −
=
g b b
σ µ µ , ,
b
σ
, .
g
M
b
M
g
S
b
S
, are standard deviations of good (bad) clients , are means of good (bad) clients
2 1 2 2
+ + = m n mS nS S
b g
m n mM nM M M
b g ALL
+ + = =
( )
2 1 2 2 2 2 2
+ + = m n S m S n S
b g ALL
ALL ALL σ
µ ,
SLIDE 25
25/30
1 2 2 2 2 − Φ ⋅ = − Φ − Φ = D D D KS
Where is the standardized normal distribution function, the normal distribution function with parameters , and is the standard quantile function.
σ µ µ
b g
D − =
( )
⋅ + Φ ⋅ Φ =
−
D p q q Lift
G ALL q 1
1 σ σ
2
D Ival = 1 2 2 − Φ ⋅ = D Gini S M M D
b g −
=
Some results for normally distributed scores
Assume that standard deviations are equal to a common value :
σ
( )
⋅ Φ ) (
1 ⋅
Φ − ) (
2
,
⋅ Φ
σ µ
µ
2
σ
( )
⋅ + Φ Φ =
−
D p q S S q Lift
G ALL q 1
1
SLIDE 26
26/30
Generally (i.e. without assumption of equality of standard deviations): ⋅ + − ⋅ Φ − ⋅ + − ⋅ Φ = c b D a b D b a c b D a b D b a KS
b g g b
2 1 2 1
2 2
* 2 * * 2 *
σ σ σ σ
Some results for normally distributed scores
,
2 2 g b
a σ σ + =
2 2 * b g b g
D σ σ µ µ + − =
2 2 * b g b g
S S M M D + − = where
=
b g
c σ σ ln
,
2 2 g b
b σ σ − =
( ) ( ) ( ) ( )
− ⋅ + + − − ⋅ − + Φ − − ⋅ + + − − ⋅ − + Φ =
b g g b g b b g b g g b g b b g g b g b g g b b g b g b
S S S S D S S S S S D S S S S S S S S S D S S S S S D S S S S S KS ln 2 1 ln 2 1
2 2 * 2 2 2 2 * 2 2 2 2 2 2 * 2 2 2 2 * 2 2 2 2
2 2
SLIDE 27
27/30
Generally (i.e. without assumption of equality of standard deviations):
( ) 1
2
* −
Φ ⋅ = D Gini
Some results for normally distributed scores
( )
( )
( )
− + Φ ⋅ Φ = Φ ⋅ + Φ =
− − b b ALL ALL ALL ALL q
q q q q Lift
b b
σ µ µ σ σ µ
σ µ 1 1 ,
1 1
2
+ = − + + =
2 2 2 2 2 *
2 1 , 1 ) 1 (
b g g b val
A A D A I σ σ σ σ
+ = − + + =
2 2 2 2 2 *
2 1 , 1 ) 1 (
b g g b val
S S S S A A D A I
( )
− + Φ ⋅ Φ =
− b b ALL q
S M M q S q Lift
1
1
SLIDE 28
28/30
KS and the Gini react much more to change of and are almost unchanged in the direction of . Gini ,
Some results for normally distributed scores
=
b
µ 1
2 = b
σ
KS: ,
=
b
µ 1
2 = b
σ
- Gini > KS
g
µ
2 g
σ
SLIDE 29
29/30
Lift10%: ,
Some results for normally distributed scores
=
b
µ 1
2 = b
σ
Ival: ,
=
b
µ 1
2 = b
σ
In case of Lift10% it is evident strong dependence on and significantly higher dependence
- n than in case
- f KS and Gini.
Again strong dependence on . Furthermore value
- f Ival
rises very quickly to infinity when tends to zero.
g
µ
2 g
σ
g
µ
2 g
σ
SLIDE 30
30/30