On the stability of comparing histograms with help of probabilistic methods Alexander Lepskiy National Research University - Higher School of Economics, Moscow, Russia The 2 st International Conference on Information Technology and Quantitative Management, June 3 - 5, 2014, Moscow, Russia Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 1 / 24
Outline Outline of Presentation 1 Comparison of histograms Problem statement of comparison of histograms Applied problems where comparison of histograms is used Main approaches for comparison of histograms Some Probabilistic Indices of Comparison 2 Distortions of Histograms 3 Conditions of Preservation for Comparison of Distorted Histograms 4 Comparison of the Sets of Admissible Distortions 5 Example. Histograms of Unified State Exam of Universities 6 Summary and conclusion Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 2 / 24
Comparison of Histograms Problem statement of Comparison of Histograms Let U = { U } be a set of all histograms of form U = ( x i , u i ) i ∈ I , x i < x i +1 , i ∈ I . We want define the total preorder relation R (reflexive, complete and transitive relation) on U : ( U, V ) ∈ R ⇔ U � V . Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 3 / 24
Comparison of Histograms Ordering Arguments of Histograms The relation R should be in accord with the condition of the ordering of histogram arguments by ascending their importance: if U ′ = ( x i , u ′ i ), U ′′ = ( x i , u ′′ i ) be two histograms for which u ′ i = u ′′ i for k ≥ 0 then U ′′ � U ′ for k > l and all i � = k, l and u ′ l − u ′′ l = u ′′ k − u ′ U ′ � U ′′ for k < l . Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 4 / 24
Comparison of Histograms Application of Comparison of Histograms comparison of results of different experiences; comparison of indicators of functioning of the organizational, technical systems etc.; decision-making under fuzzy uncertainty; simulation of fuzzy preferences; comparisons of income distribution within the framework of socio-economic analysis; ranking of histogram data etc. Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 5 / 24
Comparison of Histograms Main Approaches for Comparison of Histograms probabilistic approach; ranking methods of income distribution in the theory of social choice. Histograms income has the form U = ( i, u i ) n U i =1 = ( u i ) n U i =1 , where u 1 ≤ u 2 ≤ ... ≤ u n U in this case. These histograms are compared with help of welfare functions W ( U ) that satisfy the conditions of symmetry, monotonicity, concavity, etc. using the tools of comparison of fuzzy numbers. The histogram U = ( x i , u i ) i ∈ I is associated with fuzzy set (or fuzzy number) by means of membership function U = ( u i ) i ∈ I which is defined on the universal set X = ( x i ) i ∈ I . Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 6 / 24
Comparison of Histograms Some Probabilistic Indices of Comparison We consider a numerical index r ( U, V ) of pairwise comparison of histograms U and V in U 2 . Let index r ( U, V ) is consistent with increasing of importance of arguments: if U = ( x i , u i ), V = ( x i , v i ) be two histograms for which u i = v i for all i � = k, l and u l − v l = v k − u k ≥ 0 then r ( U, V ) ≥ 0 for k > l and r ( U, V ) ≤ 0 for k < l . In particular r ( U, U ) = 0. Let ∆ r ( U, V ) = r ( U, V ) − r ( V, U ) ≥ 0 be a differential index of comparison . Let U = ( x i , u i ) i ∈ I and V = ( x j , v j ) j ∈ I are random variables taking values { x i } i ∈ I with probabilities { u i } i ∈ I and ( v j ) j ∈ I respectively. Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 7 / 24
Comparison of Histograms Examples of Indices Pairwise Comparison of Histograms 1. Comparison of mathematical expectations Let U � V if E [ U ] ≥ E [ V ]. In general U � V if E [ f ( U )] ≥ E [ f ( V )], where f is some utility function. 1 Let E 0 [ U ] = ∆ x ( E [ V ] − x min ) be a normalized index, where ∆ x = x max − x min , E 0 [ U ] ∈ [0 , 1]. 1 Let ∆ E ( U, V ) = E 0 [ U ] − E 0 [ V ] = ∆ x ( E [ U ] − E [ V ]) be a corresponding differential comparison index. 2. Comparison of distribution functions Let U � V if F U ( x ) ≤ F V ( x ) for all x ∈ R , where F U ( x ) = � i : x i <x u i is distribution function of random variable U . This is a principle of stochastic dominance of the 1st order . Let ∆ F ( U, V ) = x ∈ ( x min ,x max ] ( F U ( x ) − F V ( x )) be a corresponding inf differential comparison index. Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 8 / 24
Comparison of Histograms 3. Comparison of probabilities Let U � V if P { U ≥ V } ≥ P { U ≤ V } . This approach to comparison called by stochastic precedence ( V precedes U ). If we assume that the random variables U = ( x i , u i ) i ∈ I and V = ( x j , v j ) j ∈ I are independent then P { U ≥ V } = � u i v j . ( i,j ): x i ≥ x j The corresponding differential comparison index is denoted by ∆ P ( U, V ) = P { U ≥ V } − P { U ≤ V } . Notice that the inequality ∆ P ( U, V ) ≥ 0 does not specify a transitive relation. Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 9 / 24
Distortions of Histograms Distortions of Histograms The compared histograms may be distorted. The reasons of distortions : random noise; deliberate distortion of data; filling gap in incomplete data; etc. The α -distortion of histogram . Let U = ( x i , u i ) i ∈ I is a “ideal” histogram and ˜ U = ( x i , ˜ u i ) i ∈ I is an interval distortion of U : ˜ u i = u i + h i , i ∈ I , where � i ∈ I h i = 0 and | h i | ≤ αu i , i ∈ I , where α ∈ [0 , 1]. The value α characterize the threshold of distortion. Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 10 / 24
Distortions of Histograms Let � � � N α ( U ) = H = ( h i ) i ∈ I : i ∈ I h i = 0 , | h i | ≤ αu i , i ∈ I be a class of all α -distortion of histogram U = ( x i , u i ) i ∈ I . Main problem Suppose that ∆ r ( U, V ) > 0. In what case do we have ∆ r ( ˜ U, ˜ V ) ≥ 0 for all H ∈ N α ( U ) and G ∈ N β ( V )? By other words, when the comparison of histograms will not changed after α -distortion of histogram U = ( x i , u i ) i ∈ I and β -distortion of histogram V = ( x j , v j ) j ∈ I ? Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 11 / 24
Conditions of Preservation for Comparison Conservation Conditions of Comparison w.r.t. 1 ∆ E ( U, V ) = ∆ x ( E [ U ] − E [ V ]) Index We consider the value �� � i ∈ I x 0 E U = sup i h i : ( h i ) i ∈ I ∈ N 1 ( U ) 1 for U = ( x i , u i ) i ∈ I , where x 0 i = ∆ x ( x i − x min ) ∈ [0 , 1] ∀ i ∈ I . Lemma The estimation 0 ≤ E U ≤ min { E 0 [ U ] , 0 . 5 } is true. Proposition Let ˜ U = ( x i , u i + h i ) i ∈ I , ˜ V = ( x j , v j + g j ) i ∈ I be a α - and β -distortion of histograms U = ( x i , u i ) n i =1 and V = ( x j , v j ) n j =1 respectively. Then we have ∆ E ( ˜ U, ˜ V ) ≥ 0 for all ( h i ) i ∈ I ∈ N α ( U ) and ( g i ) i ∈ I ∈ N β ( V ) , α, β ∈ [0 , 1] iff ∆ E ( U, V ) ≥ α E U + β E V . Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 12 / 24
Conditions of Preservation for Comparison Let ¯ E U = min { E 0 [ U ] , 0 . 5 } . Corollary If we have ∆ E ( U, V ) ≥ α ¯ E U + β ¯ E V , then inequality ∆ E ( ˜ U, ˜ V ) ≥ 0 is true for all ( h i ) i ∈ I ∈ N α ( U ) and ( g i ) i ∈ I ∈ N β ( V ) . Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 13 / 24
Conditions of Preservation for Comparison Conservation Conditions of Comparison w.r.t ∆ F ( U, V ) = x ∈ ( x min ,x max ] ( F U ( x ) − F V ( x )) Index inf �� � Let F U ( x ) = sup i : x i <x h i : ( h i ) i ∈ I ∈ N 1 ( U ) . Lemma F U ( x ) = min { F U ( x ) , 1 − F U ( x ) } for all x ∈ R . Proposition Let ˜ U = ( x i , u i + h i ) i ∈ I , ˜ V = ( x j , v j + g j ) i ∈ I be a α - and β -distortion of histograms U = ( x i , u i ) i ∈ I and V = ( x j , v j ) i ∈ I respectively. Then we have ∆ F ( ˜ U, ˜ V ) ≥ 0 for all ( h i ) i ∈ I ∈ N α ( U ) and ( g i ) i ∈ I ∈ N β ( V ) iff F U ( x ) − F V ( x ) ≥ α F U ( x ) + β F V ( x ) for all x ∈ R . Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 14 / 24
Conditions of Preservation for Comparison Corollary The inequality ∆ F ( ˜ U, ˜ V ) ≥ 0 is true for all ( h i ) i ∈ I ∈ N α ( U ) and α F U ( x )+ β F V ( x ) ( g i ) i ∈ I ∈ N β ( V ) iff 0 ≤ sup ≤ 1 (the fraction is equal to F U ( x ) − F V ( x ) x zero if its numerator and denominator are equal to zero). Corollary x { α F U ( x ) + β F V ( x ) } then inequality ∆ F ( ˜ U, ˜ If ∆ F ( U, V ) ≥ sup V ) ≥ 0 is true for all ( h i ) i ∈ I ∈ N α ( U ) and ( g i ) i ∈ I ∈ N β ( V ) . Alexander Lepskiy (HSE) Stability of comparison ITQM 2014 15 / 24
Recommend
More recommend
