estimating information flow in deep neural networks
play

Estimating Information Flow in Deep Neural Networks Ziv Goldfeld , - PowerPoint PPT Presentation

Feb 13, 2023 •561 likes •1.71k views

Estimating Information Flow in Deep Neural Networks Ziv Goldfeld , Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury and Yury Polyanskiy MIT, IBM Research, MIT-IBM Watson AI Lab International Conference on Machine

  1. Vacuous Mutual Information & Mis-Estimation Proposition (Informal) Det. DNNs with strictly monotone nonlinearities (e.g., tanh or sigmoid) ⇒ I ( X ; T ℓ ) is independent of the DNN parameters = I ( X ; T ℓ ) a.s. infinite (continuous X ) or constant H ( X ) (discrete X ) Past Works: Use binning-based proxy of I ( X ; T ℓ ) (aka quantization) � � For non-negligible bin size I X ; Bin ( T ℓ ) � = I ( X ; T ℓ ) 1 � � I X ; Bin ( T ℓ ) highly sensitive to user-defined bin size: 2 bin size = 0.0001 bin size = 0.001 bin size = 0.01 bin size = 0.1 8 Layer 1 MI (nats) Layer 2 4 Layer 3 Layer 4 Layer 5 0 10 0 10 1 10 2 10 3 10 4 Epoch 5/11

  2. Vacuous Mutual Information & Mis-Estimation Proposition (Informal) Det. DNNs with strictly monotone nonlinearities (e.g., tanh or sigmoid) ⇒ I ( X ; T ℓ ) is independent of the DNN parameters = I ( X ; T ℓ ) a.s. infinite (continuous X ) or constant H ( X ) (discrete X ) Past Works: Use binning-based proxy of I ( X ; T ℓ ) (aka quantization) � � For non-negligible bin size I X ; Bin ( T ℓ ) � = I ( X ; T ℓ ) 1 � � I X ; Bin ( T ℓ ) highly sensitive to user-defined bin size: 2 bin size = 0.0001 bin size = 0.001 bin size = 0.01 bin size = 0.1 8 Layer 1 MI (nats) Layer 2 4 Layer 3 Layer 4 Layer 5 0 10 0 10 1 10 2 10 3 10 4 Epoch ⊛ ⊛ ⊛ Real Problem: Mismatch between I ( X ; T ℓ ) measurement and model 5/11

  3. Auxiliary Framework - Noisy Deep Neural Networks Modification: Inject (small) Gaussian noise to neurons’ output 6/11

  4. Auxiliary Framework - Noisy Deep Neural Networks Modification: Inject (small) Gaussian noise to neurons’ output Formally: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) T 2 · · · f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 X Z 1 Z 1 Z 2 Z 2 6/11

  5. Auxiliary Framework - Noisy Deep Neural Networks Modification: Inject (small) Gaussian noise to neurons’ output Formally: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) T 2 · · · f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 X Z 1 Z 1 Z 2 Z 2 = ⇒ X �→ T ℓ is a parametrized channel (by DNN’s parameters) 6/11

  6. Auxiliary Framework - Noisy Deep Neural Networks Modification: Inject (small) Gaussian noise to neurons’ output Formally: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) T 2 · · · f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 X Z 1 Z 1 Z 2 Z 2 = ⇒ X �→ T ℓ is a parametrized channel (by DNN’s parameters) = ⇒ I ( X ; T ℓ ) is a function of parameters! 6/11

  7. Auxiliary Framework - Noisy Deep Neural Networks Modification: Inject (small) Gaussian noise to neurons’ output Formally: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) T 2 · · · f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 X Z 1 Z 1 Z 2 Z 2 = ⇒ X �→ T ℓ is a parametrized channel (by DNN’s parameters) = ⇒ I ( X ; T ℓ ) is a function of parameters! ⊛ ⊛ Challenge: How to accurately track I ( X ; T ℓ ) ? ⊛ 6/11

  8. High-Dim. & Nonparametric Functional Estimation 7/11

  9. High-Dim. & Nonparametric Functional Estimation Distill I ( X ; T ℓ ) Estimation into Noisy Differential Entropy Estimation: Estimate h ( P ∗ N σ ) from n i.i.d. samples S n � ( S i ) n i =1 of P ∈ F d (non- parametric class) and knowledge of N σ (Gaussian measure N (0 , σ 2 I d ) ). 7/11

  10. High-Dim. & Nonparametric Functional Estimation Distill I ( X ; T ℓ ) Estimation into Noisy Differential Entropy Estimation: Estimate h ( P ∗ N σ ) from n i.i.d. samples S n � ( S i ) n i =1 of P ∈ F d (non- parametric class) and knowledge of N σ (Gaussian measure N (0 , σ 2 I d ) ). Theorem (ZG-Greenewald-Polyanskiy-Weed’19) � � 2 d Sample complexity of any accurate estimator (additive gap η ) is Ω ηd 7/11

  11. High-Dim. & Nonparametric Functional Estimation Distill I ( X ; T ℓ ) Estimation into Noisy Differential Entropy Estimation: Estimate h ( P ∗ N σ ) from n i.i.d. samples S n � ( S i ) n i =1 of P ∈ F d (non- parametric class) and knowledge of N σ (Gaussian measure N (0 , σ 2 I d ) ). Theorem (ZG-Greenewald-Polyanskiy-Weed’19) � � 2 d Sample complexity of any accurate estimator (additive gap η ) is Ω ηd n Structured Estimator ⋆ : ˆ h ( S n , σ ) � h ( ˆ P n ∗ N σ ) , where ˆ P n = 1 � δ S i n i =1 ⋆ Efficient and parallelizable 7/11

  12. High-Dim. & Nonparametric Functional Estimation Distill I ( X ; T ℓ ) Estimation into Noisy Differential Entropy Estimation: Estimate h ( P ∗ N σ ) from n i.i.d. samples S n � ( S i ) n i =1 of P ∈ F d (non- parametric class) and knowledge of N σ (Gaussian measure N (0 , σ 2 I d ) ). Theorem (ZG-Greenewald-Polyanskiy-Weed’19) � � 2 d Sample complexity of any accurate estimator (additive gap η ) is Ω ηd n Structured Estimator ⋆ : ˆ h ( S n , σ ) � h ( ˆ P n ∗ N σ ) , where ˆ P n = 1 � δ S i n i =1 Theorem (ZG-Greenewald-Polyanskiy-Weed’19) For F ( SG ) � P is K -subgaussian in R d � , d ≥ 1 and σ > 0 , we have d,K � � P � σ,K · n − 1 � � � h ( P ∗ N σ ) − ˆ h ( S n , σ ) � ≤ c d sup P ∈F ( SG ) d,K E S n � � 2 7/11

  13. High-Dim. & Nonparametric Functional Estimation Distill I ( X ; T ℓ ) Estimation into Noisy Differential Entropy Estimation: Estimate h ( P ∗ N σ ) from n i.i.d. samples S n � ( S i ) n i =1 of P ∈ F d (non- parametric class) and knowledge of N σ (Gaussian measure N (0 , σ 2 I d ) ). Theorem (ZG-Greenewald-Polyanskiy-Weed’19) � � 2 d Sample complexity of any accurate estimator (additive gap η ) is Ω ηd n Structured Estimator ⋆ : ˆ h ( S n , σ ) � h ( ˆ P n ∗ N σ ) , where ˆ P n = 1 � δ S i n i =1 Theorem (ZG-Greenewald-Polyanskiy-Weed’19) For F ( SG ) � P is K -subgaussian in R d � , d ≥ 1 and σ > 0 , we have d,K � � P � σ,K · n − 1 � � � h ( P ∗ N σ ) − ˆ h ( S n , σ ) � ≤ c d sup P ∈F ( SG ) d,K E S n � � 2 Optimality: ˆ h ( S n , σ ) attains sharp dependence on both n and d ! 7/11

  14. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Z ∼ N (0 , σ 2 ) 8/11

  15. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) 8/11

  16. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) S 1 , 0 3 3 3 3 8/11

  17. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) S 1 , 0 3 3 3 3 ⊛ ⊛ ⊛ Center & sharpen transition ( ⇐ ⇒ increase w and keep b = − 2 w ) 8/11

  18. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) S 1 , 0 S 5 , − 10 3 3 3 3 8/11

  19. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) S 1 , 0 S 5 , − 10 3 3 3 3 ✓ Correct classification performance 8/11

  20. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) Mutual Information: 8/11

  21. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) Mutual Information: I ( X ; T ) = I ( S w,b ; S w,b + Z ) 8/11

  22. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) Mutual Information: I ( X ; T ) = I ( S w,b ; S w,b + Z ) = ⇒ I ( X ; T ) is # bits (nats) transmittable over AWGN with symbols S w,b � � tanh( − 3 w + b ) , tanh( − w + b ) , tanh( w + b ) , tanh(3 w + b ) � 8/11

  23. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) Mutual Information: I ( X ; T ) = I ( S w,b ; S w,b + Z ) = ⇒ I ( X ; T ) is # bits (nats) transmittable over AWGN with symbols � − S w,b � � tanh( − 3 w + b ) , tanh( − w + b ) , tanh( w + b ) , tanh(3 w + b ) → {± 1 } 8/11

  24. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) Mutual Information: I ( X ; T ) = I ( S w,b ; S w,b + Z ) = ⇒ I ( X ; T ) is # bits (nats) transmittable over AWGN with symbols � − S w,b � � tanh( − 3 w + b ) , tanh( − w + b ) , tanh( w + b ) , tanh(3 w + b ) → {± 1 } 8/11

  25. I ( X ; T ℓ ) Dynamics - Illustrative Minimal Example Single Neuron Classification: tanh( wX + b ) S w,b X T Input: X ∼ Unif {± 1 , ± 3 } X y = − 1 � {− 3 , − 1 , 1 } , X y =1 � { 3 } Z ∼ N (0 , σ 2 ) Mutual Information: I ( X ; T ) = I ( S w,b ; S w,b + Z ) = ⇒ I ( X ; T ) is # bits (nats) transmittable over AWGN with symbols � − S w,b � � tanh( − 3 w + b ) , tanh( − w + b ) , tanh( w + b ) , tanh(3 w + b ) → {± 1 } 1.5 ln(4) Mutual information ln(3) 1 ln(2) 0.5 0 10 0 10 2 10 4 10 6 Epoch 8/11

  26. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: 9/11

  27. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP 9/11

  28. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP 9/11

  29. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP Verified in multiple additional experiments 9/11

  30. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP Verified in multiple additional experiments = ⇒ Compression of I ( X ; T ℓ ) driven by clustering of representations 9/11

  31. Circling Back to Deterministic DNNs ⇒ I ( X ; T ℓ ) is constant/infinite = Doesn’t measure clustering 10/11

  32. Circling Back to Deterministic DNNs ⇒ I ( X ; T ℓ ) is constant/infinite = Doesn’t measure clustering � = H Reexamine Measurements: Computed I � X ; Bin ( T ℓ ) � Bin ( T ℓ ) � 10/11

  33. Circling Back to Deterministic DNNs ⇒ I ( X ; T ℓ ) is constant/infinite = Doesn’t measure clustering � = H Reexamine Measurements: Computed I � X ; Bin ( T ℓ ) � Bin ( T ℓ ) � � measures clustering (maximized by uniform distribution) � Bin ( T ℓ ) H 10/11

  34. Circling Back to Deterministic DNNs ⇒ I ( X ; T ℓ ) is constant/infinite = Doesn’t measure clustering � = H Reexamine Measurements: Computed I � X ; Bin ( T ℓ ) � Bin ( T ℓ ) � � measures clustering (maximized by uniform distribution) � Bin ( T ℓ ) H � highly correlated in noisy DNNs ⋆ Test: I ( X ; T ℓ ) and H � Bin ( T ℓ ) ⋆ When bin size chosen ∝ noise std. 10/11

  35. Circling Back to Deterministic DNNs ⇒ I ( X ; T ℓ ) is constant/infinite = Doesn’t measure clustering � = H Reexamine Measurements: Computed I � X ; Bin ( T ℓ ) � Bin ( T ℓ ) � � measures clustering (maximized by uniform distribution) � Bin ( T ℓ ) H � highly correlated in noisy DNNs ⋆ Test: I ( X ; T ℓ ) and H � Bin ( T ℓ ) = ⇒ Past works not measuring MI but clustering (via binned-MI)! 10/11

  36. Circling Back to Deterministic DNNs ⇒ I ( X ; T ℓ ) is constant/infinite = Doesn’t measure clustering � = H Reexamine Measurements: Computed I � X ; Bin ( T ℓ ) � Bin ( T ℓ ) � � measures clustering (maximized by uniform distribution) � Bin ( T ℓ ) H � highly correlated in noisy DNNs ⋆ Test: I ( X ; T ℓ ) and H � Bin ( T ℓ ) = ⇒ Past works not measuring MI but clustering (via binned-MI)! By-Product Result: 10/11

  37. Circling Back to Deterministic DNNs ⇒ I ( X ; T ℓ ) is constant/infinite = Doesn’t measure clustering � = H Reexamine Measurements: Computed I � X ; Bin ( T ℓ ) � Bin ( T ℓ ) � � measures clustering (maximized by uniform distribution) � Bin ( T ℓ ) H � highly correlated in noisy DNNs ⋆ Test: I ( X ; T ℓ ) and H � Bin ( T ℓ ) = ⇒ Past works not measuring MI but clustering (via binned-MI)! By-Product Result: Refute ‘compression (tight clustering) improves generalization’ claim [Come see us at poster #96 for details] 10/11

  38. Summary Reexamined Information Bottleneck Compression: 11/11

  39. Summary Reexamined Information Bottleneck Compression: ◮ I ( X ; T ) fluctuations in det. DNNs are theoretically impossible 11/11

  40. Summary Reexamined Information Bottleneck Compression: ◮ I ( X ; T ) fluctuations in det. DNNs are theoretically impossible ◮ Yet, past works presented (binned) I ( X ; T ) dynamics during training 11/11

  41. Summary Reexamined Information Bottleneck Compression: ◮ I ( X ; T ) fluctuations in det. DNNs are theoretically impossible ◮ Yet, past works presented (binned) I ( X ; T ) dynamics during training Noisy DNN Framework: Studying IT quantities over DNNs 11/11

  42. Summary Reexamined Information Bottleneck Compression: ◮ I ( X ; T ) fluctuations in det. DNNs are theoretically impossible ◮ Yet, past works presented (binned) I ( X ; T ) dynamics during training Noisy DNN Framework: Studying IT quantities over DNNs ◮ Optimal estimator (in n and d ) for accurate MI estimation 11/11

  43. Summary Reexamined Information Bottleneck Compression: ◮ I ( X ; T ) fluctuations in det. DNNs are theoretically impossible ◮ Yet, past works presented (binned) I ( X ; T ) dynamics during training Noisy DNN Framework: Studying IT quantities over DNNs ◮ Optimal estimator (in n and d ) for accurate MI estimation ◮ Clustering of learned representations is the source of compression 11/11

  44. Summary Reexamined Information Bottleneck Compression: ◮ I ( X ; T ) fluctuations in det. DNNs are theoretically impossible ◮ Yet, past works presented (binned) I ( X ; T ) dynamics during training Noisy DNN Framework: Studying IT quantities over DNNs ◮ Optimal estimator (in n and d ) for accurate MI estimation ◮ Clustering of learned representations is the source of compression Clarify Past Observations of Compression: in fact show clustering 11/11

  45. Summary Reexamined Information Bottleneck Compression: ◮ I ( X ; T ) fluctuations in det. DNNs are theoretically impossible ◮ Yet, past works presented (binned) I ( X ; T ) dynamics during training Noisy DNN Framework: Studying IT quantities over DNNs ◮ Optimal estimator (in n and d ) for accurate MI estimation ◮ Clustering of learned representations is the source of compression Clarify Past Observations of Compression: in fact show clustering ◮ Compression/clustering and generalization and not necessarily related 11/11

  46. Summary Reexamined Information Bottleneck Compression: ◮ I ( X ; T ) fluctuations in det. DNNs are theoretically impossible ◮ Yet, past works presented (binned) I ( X ; T ) dynamics during training Noisy DNN Framework: Studying IT quantities over DNNs ◮ Optimal estimator (in n and d ) for accurate MI estimation ◮ Clustering of learned representations is the source of compression Clarify Past Observations of Compression: in fact show clustering ◮ Compression/clustering and generalization and not necessarily related Thank you! 11/11

  47. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: 11/11

  48. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP 11/11

  49. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP 11/11

  50. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP 11/11

  51. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP ⊛ ⊛ ⊛ weight orthonormality regularization [Cisse et al. ’17] 11/11

  52. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP Verified in multiple additional experiments 11/11

  53. Clustering of Representations - Larger Networks Noisy version of DNN from [Shwartz-Tishby’17]: Binary Classification: 12-bit input & 12– 10 – 7 – 5 – 4 – 3 –2 tanh MLP Verified in multiple additional experiments = ⇒ Compression of I ( X ; T ℓ ) driven by clustering of representations 11/11

  54. Mutual Information Estimation in Noisy DNNs Noisy DNN: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 T 2 · · · X Z 1 Z 1 Z 2 Z 2 11/11

  55. Mutual Information Estimation in Noisy DNNs Noisy DNN: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 T 2 · · · X Z 1 Z 1 Z 2 Z 2 � d P X ( x ) h ( T ℓ | X = x ) Mutual Information: I ( X ; T ℓ ) = h ( T ℓ ) − 11/11

  56. Mutual Information Estimation in Noisy DNNs Noisy DNN: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 T 2 · · · X Z 1 Z 1 Z 2 Z 2 � d P X ( x ) h ( T ℓ | X = x ) Mutual Information: I ( X ; T ℓ ) = h ( T ℓ ) − Structure: S ℓ ⊥ Z ℓ = ⇒ T ℓ = S ℓ + Z ℓ ∼ P ∗ N σ 11/11

  57. Mutual Information Estimation in Noisy DNNs Noisy DNN: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 T 2 · · · X Z 1 Z 1 Z 2 Z 2 � d P X ( x ) h ( T ℓ | X = x ) Mutual Information: I ( X ; T ℓ ) = h ( T ℓ ) − Structure: S ℓ ⊥ Z ℓ = ⇒ T ℓ = S ℓ + Z ℓ ∼ P ∗ N σ 11/11

  58. Mutual Information Estimation in Noisy DNNs Noisy DNN: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 T 2 · · · X Z 1 Z 1 Z 2 Z 2 � d P X ( x ) h ( T ℓ | X = x ) Mutual Information: I ( X ; T ℓ ) = h ( T ℓ ) − T ℓ = S ℓ + Z ℓ ∼ P ∗ N σ Structure: S ℓ ⊥ Z ℓ = ⇒ 11/11

  59. Mutual Information Estimation in Noisy DNNs Noisy DNN: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) f 1 f 1 S 1 S 1 T 1 f 2 f 2 S 2 S 2 T 2 · · · X Z 1 Z 1 Z 2 Z 2 � d P X ( x ) h ( T ℓ | X = x ) Mutual Information: I ( X ; T ℓ ) = h ( T ℓ ) − Structure: S ℓ ⊥ Z ℓ = ⇒ T ℓ = S ℓ + Z ℓ ∼ P ∗ N σ ⊛ ⊛ Know the distribution N σ of Z ℓ (noise injected by design) ⊛ 11/11

  60. Mutual Information Estimation in Noisy DNNs Noisy DNN: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) f 1 f 1 S 1 S 1 T 1 T 1 f 2 f 2 S 2 S 2 T 2 · · · X Z 1 Z 1 Z 2 Z 2 � d P X ( x ) h ( T ℓ | X = x ) Mutual Information: I ( X ; T ℓ ) = h ( T ℓ ) − Structure: S ℓ ⊥ Z ℓ = ⇒ T ℓ = S ℓ + Z ℓ ∼ P ∗ N σ ⊛ ⊛ Know the distribution N σ of Z ℓ (noise injected by design) ⊛ 11/11

  61. Mutual Information Estimation in Noisy DNNs Noisy DNN: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) f 1 f 1 S 1 S 1 T 1 T 1 f 2 f 2 S 2 S 2 T 2 · · · X Z 1 Z 1 Z 2 Z 2 � d P X ( x ) h ( T ℓ | X = x ) Mutual Information: I ( X ; T ℓ ) = h ( T ℓ ) − Structure: S ℓ ⊥ Z ℓ = ⇒ T ℓ = S ℓ + Z ℓ ∼ P ∗ N σ ⊛ ⊛ ⊛ Know the distribution N σ of Z ℓ (noise injected by design) ⊛ ⊛ ⊛ Extremely complicated P = ⇒ Treat as unknown 11/11

  62. Mutual Information Estimation in Noisy DNNs Noisy DNN: T ℓ = S ℓ + Z ℓ , where S ℓ � f ℓ ( T ℓ − 1 ) and Z ℓ ∼ N (0 , σ 2 I d ) f 1 f 1 S 1 S 1 T 1 T 1 f 2 f 2 S 2 S 2 T 2 · · · X Z 1 Z 1 Z 2 Z 2 � d P X ( x ) h ( T ℓ | X = x ) Mutual Information: I ( X ; T ℓ ) = h ( T ℓ ) − Structure: S ℓ ⊥ Z ℓ = ⇒ T ℓ = S ℓ + Z ℓ ∼ P ∗ N σ ⊛ ⊛ Know the distribution N σ of Z ℓ (noise injected by design) ⊛ ⊛ ⊛ ⊛ Extremely complicated P = ⇒ Treat as unknown ⊛ ⊛ ⊛ Easily get i.i.d. samples from P via DNN forward pass 11/11

  63. Structured Estimator (with Implementation in Mind) Differential Entropy Estimation under Gaussian Convolutions Estimate h ( P ∗ N σ ) via n i.i.d. samples S n � ( S i ) n i =1 from unknown P ∈ F d (nonparametric class) and knowledge of N σ (noise distribution). 11/11

  64. Structured Estimator (with Implementation in Mind) Differential Entropy Estimation under Gaussian Convolutions Estimate h ( P ∗ N σ ) via n i.i.d. samples S n � ( S i ) n i =1 from unknown P ∈ F d (nonparametric class) and knowledge of N σ (noise distribution). Nonparametric Class: Specified by DNN architecture ( d = T ℓ ‘width’) 11/11

  65. Structured Estimator (with Implementation in Mind) Differential Entropy Estimation under Gaussian Convolutions Estimate h ( P ∗ N σ ) via n i.i.d. samples S n � ( S i ) n i =1 from unknown P ∈ F d (nonparametric class) and knowledge of N σ (noise distribution). Nonparametric Class: Specified by DNN architecture ( d = T ℓ ‘width’) Goal: Simple & parallelizable for efficient implementation 11/11

  66. Structured Estimator (with Implementation in Mind) Differential Entropy Estimation under Gaussian Convolutions Estimate h ( P ∗ N σ ) via n i.i.d. samples S n � ( S i ) n i =1 from unknown P ∈ F d (nonparametric class) and knowledge of N σ (noise distribution). Nonparametric Class: Specified by DNN architecture ( d = T ℓ ‘width’) Goal: Simple & parallelizable for efficient implementation n Estimator: ˆ h ( S n , σ ) � h ( ˆ P S n ∗ N σ ) , where ˆ P S n � 1 � δ S i n i =1 11/11

  67. Structured Estimator (with Implementation in Mind) Differential Entropy Estimation under Gaussian Convolutions Estimate h ( P ∗ N σ ) via n i.i.d. samples S n � ( S i ) n i =1 from unknown P ∈ F d (nonparametric class) and knowledge of N σ (noise distribution). Nonparametric Class: Specified by DNN architecture ( d = T ℓ ‘width’) Goal: Simple & parallelizable for efficient implementation n Estimator: ˆ h ( S n , σ ) � h ( ˆ P S n ∗ N σ ) , where ˆ P S n � 1 � δ S i n i =1 Plug-in: ˆ h is plug-in est. for the functional T σ ( P ) � h ( P ∗ N σ ) 11/11

  68. Structured Estimator - Convergence Rate Theorem (ZG-Greenewald-Weed-Polyanskiy’19) For any σ > 0 , d ≥ 1 , we have � � ≤ C σ,d,K · n − 1 � � h ( P ∗ N σ ) − h ( ˆ P S n ∗ N σ ) sup E � � 2 P ∈F ( SG ) d,K where C σ,d,K = O σ,K ( c d ) for a constant c . 11/11

  69. Structured Estimator - Convergence Rate Theorem (ZG-Greenewald-Weed-Polyanskiy’19) For any σ > 0 , d ≥ 1 , we have � � ≤ C σ,d,K · n − 1 � � h ( P ∗ N σ ) − h ( ˆ P S n ∗ N σ ) sup E � � 2 P ∈F ( SG ) d,K where C σ,d,K = O σ,K ( c d ) for a constant c . Comments: 11/11

  70. Structured Estimator - Convergence Rate Theorem (ZG-Greenewald-Weed-Polyanskiy’19) For any σ > 0 , d ≥ 1 , we have � � ≤ C σ,d,K · n − 1 � � h ( P ∗ N σ ) − h ( ˆ P S n ∗ N σ ) sup E � � 2 P ∈F ( SG ) d,K where C σ,d,K = O σ,K ( c d ) for a constant c . Comments: Explicit Expression: Enables concrete error bounds in simulations 11/11

  71. Structured Estimator - Convergence Rate Theorem (ZG-Greenewald-Weed-Polyanskiy’19) For any σ > 0 , d ≥ 1 , we have � � ≤ C σ,d,K · n − 1 � � h ( P ∗ N σ ) − h ( ˆ P S n ∗ N σ ) sup E � � 2 P ∈F ( SG ) d,K where C σ,d,K = O σ,K ( c d ) for a constant c . Comments: Explicit Expression: Enables concrete error bounds in simulations � n − 1 2 � Minimax Rate Optimal: Attains parametric estimation rate O 11/11

  72. Structured Estimator - Convergence Rate Theorem (ZG-Greenewald-Weed-Polyanskiy’19) For any σ > 0 , d ≥ 1 , we have � � � ≤ C σ,d,K · n − 1 � h ( P ∗ N σ ) − h ( ˆ P S n ∗ N σ ) sup E � � 2 P ∈F ( SG ) d,K where C σ,d,K = O σ,K ( c d ) for a constant c . Comments: Explicit Expression: Enables concrete error bounds in simulations � n − 1 2 � Minimax Rate Optimal: Attains parametric estimation rate O Proof (initial step): Based on [Polyanskiy-Wu’16] � � � h ( P ∗ N σ ) − h ( ˆ � � W 1 ( P ∗ N σ , ˆ P S n ∗ N σ ) P S n ∗ N σ ) � � 11/11

  73. Structured Estimator - Convergence Rate Theorem (ZG-Greenewald-Weed-Polyanskiy’19) For any σ > 0 , d ≥ 1 , we have � � � ≤ C σ,d,K · n − 1 � h ( P ∗ N σ ) − h ( ˆ P S n ∗ N σ ) sup E � � 2 P ∈F ( SG ) d,K where C σ,d,K = O σ,K ( c d ) for a constant c . Comments: Explicit Expression: Enables concrete error bounds in simulations � n − 1 2 � Minimax Rate Optimal: Attains parametric estimation rate O Proof (initial step): Based on [Polyanskiy-Wu’16] � � � h ( P ∗ N σ ) − h ( ˆ � � W 1 ( P ∗ N σ , ˆ P S n ∗ N σ ) P S n ∗ N σ ) � � = ⇒ Analyze empirical 1-Wasserstein distance under Gaussian convolutions 11/11

  74. Empirical W 1 & The Magic of Gaussian Convolution p -Wasserstein Distance: For two distributions P and Q on R d and p ≥ 1 � E � X − Y � p � 1 /p W p ( P, Q ) � inf infimum over all couplings of P and Q 11/11

  75. Empirical W 1 & The Magic of Gaussian Convolution p -Wasserstein Distance: For two distributions P and Q on R d and p ≥ 1 � E � X − Y � p � 1 /p W p ( P, Q ) � inf infimum over all couplings of P and Q Empirical 1-Wasserstein Distance: 11/11

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1.

WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1.

OPTICAL FLOW ESTIMATION WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1. Optical Flow Motivation 2. Neural Networks Basics 3. Optical Flow with Deep Neural Networks 1. PWC-Net Model 2. PWC-Net Results

481 views • 20 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Introduction to Artificial Intelligence Deep Learning - Tensor Flow Janyl Jumadinova December 2,

Introduction to Artificial Intelligence Deep Learning - Tensor Flow Janyl Jumadinova December 2,

Introduction to Artificial Intelligence Deep Learning - Tensor Flow Janyl Jumadinova December 2, 2016 Credit: Google Workshop Neural Networks 2/24 Neural Networks 3/24 Neural Networks A fully connected NN layer 4/24 Implementation as

1.2k views • 34 slides
ON THE ADVERSARIAL ROBUSTNESS OF UNCERTAINTY AWARE DEEP NEURAL NETWORKS APRIL 29 TH , 2019

ON THE ADVERSARIAL ROBUSTNESS OF UNCERTAINTY AWARE DEEP NEURAL NETWORKS APRIL 29 TH , 2019

ON THE ADVERSARIAL ROBUSTNESS OF UNCERTAINTY AWARE DEEP NEURAL NETWORKS APRIL 29 TH , 2019 PREPARED BY: ALI HARAKEH QUESTION Can a neural network mitigate the effects of adversarial attacks by estimating the uncertainty in its predictions ?

1.14k views • 26 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural Networks and loss surfaces Problems of Deep Architectures Optimization in Neural Networks Under fitting Proliferation of Saddle

1.67k views • 41 slides
Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference ICTP, Trieste, Jan 29-31 2018 Deep Neural Networks When properly trained, deep NN are capable of learning the complex functional dependence of

442 views • 17 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream Umut Gcl and Marcel A. J. van Gerven Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University of Tartu

852 views • 63 slides
Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning and AI y t-1 y t y t+1 Computer vision h t-1 h t h t+1 Language translation Speech recognition h t-1 h t h t+1 Question answering x t-1 x t x t+1

454 views • 43 slides
Statistical mechanics as a paradigm for complex systems Motivation, foundations and limitations

Statistical mechanics as a paradigm for complex systems Motivation, foundations and limitations

Motivation Foundations Limitations Statistical mechanics as a paradigm for complex systems Motivation, foundations and limitations Roberto Fern andez Utrecht University and Neuromat Onthology droplet S ao Paulo, January 2014

632 views • 42 slides
egoSlider: Visual Analysis of Egocentric Network Evolution Yanhong Wu , Naveen Pitipornvivat, Jian

egoSlider: Visual Analysis of Egocentric Network Evolution Yanhong Wu , Naveen Pitipornvivat, Jian

egoSlider: Visual Analysis of Egocentric Network Evolution Yanhong Wu , Naveen Pitipornvivat, Jian Zhao, Sixiao Yang, Guowei Huang, and Huamin Qu Definition Ego-network: the relationships between a specific individual, i.e., the ego and

516 views • 39 slides
Macroscopic Zeno effect and stationary flows in nonlinear waveguides with localized dissipation D.

Macroscopic Zeno effect and stationary flows in nonlinear waveguides with localized dissipation D.

Macroscopic Zeno effect and stationary flows in nonlinear waveguides with localized dissipation D. A. Zezyulin 1 , V. V. Konotop 1 , G. Barontini 2 , and H. Ott 2 1 Center for Theoretical and Computational Physics, University of Lisbon, Portugal 2

438 views • 23 slides
The intrinsic hypoelliptic Laplacian on sub-Riemannian manifolds Ugo Boscain (CNRS, CMAP, Ecole

The intrinsic hypoelliptic Laplacian on sub-Riemannian manifolds Ugo Boscain (CNRS, CMAP, Ecole

The intrinsic hypoelliptic Laplacian on sub-Riemannian manifolds Ugo Boscain (CNRS, CMAP, Ecole Polytechnique, Paris) Robert Neel , (Lehigh University) Luca Rizzi , (CMAP, Ecole Polytechnique, Paris) March 11, 2015 Plan of the Talk definition of

797 views • 33 slides
The densest subgraph of sparse random graphs Justin Salez (Universit e Paris 7) with Venkat

The densest subgraph of sparse random graphs Justin Salez (Universit e Paris 7) with Venkat

The densest subgraph of sparse random graphs Justin Salez (Universit e Paris 7) with Venkat Anantharam (UC Berkeley) The objective method (Aldous-Steele, 2004) The objective method (Aldous-Steele, 2004) Context: given a large interacting

1.59k views • 157 slides
Electrons: waves, particles ... or jellies? Sharif QI Group 6 th August 2020 (Angelo Bassi

Electrons: waves, particles ... or jellies? Sharif QI Group 6 th August 2020 (Angelo Bassi

Electrons: waves, particles ... or jellies? Sharif QI Group 6 th August 2020 (Angelo Bassi University of Trieste & INFN) The trouble with quantum mechanics I think I can Quantum safely say that mechanics is no one certainly

923 views • 32 slides
Origins: Is It Reasonable To Believe in God in This Scientific Age? Belief in God: Is It

Origins: Is It Reasonable To Believe in God in This Scientific Age? Belief in God: Is It

Origins: Is It Reasonable To Believe in God in This Scientific Age? Belief in God: Is It Reasonable In This Scientific Age? Rom. 1:20 - His invisible attributes understood by the things that are made His Eternal Power His Divine

791 views • 38 slides
A continuum based macroscopic unified low- and high cycle fatigue model Tero Frondelius 1 , 2 ,

A continuum based macroscopic unified low- and high cycle fatigue model Tero Frondelius 1 , 2 ,

A continuum based macroscopic unified low- and high cycle fatigue model Tero Frondelius 1 , 2 , Sami Holopainen 3 , Reijo Kouhia 3 , Niels Saabye Ottosen 4 , Matti Ristinmaa 4 , Joona Vaara 1 1 W artsil a Finland Oy, J arvikatu 2-4,

839 views • 11 slides

More recommend