[PPT] - Towards Verified Stochastic Variational Inference for Probabilistic PowerPoint Presentation

SLIDE 1

Towards Verified Stochastic Variational Inference for Probabilistic Programs

Wonyeol Lee1 Hangyeol Yu1 Xavier Rival2 Hongseok Yang1

1KAIST, South Korea 2INRIA/ENS/CNRS, France

POPL 2020

SLIDE 2

Probabilistic c Programming

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) 2

SLIDE 3

Probabilistic c Programming

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) 3

SLIDE 4

Probabilistic c Programming

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) 4

SLIDE 5

Probabilistic c Programming

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) 5

SLIDE 6

Probabilistic c Programming

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.)

density !

prior " ! posterior " ! # 6

SLIDE 7

Probabilistic c Programming

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.)

density !

prior " ! posterior # $ % 7

SLIDE 8

Stoch chastic c Va Variational Inference ce

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) def qθ(): # guide_eg1 θ = pyro.param(“θ”, 0.) z = pyro.sample(“z”, Normal(θ, 1.))

≈

density !

prior " ! posterior " ! # 8

SLIDE 9

Stoch chastic c Va Variational Inference ce

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) def qθ(): # guide_eg1 θ = pyro.param(“θ”, 0.) z = pyro.sample(“z”, Normal(θ, 1.))

≈

density !

prior " ! posterior " ! # 9

SLIDE 10

Stoch chastic c Va Variational Inference ce

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) def qθ(): # guide_eg1 θ = pyro.param(“θ”, 0.) z = pyro.sample(“z”, Normal(θ, 1.))

≈

density !

prior " ! posterior " ! # 10

SLIDE 11

Stoch chastic c Va Variational Inference ce

Example 1:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) def qθ(): # guide_eg1 θ = pyro.param(“θ”, 0.) z = pyro.sample(“z”, Normal(θ, 1.))

≈

guide !"($) with optimal &

density $

prior ' $ posterior ' $ ( 11

SLIDE 12

Stoch chastic c Va Variational Inference ce

Typical optimization objective:

argmin' KL *' + || - + . ≜ 012 3 log

12 3 6 3|7

.

12

SLIDE 13

Stoch chastic c Va Variational Inference ce

Typical optimization objective:
Notation:

KL# = KL %# & || ( & ) . argmin# KL# ≜ 123 4 log

23 4 7 4|8

.

13

SLIDE 14

Stoch chastic c Va Variational Inference ce

Typical optimization objective:
Notation:

KL# = KL %# & || ( & ) .

Optimization by stochastic gradient descent:

*+,- ← *+ − 0.01× 4#KL# |#5#6. argmin# KL# ≜ >?@ A log

?@ A D A|E

.

14

SLIDE 15

Stoch chastic c Va Variational Inference ce

Typical optimization objective:
Notation:

KL# = KL %# & || ( & ) .

Optimization by stochastic gradient descent:

*+,- ← *+ − 0.01× 4#KL# |#5#6. argmin# KL# ≜ >?@ A log

?@ A D A|E

.

15

SLIDE 16

Stoch chastic c Va Variational Inference ce

Typical optimization objective:
Notation:

KL# = KL %# & || ( & ) .

Optimization by stochastic gradient descent:

*+,- ← *+ − 0.01× 4#KL# |#5#6. argmin# KL# ≜ >?@ A log

?@ A D A|E

.

16

SLIDE 17

Stoch chastic c Va Variational Inference ce

Typical optimization objective:
Notation:

KL# = KL %# & || ( & ) .

Optimization by stochastic gradient descent:

*+,- ← *+ − 0.01× 4#KL# |#5#6. argmin# KL# ≜ >?@ A log

?@ A D A|E

Wh What t can an go wrong? g?

17

SLIDE 18

Issues in Stoch chastic c Variational Inference ce

Typical optimization objective:
Notation:

KL# = KL %# & || ( & ) .

Optimization by stochastic gradient descent:

*+,- ← *+ − 0.01× 4#KL# |#5#6. argmin# KL# ≜ >?@ A log

?@ A D A|E

Issue 1: Undefined KL#

18

SLIDE 19

Issues in Stoch chastic c Variational Inference ce

Typical optimization objective:
Notation:

KL# = KL %# & || ( & ) .

Optimization by stochastic gradient descent:

*+,- ← *+ − 0.01× 4#KL# |#5#6. argmin# KL# ≜ >?@ A log

?@ A D A|E

Issue 2: Undefined 4#KL# Issue 1: Undefined KL#

19

SLIDE 20

Issues in Stoch chastic c Variational Inference ce

Typical optimization objective:
Notation:

KL# = KL %# & || ( & ) .

Optimization by stochastic gradient descent:

*+,- ← *+ − 0.01× 4#KL# |#5#6. argmin# KL# ≜ >?@ A log

?@ A D A|E

Issue 3: Wrong estimate Issue 2: Undefined 4#KL# Issue 1: Undefined KL#

20

SLIDE 21

Is Issue e 1: 1: Undefined ed KL#

KL# = %&' ( log

&' ( , (|.

= ∫ 01 2# 1 log

&' ( , (|.

KL# could be undefined for two reasons.

21

SLIDE 22

Is Issue e 1: 1: Undefined ed KL#

KL# = %&' ( log

&' ( , (|.

= ∫ 01 2# 1 log

&' ( , (|.

KL# could be undefined for two reasons. (a) Undefined integrand: 2# 1 ≠ 0 and 5 1 6 = 0 for some 1.

22

SLIDE 23

Is Issue e 1: 1: Undefined ed KL#

KL# = %&' ( log

&' ( , (|.

= ∫ 01 2# 1 log

&' ( , (|.

KL# could be undefined for two reasons. (a) Undefined integrand: 2# 1 ≠ 0 and 5 1 6 = 0 for some 1. (b) Undefined integral: ∫ 01 ⋯ is not integrable.

23

SLIDE 24

Is Issue e 1: 1: Examp mple

Example 2: Bayesian regression (from Pyro webpage).
KL# is undefined.
[Q] How to fix it?

def p(): # model_eg2 ... sigma = pyro.sample(“sigma”, Uniform(0., 10.)) ... pyro.sample(“obs”, Normal(..., sigma), obs=...) def qθ(): # guide_eg2 ... sigma = pyro.sample(“sigma”, Normal(θ, 0.05)) 24

SLIDE 25

Is Issue e 1: 1: Examp mple

Example 2: Bayesian regression (from Pyro webpage).
KL# is undefined.
[Q] How to fix it?

def p(): # model_eg2 ... sigma = pyro.sample(“sigma”, Uniform(0., 10.)) ... pyro.sample(“obs”, Normal(..., sigma), obs=...) def qθ(): # guide_eg2 ... sigma = pyro.sample(“sigma”, Normal(θ, 0.05)) 25

SLIDE 26

Is Issue e 1: 1: Examp mple

Example 2: Bayesian regression (from Pyro webpage).
KL# is undefined. [Q] How to fix it?

def p(): # model_eg2 ... sigma = pyro.sample(“sigma”, Uniform(0., 10.)) ... pyro.sample(“obs”, Normal(..., sigma), obs=...) def qθ(): # guide_eg2 ... sigma = pyro.sample(“sigma”, Normal(θ, 0.05)) 26

SLIDE 27

Is Issue e 1: 1: Examp mple

Example 2: Bayesian regression (from Pyro webpage).
KL# is undefined. Reason: (a) undefined integrand.
[Q] How to fix it?

def p(): # model_eg2 ... sigma = pyro.sample(“sigma”, Uniform(0., 10.)) ... pyro.sample(“obs”, Normal(..., sigma), obs=...) def qθ(): # guide_eg2 ... sigma = pyro.sample(“sigma”, Normal(θ, 0.05))

$# % ≠ 0 and ( % ) = 0 for % < 0.

27

SLIDE 28

Is Issue e 1: 1: Examp mple

Example 2: Bayesian regression (from Pyro webpage).
KL# is undefined. Reason: (a) undefined integrand.
[Q] How to fix model?

def p(): # model_eg2 ... sigma = pyro.sample(“sigma”, Uniform(0., 10.)) ... pyro.sample(“obs”, Normal(..., sigma), obs=...) def qθ(): # guide_eg2 ... sigma = pyro.sample(“sigma”, Normal(θ, 0.05)) 28

SLIDE 29

Is Issue e 1: 1: Examp mple

Example 2: Bayesian regression (from Pyro webpage).
[Q] How to fix model?

def p(): # model_eg2’ ... sigma = pyro.sample(“sigma”, Uniform(0., 10.)) ... pyro.sample(“obs”, Normal(..., sigma), obs=...) def qθ(): # guide_eg2 ... sigma = pyro.sample(“sigma”, Normal(θ, 0.05)) Normal(5., 5.) abs(sigma) 29

SLIDE 30

Is Issue e 1: 1: Examp mple

Example 2: Bayesian regression (from Pyro webpage).
KL# is still undefined.
[Q] How to fix model?

def p(): # model_eg2’ ... sigma = pyro.sample(“sigma”, Uniform(0., 10.)) ... pyro.sample(“obs”, Normal(..., sigma), obs=...) def qθ(): # guide_eg2 ... sigma = pyro.sample(“sigma”, Normal(θ, 0.05)) Normal(5., 5.) abs(sigma) 30

SLIDE 31

Is Issue e 1: 1: Examp mple

Example 2: Bayesian regression (from Pyro webpage).
KL# is still undefined. Reason: (b) undefined integral.
[Q] How to fix model?

def p(): # model_eg2’ ... sigma = pyro.sample(“sigma”, Uniform(0., 10.)) ... pyro.sample(“obs”, Normal(..., sigma), obs=...) def qθ(): # guide_eg2 ... sigma = pyro.sample(“sigma”, Normal(θ, 0.05)) Normal(5., 5.) abs(sigma)

$

%& &

1 ( ×* (; ,, 0.05 1( = ∞

31

SLIDE 32

Is Issue e 2: 2: Undefined ed !"KL"

KL" could be non-differentiable w.r.t. %.

32

SLIDE 33

Is Issue e 2: 2: Undefined ed !"KL"

KL" could be non-differentiable w.r.t. %.

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) def qθ(): # guide_eg1’ θ = pyro.param(“θ”, 0.) z = pyro.sample(“z”, Normal(θ, 1.)) Uniform(θ-1.,θ+1.) 33

SLIDE 34

Is Issue e 2: 2: Undefined ed !"KL"

KL" could be non-differentiable w.r.t. %.

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) def qθ(): # guide_eg1’ θ = pyro.param(“θ”, 0.) z = pyro.sample(“z”, Normal(θ, 1.)) Uniform(θ-1.,θ+1.)

KL" % not differentiable

34

SLIDE 35

Is Issue e 3: 3: Wrong g Estima mate e of !"KL"

Score Estimator:

!"KL" = !" log )" *+ × log

. /0

1 /0,3

where *+ is sampled from )".

Theorem:

!"KL" = 4/0[!"KL"] if some requirements are met.

35

SLIDE 36

Is Issue e 3: 3: Wrong g Estima mate e of !"KL"

Score Estimator:

!"KL" = !" log )" *+ × log

. /0

1 /0,3

where *+ is sampled from )".

Theorem:

!"KL" = 4/0[!"KL"] if some requirements are met.

36

SLIDE 37

Is Issue e 3: 3: Wrong g Estima mate e of !"KL"

Score Estimator:

!"KL" = !" log )" *+ × log

. /0

1 /0,3

where *+ is sampled from )".

Theorem:

!"KL" = 4/0[!"KL"] if some requirements are met. !"KL" could be an incorrect estimator if the requirements are not met.

37

SLIDE 38

Is Issue e 3: 3: Wrong g Estima mate e of !"KL"

Proof of Theorem:

!"KL" = !"∫ '" ( log

,- . / .|1

2( = ∫ !" '" ( log

,- . / .|1

2( ... = 3.4 !" log '" (5 × log

,- .4 / .4,1

= 3.4 !"KL" .

38

SLIDE 39

Is Issue e 3: 3: Wrong g Estima mate e of !"KL"

Proof of Theorem:

!"KL" = !"∫ '" ( log

,- . / .|1

2( = ∫ !" '" ( log

,- . / .|1

2( ... = 3.4 !" log '" (5 × log

,- .4 / .4,1

= 3.4 !"KL" . This might fail.

39

SLIDE 40

Is Issue e 3: 3: Wrong g Estima mate e of !"KL"

Proof of Theorem:

!"KL" = !"∫ '" ( log

,- . / .|1

2( = ∫ !" '" ( log

,- . / .|1

2( ... = 3.4 !" log '" (5 × log

,- .4 / .4,1

= 3.4 !"KL" . This might fail. !" 8

9(")

⋯ 2( ≠ 8

9(")

!" ⋯ 2(

40

SLIDE 41

Is Issue e 3: 3: Wrong g Estima mate e of !"KL"

Proof of Theorem:

!"KL" = !"∫ '" ( log

,- . / .|1

2( = ∫ !" '" ( log

,- . / .|1

2( ... = 3.4 !" log '" (5 × log

,- .4 / .4,1

= 3.4 !"KL" . This might fail. !" 8

9(")

⋯ 2( ≠ 8

9(")

!" ⋯ 2(

How to ch check ck that such ch issues do not happen?

41

SLIDE 42

Ho How to Ver erify y Is Issues es?

1. Undefined KL#.

(a) $# % ≠ 0 and ( % ) = 0 for some %. (b) ∫ ⋯ -% is not integrable.

2. Undefined .#KL#.
3. Wrong estimate of .#KL#.
.#∫ ⋯ -% ≠ ∫ .# ⋯ -%.

42

SLIDE 43

Ho How to Ver erify y Is Issues es?

1. Undefined KL#.

(a) $# % ≠ 0 and ( % ) = 0 for some %. (b) ∫ ⋯ -% is not integrable.

2. Undefined .#KL#.
3. Wrong estimate of .#KL#.
.#∫ ⋯ -% ≠ ∫ .# ⋯ -%.

43

SLIDE 44

Ho How to Ver erify y Is Issues es?

1. Undefined KL#.

(a) $# % ≠ 0 and ( % ) = 0 for some %. (b) ∫ ⋯ -% is not integrable.

2. Undefined .#KL#.
3. Wrong estimate of .#KL#.
.#∫ ⋯ -% ≠ ∫ .# ⋯ -%.

E.g., / %; _, _ > 0 for all % ∈ ℝ.

44

SLIDE 45

Ho How to Ver erify y Is Issues es?

1. Undefined KL#.

(a) $# % ≠ 0 and ( % ) = 0 for some %. (b) ∫ ⋯ -% is not integrable.

2. Undefined .#KL#.
3. Wrong estimate of .#KL#.
.#∫ ⋯ -% ≠ ∫ .# ⋯ -%.

E.g., / %; _, _ > 0 for all % ∈ ℝ. familiar type of static analysis problem

⇔

45

SLIDE 46

Ho How to Ver erify y Is Issues es?

1. Undefined KL#.

(a) $# % ≠ 0 and ( % ) = 0 for some %. (b) ∫ ⋯ -% is not integrable.

2. Undefined .#KL#.
3. Wrong estimate of .#KL#.
.#∫ ⋯ -% ≠ ∫ .# ⋯ -%.

46

SLIDE 47

Ho How to Ver erify y Is Issues es?

1. Undefined KL#.

(a) $# % ≠ 0 and ( % ) = 0 for some %. (b) ∫ ⋯ -% is not integrable.

2. Undefined .#KL#.
3. Wrong estimate of .#KL#.
.#∫ ⋯ -% ≠ ∫ .# ⋯ -%.

familiar type of static analysis problems

⇔

47

SLIDE 48

Ho How to Ver erify y Is Issues es?

1. Undefined KL#.

(a) $# % ≠ 0 and ( % ) = 0 for some %. (b) ∫ ⋯ -% is not integrable.

2. Undefined .#KL#.
3. Wrong estimate of .#KL#.
.#∫ ⋯ -% ≠ ∫ .# ⋯ -%.

Sufficient conditions about

differentiability of simpler func’s,
boundedness of simpler func’s.

¬ ¬ ¬

familiar type of static analysis problems

⟸ ⟸

48

SLIDE 49

Ho How to Ver erify y Is Issues es?

1. Undefined KL#.

(a) $# % ≠ 0 and ( % ) = 0 for some %. (b) ∫ ⋯ -% is not integrable.

2. Undefined .#KL#.
3. Wrong estimate of .#KL#.
.#∫ ⋯ -% ≠ ∫ .# ⋯ -%.

Sufficient conditions about

differentiability of simpler func’s,
boundedness of simpler func’s.

¬ ¬ ¬

familiar type of static analysis problems

⟸ ⟸

⇔

49

SLIDE 50

Ho How to Ver erify y Is Issues es?

1. Undefined KL#.

(a) $# % ≠ 0 and ( % ) = 0 for some %. (b) ∫ ⋯ -% is not integrable.

2. Undefined .#KL#.
3. Wrong estimate of .#KL#.
.#∫ ⋯ -% ≠ ∫ .# ⋯ -%.

Sufficient conditions about

differentiability of simpler func’s,
boundedness of simpler func’s.

¬ ¬ ¬

familiar type of static analysis problems

⟸ ⟸

⇔

Density semantics is used for formalization.

50

SLIDE 51

Useful in pract ctice ce? Our automatic c verifier.

51

SLIDE 52

Our Automatic c Verifier

Analyzes models/guides written in Pyro.
Verifies that issue 1a (below) does not happen:

!" # ≠ 0 and & # ' = 0 for some #.

52

SLIDE 53

Our Automatic c Verifier

Analyzes models/guides written in Pyro.
Verifies that issue 1a (below) does not happen:

!" # ≠ 0 and & # ' = 0 for some #.

Handles many (but not all) features of Python/PyTorch/Pyro

(e.g., tensor broadcasting). github.com/wonyeol/static-analysis-for-support-match

53

SLIDE 54

Ex Experimental Evaluation

Analyzed 8 representative examples (from Pyro webpage).

54

SLIDE 55

Ex Experimental Evaluation

55

SLIDE 56

Ex Experimental Evaluation

Example 2 ! uses Uniform. "# uses Normal.

56

SLIDE 57

Ex Experimental Evaluation

Example 2 ! uses Uniform. "# uses Normal. ! uses Dirichlet. "# uses Delta.

57

SLIDE 58

Ex Experimental Evaluation

! uses Uniform. "# uses Normal. ! uses Dirichlet. "# uses Delta. Performs different inference algorithm, using variational inference engine. Example 2

58

SLIDE 59

Ke Key Messages

ML algorithms often make nontrivial assumptions implicitly.

They could be violated sometimes; need to be checked carefully.

59

SLIDE 60

Ke Key Messages

ML algorithms often make nontrivial assumptions implicitly.

They could be violated sometimes; need to be checked carefully.

This gives opportunities for PL research.

E.g., how to check these assumptions automatically?

60

SLIDE 61

Ke Key Messages

ML algorithms often make nontrivial assumptions implicitly.

They could be violated sometimes; need to be checked carefully.

This gives opportunities for PL research.

E.g., how to check these assumptions automatically?

Th Thank k you!

61

SLIDE 62

Is Issue e 3: 3: Examp mple

Example 1’:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) def qθ(): # guide_eg1’ θ = pyro.param(“θ”, 0.) z = pyro.sample(“z”, Uniform(θ-1., θ+1.))

KL# $

%#KL# ≠ 0 for all $ %#KL# = 0 for all $, *′

SLIDE 63

Is Issue e 3: 3: Examp mple

Example 1’:

def p(): # model_eg1 z = pyro.sample(“z”, Normal(0., 5.)) if (z > 0): pyro.sample(“x”, Normal( 1., 1.), obs=0.) else: pyro.sample(“x”, Normal(-2., 1.), obs=0.) def qθ(): # guide_eg1’ θ = pyro.param(“θ”, 0.) z = pyro.sample(“z”, Uniform(θ-1., θ+1.))

KL# $

%#KL# ≠ 0 for all $ %#KL# = 0 for all $, *′ %# ,

#-. #/.

⋯ 1* ≠ ,

#-. #/.

%# ⋯ 1*

SLIDE 64

Suffici cient Conditions

Assume: !(#, %) and '((#) use only normal distributions.

), * : mean, standard deviation in !(#, %). )′, *′: mean, standard deviation in '((#).

Theorem: The following ensure non-occurrence of the issues.

§ ) # ≤ exp(0( # )) for some affine 0. § exp(1( # )) ≤ * # ≤ exp(ℎ( # )) for some affine 1, ℎ. § )3, *3 are continuously differentiable w.r.t. 4.

SLIDE 65

Suffici cient Conditions

Assume: !(#, %) and '((#) use only normal distributions.

), * : mean, standard deviation in !(#, %). )′, *′: mean, standard deviation in '((#).

Theorem: The following ensure non-occurrence of the issues.

§ ) # ≤ exp(0( # )) for some affine 0. § exp(1( # )) ≤ * # ≤ exp(ℎ( # )) for some affine 1, ℎ. § )3, *3 are continuously differentiable w.r.t. 4.

1b. ∫ ⋯ 7# is not integrable.
2. ∫ ⋯ 7# is not differentiable.
3. 8(∫ ⋯ 7# ≠ ∫ 8( ⋯ 7#.

:

;< <

exp(0 # )×> #; ⋯ 7# < ∞ for any affine 0.

SLIDE 66

Suffici cient Conditions

Assume: !(#, %) and '((#) use only normal distributions.

), * : mean, standard deviation in !(#, %). )′, *′: mean, standard deviation in '((#).

Theorem: The following ensure non-occurrence of the issues.

§ ) # ≤ exp(0( # )) for some affine 0. § exp(1( # )) ≤ * # ≤ exp(ℎ( # )) for some affine 1, ℎ. § )3, *3 are continuously differentiable w.r.t. 4.

1b. ∫ ⋯ 7# is not integrable.
2. ∫ ⋯ 7# is not differentiable.
3. 8(∫ ⋯ 7# ≠ ∫ 8( ⋯ 7#.

well-behaved function of 4, #