autodiff reverse mode
play

AutoDiff: Reverse Mode v 0 v 5 v 2 v 3 ln x 1 v 0 v 5 v 2 v 1 - PowerPoint PPT Presentation

x 1 AutoDiff: Reverse Mode v 0 v 5 v 2 v 3 ln x 1 v 0 v 5 v 2 v 1 + x 2 v 4 y v 6 + v 3 v 1 Traverse the original graph in the reverse topological v 4 x 2 y v 6 sin order and for each node in the original


  1. x 1 ¯ ¯ ¯ AutoDiff: Reverse Mode v 0 v 5 v 2 ¯ v 3 ln x 1 v 0 v 5 v 2 ¯ v 1 ¯ + x 2 v 4 ¯ y v 6 + v 3 − v 1 Traverse the original graph in the reverse topological v 4 x 2 y v 6 sin order and for each node in the original graph Forwar ard Eval valuat ation Trace: introduce an ad adjo join int node node , which computes f (2 , 5) derivative of the output with respect to the local node (using Chain rule): 2 v 0 = x 1 v 1 = x 2 5 ln(2) = 0.693 v 2 = ln( v 0 ) 2 x 5 = 10 v 3 = v 0 · v 1 sin(5) = -0.959 v 4 = sin ( v 1 ) v 5 = v 2 + v 3 0.693 + 10 = 10.693 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 "local cal" derivative y = v 6 11.652 104

  2. AutoDiff: Reverse Mode ln x 1 v 0 v 5 v 2 + + v 3 Backwards Derivative Backw ackwar ards Derivat vative ve Trace: − v 1 v 4 x 2 y v 6 sin ∂ v 3 ∂ v 2 1 v 0 = ¯ ¯ + ¯ = ¯ v 3 v 1 + ¯ 5.5 v 3 v 2 v 2 Forwar ard Eval valuat ation Trace: ∂ v 0 ∂ v 0 v 0 ∂ v 1 ∂ v 0 ∂ v 3 ∂ v 4 f (2 , 5) = ¯ v 3 v 0 + ¯ v 4 cos ( v 1 ) 1.716 v 1 = ¯ ¯ 1 = ¯ + ¯ v 3 v 4 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v ∂ v 5 2 v 0 = x 1 = ¯ v 5 · (1) 1x1 = 1 ¯ v 2 = ¯ v 5 ∂ v 2 v 1 = x 2 5 ∂ v 2 ∂ v 5 ln(2) = 0.693 v 2 = ln( v 0 ) = ¯ v 5 · (1) 1x1 = 1 v 3 = ¯ ¯ v 5 ∂ v 3 ∂ v 3 2 x 5 = 10 v 3 = v 0 · v 1 ∂ v 6 = ¯ v 6 · ( − 1) sin(5) = -0.959 v 4 = sin ( v 1 ) 1x-1 = -1 ¯ v 4 = ¯ v 6 ∂ v 4 ∂ v 4 v 5 = v 2 + v 3 0.693 + 10 = 10.693 ∂ v 6 = ¯ v 6 · 1 ¯ v 5 = ¯ 1x1 = 1 v 6 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 ∂ v 5 ∂ v 5 ∂ y v 6 = ∂ y y = v 6 11.652 1 ¯ ∂ v 6 105

  3. AutoDiff: Reverse Mode ln x 1 v 0 v 5 v 2 + + v 3 Backwards Derivative Backw ackwar ards Derivat vative ve Trace: − v 1 v 4 x 2 y v 6 sin ∂ v 3 ∂ v 2 1 v 0 = ¯ ¯ + ¯ = ¯ v 3 v 1 + ¯ 5.5 v 3 v 2 v 2 Forwar ard Eval valuat ation Trace: ∂ v 0 ∂ v 0 v 0 ∂ v 1 ∂ v 0 ∂ v 3 ∂ v 4 f (2 , 5) = ¯ v 3 v 0 + ¯ v 4 cos ( v 1 ) 1.716 v 1 = ¯ ¯ 1 = ¯ + ¯ v 3 v 4 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v ∂ v 5 2 v 0 = x 1 = ¯ v 5 · (1) 1x1 = 1 ¯ v 2 = ¯ v 5 ∂ v 2 v 1 = x 2 5 ∂ v 2 ∂ v 5 ln(2) = 0.693 v 2 = ln( v 0 ) = ¯ v 5 · (1) 1x1 = 1 v 3 = ¯ ¯ v 5 ∂ v 3 ∂ v 3 2 x 5 = 10 v 3 = v 0 · v 1 ∂ v 6 = ¯ v 6 · ( − 1) sin(5) = -0.959 v 4 = sin ( v 1 ) 1x-1 = -1 ¯ v 4 = ¯ v 6 ∂ v 4 ∂ v 4 v 5 = v 2 + v 3 0.693 + 10 = 10.693 ∂ v 6 = ¯ v 6 · 1 ¯ v 5 = ¯ 1x1 = 1 v 6 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 ∂ v 5 ∂ v 5 ∂ y v 6 = ∂ y y = v 6 11.652 1 ¯ ∂ v 6 106

  4. AutoDiff: Reverse Mode ln x 1 v 0 v 5 v 2 + + v 3 Backwards Derivative Backw ackwar ards Derivat vative ve Trace: − v 1 v 4 x 2 y v 6 sin ∂ v 3 ∂ v 2 1 v 0 = ¯ ¯ + ¯ = ¯ v 3 v 1 + ¯ 5.5 v 3 v 2 v 2 Forwar ard Eval valuat ation Trace: ∂ v 0 ∂ v 0 v 0 ∂ v 1 ∂ v 0 ∂ v 3 ∂ v 4 f (2 , 5) = ¯ v 3 v 0 + ¯ v 4 cos ( v 1 ) 1.716 v 1 = ¯ ¯ 1 = ¯ + ¯ v 3 v 4 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v ∂ v 5 2 v 0 = x 1 = ¯ v 5 · (1) 1x1 = 1 v 2 = ¯ ¯ v 5 ∂ v 2 v 1 = x 2 5 ∂ v 2 ∂ v 5 ln(2) = 0.693 v 2 = ln( v 0 ) = ¯ v 5 · (1) 1x1 = 1 v 3 = ¯ ¯ v 5 ∂ v 3 ∂ v 3 2 x 5 = 10 v 3 = v 0 · v 1 ∂ v 6 = ¯ v 6 · ( − 1) sin(5) = -0.959 v 4 = sin ( v 1 ) 1x-1 = -1 ¯ v 4 = ¯ v 6 ∂ v 4 ∂ v 4 v 5 = v 2 + v 3 0.693 + 10 = 10.693 ∂ v 6 = ¯ v 6 · 1 ¯ v 5 = ¯ 1x1 = 1 v 6 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 ∂ v 5 ∂ v 5 ∂ y v 6 = ∂ y y = v 6 11.652 1 1 ¯ ∂ v 6 107

  5. AutoDiff: Reverse Mode ln x 1 v 0 v 5 v 2 + + v 3 Backwards Derivative Backw ackwar ards Derivat vative ve Trace: − v 1 v 4 x 2 y v 6 sin ∂ v 3 ∂ v 2 1 v 0 = ¯ ¯ + ¯ = ¯ v 3 v 1 + ¯ 5.5 v 3 v 2 v 2 Forwar ard Eval valuat ation Trace: ∂ v 0 ∂ v 0 v 0 ∂ v 1 ∂ v 0 ∂ v 3 ∂ v 4 f (2 , 5) = ¯ v 3 v 0 + ¯ v 4 cos ( v 1 ) 1.716 v 1 = ¯ ¯ 1 = ¯ + ¯ v 3 v 4 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v ∂ v 5 2 v 0 = x 1 = ¯ v 5 · (1) 1x1 = 1 v 2 = ¯ ¯ v 5 ∂ v 2 v 1 = x 2 5 ∂ v 2 ∂ v 5 ln(2) = 0.693 v 2 = ln( v 0 ) = ¯ v 5 · (1) 1x1 = 1 v 3 = ¯ ¯ v 5 ∂ v 3 ∂ v 3 2 x 5 = 10 v 3 = v 0 · v 1 ∂ v 6 = ¯ v 6 · ( − 1) sin(5) = -0.959 v 4 = sin ( v 1 ) 1x-1 = -1 ¯ v 4 = ¯ v 6 ∂ v 4 ∂ v 4 v 5 = v 2 + v 3 0.693 + 10 = 10.693 ∂ v 6 = ¯ v 6 · 1 ¯ v 5 = ¯ 1x1 = 1 v 6 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 ∂ v 5 ∂ v 5 ∂ y v 6 = ∂ y y = v 6 11.652 1 1 ¯ ∂ v 6 108

  6. AutoDiff: Reverse Mode ln x 1 v 0 v 5 v 2 + + v 3 Backwards Derivative Backw ackwar ards Derivat vative ve Trace: − v 1 v 4 x 2 y v 6 sin ∂ v 3 ∂ v 2 1 v 0 = ¯ ¯ + ¯ = ¯ v 3 v 1 + ¯ 5.5 v 3 v 2 v 2 Forwar ard Eval valuat ation Trace: ∂ v 0 ∂ v 0 v 0 ∂ v 1 ∂ v 0 ∂ v 3 ∂ v 4 f (2 , 5) = ¯ v 3 v 0 + ¯ v 4 cos ( v 1 ) 1.716 v 1 = ¯ ¯ 1 = ¯ + ¯ v 3 v 4 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v ∂ v 5 2 v 0 = x 1 = ¯ v 5 · (1) 1x1 = 1 v 2 = ¯ ¯ v 5 ∂ v 2 v 1 = x 2 5 ∂ v 2 ∂ v 5 ln(2) = 0.693 v 2 = ln( v 0 ) = ¯ v 5 · (1) 1x1 = 1 v 3 = ¯ ¯ v 5 ∂ v 3 ∂ v 3 2 x 5 = 10 v 3 = v 0 · v 1 ∂ v 6 = ¯ v 6 · ( − 1) sin(5) = -0.959 v 4 = sin ( v 1 ) 1x-1 = -1 ¯ v 4 = ¯ v 6 ∂ v 4 ∂ v 4 v 5 = v 2 + v 3 0.693 + 10 = 10.693 ∂ v 6 = ¯ v 6 · 1 ¯ v 5 = ¯ 1x1 = 1 v 6 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 ∂ v 5 ∂ v 5 ∂ y v 6 = ∂ y y = v 6 11.652 1 1 ¯ ∂ v 6 109

  7. AutoDiff: Reverse Mode ln x 1 v 0 v 5 v 2 + + v 3 Backwards Derivative Backw ackwar ards Derivat vative ve Trace: − v 1 v 4 x 2 y v 6 sin ∂ v 3 ∂ v 2 1 v 0 = ¯ ¯ + ¯ = ¯ v 3 v 1 + ¯ 5.5 v 3 v 2 v 2 Forwar ard Eval valuat ation Trace: ∂ v 0 ∂ v 0 v 0 ∂ v 1 ∂ v 0 ∂ v 3 ∂ v 4 f (2 , 5) = ¯ v 3 v 0 + ¯ v 4 cos ( v 1 ) 1.716 v 1 = ¯ ¯ 1 = ¯ + ¯ v 3 v 4 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v ∂ v 5 2 v 0 = x 1 = ¯ v 5 · (1) 1x1 = 1 v 2 = ¯ ¯ v 5 ∂ v 2 v 1 = x 2 5 ∂ v 2 ∂ v 5 ln(2) = 0.693 v 2 = ln( v 0 ) = ¯ v 5 · (1) 1x1 = 1 v 3 = ¯ ¯ v 5 ∂ v 3 ∂ v 3 2 x 5 = 10 v 3 = v 0 · v 1 ∂ v 6 = ¯ v 6 · ( − 1) sin(5) = -0.959 v 4 = sin ( v 1 ) 1x-1 = -1 ¯ v 4 = ¯ v 6 ∂ v 4 ∂ v 4 v 5 = v 2 + v 3 0.693 + 10 = 10.693 ∂ v 6 = ¯ v 6 · 1 ¯ v 5 = ¯ 1x1 = 1 v 6 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 ∂ v 5 ∂ v 5 ∂ y v 6 = ∂ y y = v 6 11.652 1 1 ¯ ∂ v 6 110

  8. AutoDiff: Reverse Mode ln x 1 v 0 v 5 v 2 + + v 3 Backwards Derivative Backw ackwar ards Derivat vative ve Trace: − v 1 v 4 x 2 y v 6 sin ∂ v 3 ∂ v 2 1 v 0 = ¯ ¯ + ¯ = ¯ v 3 v 1 + ¯ 5.5 v 3 v 2 v 2 Forwar ard Eval valuat ation Trace: ∂ v 0 ∂ v 0 v 0 ∂ v 1 ∂ v 0 ∂ v 3 ∂ v 4 f (2 , 5) = ¯ v 3 v 0 + ¯ v 4 cos ( v 1 ) 1.716 v 1 = ¯ ¯ 1 = ¯ + ¯ v 3 v 4 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v ∂ v 5 2 v 0 = x 1 = ¯ v 5 · (1) 1x1 = 1 v 2 = ¯ ¯ v 5 ∂ v 2 v 1 = x 2 5 ∂ v 2 ∂ v 5 ln(2) = 0.693 v 2 = ln( v 0 ) = ¯ v 5 · (1) 1x1 = 1 v 3 = ¯ ¯ v 5 ∂ v 3 ∂ v 3 2 x 5 = 10 v 3 = v 0 · v 1 ∂ v 6 = ¯ v 6 · ( − 1) sin(5) = -0.959 v 4 = sin ( v 1 ) 1x-1 = -1 ¯ v 4 = ¯ v 6 ∂ v 4 ∂ v 4 v 5 = v 2 + v 3 0.693 + 10 = 10.693 ∂ v 6 = ¯ v 6 · 1 1x1 = 1 v 5 = ¯ ¯ 1x1 = 1 v 6 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 ∂ v 5 ∂ v 5 ∂ y v 6 = ∂ y y = v 6 11.652 1 1 ¯ ∂ v 6 111

  9. AutoDiff: Reverse Mode ln x 1 v 0 v 5 v 2 + + v 3 Backwards Derivative Backw ackwar ards Derivat vative ve Trace: − v 1 v 4 x 2 y v 6 sin ∂ v 3 ∂ v 2 1 v 0 = ¯ ¯ + ¯ = ¯ v 3 v 1 + ¯ 5.5 v 3 v 2 v 2 Forwar ard Eval valuat ation Trace: ∂ v 0 ∂ v 0 v 0 ∂ v 1 ∂ v 0 ∂ v 3 ∂ v 4 f (2 , 5) = ¯ v 3 v 0 + ¯ v 4 cos ( v 1 ) 1.716 v 1 = ¯ ¯ 1 = ¯ + ¯ v 3 v 4 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v ∂ v 5 2 v 0 = x 1 = ¯ v 5 · (1) 1x1 = 1 v 2 = ¯ ¯ v 5 ∂ v 2 v 1 = x 2 5 ∂ v 2 ∂ v 5 ln(2) = 0.693 v 2 = ln( v 0 ) = ¯ v 5 · (1) 1x1 = 1 v 3 = ¯ ¯ v 5 ∂ v 3 ∂ v 3 2 x 5 = 10 v 3 = v 0 · v 1 ∂ v 6 = ¯ v 6 · ( − 1) sin(5) = -0.959 v 4 = sin ( v 1 ) 1x-1 = -1 ¯ v 4 = ¯ v 6 ∂ v 4 ∂ v 4 v 5 = v 2 + v 3 0.693 + 10 = 10.693 ∂ v 6 = ¯ v 6 · 1 1x1 = 1 v 5 = ¯ ¯ 1x1 = 1 v 6 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 ∂ v 5 ∂ v 5 ∂ y v 6 = ∂ y y = v 6 11.652 1 1 ¯ ∂ v 6 112

  10. AutoDiff: Reverse Mode ln x 1 v 0 v 5 v 2 + + v 3 Backwards Derivative Backw ackwar ards Derivat vative ve Trace: − v 1 v 4 x 2 y v 6 sin ∂ v 3 ∂ v 2 1 v 0 = ¯ ¯ + ¯ = ¯ v 3 v 1 + ¯ 5.5 v 3 v 2 v 2 Forwar ard Eval valuat ation Trace: ∂ v 0 ∂ v 0 v 0 ∂ v 1 ∂ v 0 ∂ v 3 ∂ v 4 f (2 , 5) = ¯ v 3 v 0 + ¯ v 4 cos ( v 1 ) 1.716 v 1 = ¯ ¯ 1 = ¯ + ¯ v 3 v 4 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v 1 ∂ v ∂ v 5 2 v 0 = x 1 = ¯ v 5 · (1) 1x1 = 1 v 2 = ¯ ¯ v 5 ∂ v 2 v 1 = x 2 5 ∂ v 2 ∂ v 5 ln(2) = 0.693 v 2 = ln( v 0 ) = ¯ v 5 · (1) 1x1 = 1 v 3 = ¯ ¯ v 5 ∂ v 3 ∂ v 3 2 x 5 = 10 v 3 = v 0 · v 1 ∂ v 6 = ¯ v 6 · ( − 1) sin(5) = -0.959 v 4 = sin ( v 1 ) 1x-1 = -1 ¯ v 4 = ¯ v 6 ∂ v 4 ∂ v 4 v 5 = v 2 + v 3 0.693 + 10 = 10.693 ∂ v 6 = ¯ v 6 · 1 1x1 = 1 v 5 = ¯ ¯ 1x1 = 1 v 6 v 6 = v 5 − v 4 10.693 + 0.959 = 11.652 ∂ v 5 ∂ v 5 ∂ y v 6 = ∂ y y = v 6 11.652 1 1 ¯ ∂ v 6 113

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend


More recommend