Dissecting tf.function to discover AutoGraph strengths and - PowerPoint PPT Presentation

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties Dissecting tf.function to discover AutoGraph strengths and subtleties How to correctly write graph-convertible functions in TensorFlow 2.0. file:///home/paolo/projects/tf-function-talk/index.html#slide=1 1/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties About me Computer engineer | Head of ML & CV @ ZURU Tech Italy | Machine Learning GDE Blog: https://pgaleone.eu/ Github: https://github.com/galeone/ Twitter: @paolo_galeone file:///home/paolo/projects/tf-function-talk/index.html#slide=1 2/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties TensorFlow 2.0 & DataFlow Graphs Execution Speed Language Agnostic Representation Easy to replicate and distribute Automatic Differentiation file:///home/paolo/projects/tf-function-talk/index.html#slide=1 3/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties tf.function and AutoGraph # tf . function signature : it is a decorator . def function ( func = None , input _ signature = None , autograph = True , experimental _ autograph _ options = None ) tf.function uses AutoGraph AutoGraph converts Python control flow statements into appropriate TensorFlow graph ops. file:///home/paolo/projects/tf-function-talk/index.html#slide=1 4/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties tf.function and AutoGraph file:///home/paolo/projects/tf-function-talk/index.html#slide=1 5/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties The problem Given the constant matrices And the scalar variable Compute file:///home/paolo/projects/tf-function-talk/index.html#slide=1 6/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties TensorFlow 1.x solution g = tf . Graph () with g . as _ default (): a = tf . constant ([[10,10],[11.,1.]]) x = tf . constant ([[1.,0.],[0.,1.]]) b = tf . Variable (12.) y = tf . matmul ( a , x ) + b init _ op = tf . global _ variables _ initializer () with tf . Session () as sess : sess . run ( init _ op ) print ( sess . run ( y )) file:///home/paolo/projects/tf-function-talk/index.html#slide=1 7/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties TensorFlow 2.0 solution: eager execution def f (): a = tf . constant ([[10,10],[11.,1.]]) x = tf . constant ([[1.,0.],[0.,1.]]) b = tf . Variable (12.) y = tf . matmul ( a , x ) + b return y print ([ f (). numpy () for _ in range (10)]) Every tf.* op, produces a tf . Tensor object file:///home/paolo/projects/tf-function-talk/index.html#slide=1 8/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties From eager function to tf.function @ tf . function def f (): a = tf . constant ([[10,10],[11.,1.]]) x = tf . constant ([[1.,0.],[0.,1.]]) b = tf . Variable (12.) y = tf . matmul ( a , x ) + b print (" PRINT : ", y ) tf . print (" TF - PRINT : ", y ) return y f () file:///home/paolo/projects/tf-function-talk/index.html#slide=1 9/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties From eager function to tf.function PRINT : Tensor (" add :0", shape = (2, 2), dtype = float 32) file:///home/paolo/projects/tf-function-talk/index.html#slide=1 10/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties From eager function to tf.function ValueError : tf . function - decorated function tried to create variables on non - first call . file:///home/paolo/projects/tf-function-talk/index.html#slide=1 11/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties Lesson #1: functions that create a state A tf . Variable object in eager mode is just a Python object that gets destroyed as soon as it goes out of scope. A tf . Variable object in a tf.function-decorated function is the definition of a node in a persistent graph (eager execution disabled). file:///home/paolo/projects/tf-function-talk/index.html#slide=1 12/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties The solution class F (): def __ init __( self ): self ._ b = None @ tf . function def __ call __( self ): a = tf . constant ([[10, 10], [11., 1.]]) x = tf . constant ([[1., 0.], [0., 1.]]) if self ._ b is None : self ._ b = tf . Variable (12.) y = tf . matmul ( a , x ) + self ._ b print (" PRINT : ", y ) tf . print (" TF - PRINT : ", y ) return y f = F () f () file:///home/paolo/projects/tf-function-talk/index.html#slide=1 13/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties Lesson #2: eager function is not graph- convertible as is There is no 1:1 match between eager execution and the graph built by @ tf . function . Define the function thinking about the graph that is being built. file:///home/paolo/projects/tf-function-talk/index.html#slide=1 14/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties Change the input type Python is a dynamically-typed language TensorFlow is a strictly statically typed framework file:///home/paolo/projects/tf-function-talk/index.html#slide=1 15/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties The function @ tf . function def f ( x ): print (" Python execution : ", x ) tf . print (" Graph execution : ", x ) return x The function parameters type is used to create a graph, that is a statically typed object, and to assign it an ID file:///home/paolo/projects/tf-function-talk/index.html#slide=1 16/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties tf.Tensor as input print ("##### float 32 test #####") a = tf . constant (1, dtype = tf . float 32) print (" first call ") f ( a ) a = tf . constant (1.1, dtype = tf . float 32) print (" second call ") f ( a ) print ("##### uint 8 test #####") b = tf . constant (2, dtype = tf . uint 8) print (" first call ") f ( b ) b = tf . constant (3, dtype = tf . uint 8) print (" second call ") f ( b ) file:///home/paolo/projects/tf-function-talk/index.html#slide=1 17/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties tf.Tensor as input ##### float 32 test ##### first call Python execution : Tensor (" x :0", shape = (), dtype = float 32) Graph execution : 1 second call Graph execution : 1.1 ##### uint 8 test ##### first call Python execution : Tensor (" x :0", shape = (), dtype = uint 8) Graph execution : 2 second call Graph execution : 3 Everything goes as we expect file:///home/paolo/projects/tf-function-talk/index.html#slide=1 18/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties Inspecting the function tf . autograph . to _ code ( f . python _ function ) def tf __ f ( x ): try : with ag __. function _ scope (' f '): do _ return = False retval _ = None with ag __. utils . control _ dependency _ on _ returns ( ag __. converted _ call ( print , None , ag __. ConversionOptions ( re tf _1, x _1 = ag __. utils . alias _ tensors ( tf , x ) with ag __. utils . control _ dependency _ on _ returns ( ag __. converted _ call (' print ', tf _1, ag __. ConversionOption x _2 = ag __. utils . alias _ tensors ( x _1) do _ return = True retval _ = x _1 return retval _ except : ag __. rewrite _ graph _ construction _ error ( ag _ source _ map __) file:///home/paolo/projects/tf-function-talk/index.html#slide=1 19/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties Inspecting the function with ag __. utils . control _ dependency _ on _ returns ( ag __. converted _ call ( print , None , ag __. ConversionOptions ( recursive = True , force _ conversion = False , optional _ features = ag __. Feature . ALL , internal _ convert _ user _ code = True ), (' Python execution : ', x ), {}) ): converted _ call owner parameter is None : this line becomes a tf . no _ op () file:///home/paolo/projects/tf-function-talk/index.html#slide=1 20/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties Python native type as input def printinfo ( x ): print (" Type : ", type ( x ), " value : ", x ) print ("##### int test #####") print (" first call ") a = 1 printinfo ( a ) f ( a ) print (" second call ") b = 2 printinfo ( b ) f ( b ) print ("##### float test #####") print (" first call ") a = 1.0 printinfo ( a ) f ( a ) print (" second call ") b = 2.0 printinfo ( b ) f ( b ) file:///home/paolo/projects/tf-function-talk/index.html#slide=1 21/36

12/7/2019 Dissecting tf.function to discover AutoGraph strengths and subtleties Python native type as input Call with Python int ##### int test ##### first call Type : < class ' int ' > value : 1 Python execution : 1 Graph execution : 1 second call Type : < class ' int ' > value : 2 Python execution : 2 Graph execution : 2 The graph is being recreated at every function invocation file:///home/paolo/projects/tf-function-talk/index.html#slide=1 22/36