Myia: A Differentiable Language for Deep Learning Olivier Breuleux - PowerPoint PPT Presentation

Institut des algorithmes d’apprentissage de Montréal Myia: A Differentiable Language for Deep Learning Olivier Breuleux Computer Analyst, MILA Bart van Merriënboer (MILA, Google) Arnaud Bergeron (MILA)

Paradigm How deep learning and language design intersect Autodiff What it is. How it works Myia Our proposed solution

Paradigm How deep learning and language design intersect Autodiff What it is. How it works Myia Our proposed solution

Deep Learning (Elgammal & al., 2017)

Deep Learning Features + Composition + Learning Gradient descent (automatable) ? ? ? ? Data ? ? ? ? ? Initial structure Trained structure

Needs Goal: a language adapted to the needs of machine learning, past and future General purpose: Express complex compositions using control flow. Fast: Leverage parallelism and GPU to process millions of features. Portable: Serializable, support multiple hardware. Differentiable: Language support for gradient descent.

General purpose DL algorithms are increasingly complex …? Feedforward Recurrent Recursive (trivial) (loops) (recursion)

General purpose DL algorithms are increasingly complex More and more language features needed • Most existing frameworks are limited • • Awkward abstractions • No recursion High level abstraction increases productivity • • Focus on the algorithm over implementation details Effortless abstractions encourage their use •

Fast Scale to millions of parameters • Lots of parallel operations • • Matrix multiplication, map, reduce Can work with low precision (float32, 16, even 8 bits) • Leverage adapted hardware (GPU, TPU) • • Loop fusion • Automatic parallelization • Memory management

Portable Serializable models • • Code + parameters (data) Run on mobile and embedded systems • Seamless transfer from research to • production Avoid being tied to an ecosystem (e.g. • Python/numpy) e.g. ONNX (but more general) •

Differentiable

Paradigm How deep learning and language design intersect Autodiff What it is. How it works Myia Our proposed solution

<latexit sha1_base64="rwitAWd0MVf2KHbXqGku4nti/I=">ACWHicbVFbS8MwGE3rZRdvcz76EhyiIo5OBPVBGPji4xTrhLWMNE23sDQpSaobpX/SB0H8Kz6YbV0+kHg5JzvfElOgoRpR3nzbKXldWS+VKdW19Y3Ortl1/UCKVmLhYMCEfA6QIo5y4mpGHhNJUBw0g1G1O9+0SkoLf60lC/BgNOI0oRtpQ/VriRLhLMyzcJxHh+MjeAULKpxZh8dFDyjcT/zSKIoEx6kg6GkpnqGTFy4zAh7Dr54jeAKnQ/NvV96vNZymMyv4F7QK0ABFdfq1Fy8UOI0J15ghpXotJ9F+hqSmJG86qWKJAiP0ID0DOQoJsrPZsnkcN8wIYyENItrOGN/OjIUKzWJA9MZIz1Ui9qU/E/rpTq68DPKk1QTjucHRSmDWsBpzDCkmDNJgYgLKm5K8RDZBLS5jOqJoTW4pP/Ave0edl0bs8a7bsijTLYBXvgELTAOWiDG9ABLsDgFXxYK9aq9W5bdsmuzFtq/DsgF9l1z8BwuCy8A=</latexit> <latexit sha1_base64="rwitAWd0MVf2KHbXqGku4nti/I=">ACWHicbVFbS8MwGE3rZRdvcz76EhyiIo5OBPVBGPji4xTrhLWMNE23sDQpSaobpX/SB0H8Kz6YbV0+kHg5JzvfElOgoRpR3nzbKXldWS+VKdW19Y3Ortl1/UCKVmLhYMCEfA6QIo5y4mpGHhNJUBw0g1G1O9+0SkoLf60lC/BgNOI0oRtpQ/VriRLhLMyzcJxHh+MjeAULKpxZh8dFDyjcT/zSKIoEx6kg6GkpnqGTFy4zAh7Dr54jeAKnQ/NvV96vNZymMyv4F7QK0ABFdfq1Fy8UOI0J15ghpXotJ9F+hqSmJG86qWKJAiP0ID0DOQoJsrPZsnkcN8wIYyENItrOGN/OjIUKzWJA9MZIz1Ui9qU/E/rpTq68DPKk1QTjucHRSmDWsBpzDCkmDNJgYgLKm5K8RDZBLS5jOqJoTW4pP/Ave0edl0bs8a7bsijTLYBXvgELTAOWiDG9ABLsDgFXxYK9aq9W5bdsmuzFtq/DsgF9l1z8BwuCy8A=</latexit> <latexit sha1_base64="rwitAWd0MVf2KHbXqGku4nti/I=">ACWHicbVFbS8MwGE3rZRdvcz76EhyiIo5OBPVBGPji4xTrhLWMNE23sDQpSaobpX/SB0H8Kz6YbV0+kHg5JzvfElOgoRpR3nzbKXldWS+VKdW19Y3Ortl1/UCKVmLhYMCEfA6QIo5y4mpGHhNJUBw0g1G1O9+0SkoLf60lC/BgNOI0oRtpQ/VriRLhLMyzcJxHh+MjeAULKpxZh8dFDyjcT/zSKIoEx6kg6GkpnqGTFy4zAh7Dr54jeAKnQ/NvV96vNZymMyv4F7QK0ABFdfq1Fy8UOI0J15ghpXotJ9F+hqSmJG86qWKJAiP0ID0DOQoJsrPZsnkcN8wIYyENItrOGN/OjIUKzWJA9MZIz1Ui9qU/E/rpTq68DPKk1QTjucHRSmDWsBpzDCkmDNJgYgLKm5K8RDZBLS5jOqJoTW4pP/Ave0edl0bs8a7bsijTLYBXvgELTAOWiDG9ABLsDgFXxYK9aq9W5bdsmuzFtq/DsgF9l1z8BwuCy8A=</latexit> Derivative f ( x + ✏ ) − f ( x ) dxf ( x ) = d d f dx = f 0 ( x ) = lim ✏ ! 0 ✏

<latexit sha1_base64="rwitAWd0MVf2KHbXqGku4nti/I=">ACWHicbVFbS8MwGE3rZRdvcz76EhyiIo5OBPVBGPji4xTrhLWMNE23sDQpSaobpX/SB0H8Kz6YbV0+kHg5JzvfElOgoRpR3nzbKXldWS+VKdW19Y3Ortl1/UCKVmLhYMCEfA6QIo5y4mpGHhNJUBw0g1G1O9+0SkoLf60lC/BgNOI0oRtpQ/VriRLhLMyzcJxHh+MjeAULKpxZh8dFDyjcT/zSKIoEx6kg6GkpnqGTFy4zAh7Dr54jeAKnQ/NvV96vNZymMyv4F7QK0ABFdfq1Fy8UOI0J15ghpXotJ9F+hqSmJG86qWKJAiP0ID0DOQoJsrPZsnkcN8wIYyENItrOGN/OjIUKzWJA9MZIz1Ui9qU/E/rpTq68DPKk1QTjucHRSmDWsBpzDCkmDNJgYgLKm5K8RDZBLS5jOqJoTW4pP/Ave0edl0bs8a7bsijTLYBXvgELTAOWiDG9ABLsDgFXxYK9aq9W5bdsmuzFtq/DsgF9l1z8BwuCy8A=</latexit> <latexit sha1_base64="rwitAWd0MVf2KHbXqGku4nti/I=">ACWHicbVFbS8MwGE3rZRdvcz76EhyiIo5OBPVBGPji4xTrhLWMNE23sDQpSaobpX/SB0H8Kz6YbV0+kHg5JzvfElOgoRpR3nzbKXldWS+VKdW19Y3Ortl1/UCKVmLhYMCEfA6QIo5y4mpGHhNJUBw0g1G1O9+0SkoLf60lC/BgNOI0oRtpQ/VriRLhLMyzcJxHh+MjeAULKpxZh8dFDyjcT/zSKIoEx6kg6GkpnqGTFy4zAh7Dr54jeAKnQ/NvV96vNZymMyv4F7QK0ABFdfq1Fy8UOI0J15ghpXotJ9F+hqSmJG86qWKJAiP0ID0DOQoJsrPZsnkcN8wIYyENItrOGN/OjIUKzWJA9MZIz1Ui9qU/E/rpTq68DPKk1QTjucHRSmDWsBpzDCkmDNJgYgLKm5K8RDZBLS5jOqJoTW4pP/Ave0edl0bs8a7bsijTLYBXvgELTAOWiDG9ABLsDgFXxYK9aq9W5bdsmuzFtq/DsgF9l1z8BwuCy8A=</latexit> <latexit sha1_base64="rwitAWd0MVf2KHbXqGku4nti/I=">ACWHicbVFbS8MwGE3rZRdvcz76EhyiIo5OBPVBGPji4xTrhLWMNE23sDQpSaobpX/SB0H8Kz6YbV0+kHg5JzvfElOgoRpR3nzbKXldWS+VKdW19Y3Ortl1/UCKVmLhYMCEfA6QIo5y4mpGHhNJUBw0g1G1O9+0SkoLf60lC/BgNOI0oRtpQ/VriRLhLMyzcJxHh+MjeAULKpxZh8dFDyjcT/zSKIoEx6kg6GkpnqGTFy4zAh7Dr54jeAKnQ/NvV96vNZymMyv4F7QK0ABFdfq1Fy8UOI0J15ghpXotJ9F+hqSmJG86qWKJAiP0ID0DOQoJsrPZsnkcN8wIYyENItrOGN/OjIUKzWJA9MZIz1Ui9qU/E/rpTq68DPKk1QTjucHRSmDWsBpzDCkmDNJgYgLKm5K8RDZBLS5jOqJoTW4pP/Ave0edl0bs8a7bsijTLYBXvgELTAOWiDG9ABLsDgFXxYK9aq9W5bdsmuzFtq/DsgF9l1z8BwuCy8A=</latexit> Derivative f ( x + ✏ ) − f ( x ) dxf ( x ) = d d f dx = f 0 ( x ) = lim ✏ ! 0 ✏ Chain rule dxg ( f ( x )) = dg d d f d f dx Total derivative dxh ( f ( x ) , g ( x )) = ∂ h dx + ∂ h d d f d f ∂ f ∂ g dx