SAFE: Self Attentive Function Embedding for binary similarity 16th - - PowerPoint PPT Presentation

SLIDE 1

Introduction State of the art Solution overview Evaluation

SAFE: Self Attentive Function Embedding for binary similarity

16th Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA 2019) Luca Massarelli1, Giuseppe Antonio Di Luna2, Fabio Petroni3, Roberto Baldoni1, Leonardo Querzoni1,

1University of Rome ”La Sapienza” 2CINI, National Laboratory of Cyber Security 3Facebook AI Research

Gothenburg, June 20, 2019

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 1 / 25

SLIDE 2

Introduction State of the art Solution overview Evaluation

A world of interconnected devices

Intelligent devices enable new ”smart” production processes. More and more organization rely on them every day. Organizations do not develop their own devices, but mostly rely on commercial ones.

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 2 / 25

SLIDE 3

Introduction State of the art Solution overview Evaluation

The dark side

COTSa devices are provided as Black-Box with no access to their firmware’s source code. While improving production processes,

• rganizations have to trust devices manufacturers for assessing the

absence of vulnerabilities or backdoors.

aCommercial Off-the-Shelf Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 3 / 25

SLIDE 4

Introduction State of the art Solution overview Evaluation

“Trust is good, control is even better.”

Even for a COTS device it is still possible to analyze its binary firmware, but this process is time consuming and requires skilled personnel. There is a strong need of new tools that enable more efficient analysis of binary code. Natural Language Processing (NLP) techniques has proved to be powerful when applied to binary code.

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 4 / 25

SLIDE 5

Introduction State of the art Solution overview Evaluation

Embeddings

A common approach in NLP is to associate to an entity (e.g. a word, a sentence, a whole text ...) an embedding vector, i.e. a fixed size vector of real numbers that contains information

• n the entity it represents.

Defining a relation between entities we can build a model that can be able to represent entities with embedding preserving the chosen relation. ”binary” − →

• 0.846

0.332 ... 0.954

• ”binaries” −

→

• 0.844

0.334 ... 0.984

• Massarelli, Di Luna, Petroni, Querzoni, Baldoni

SAFE: Self Attentive Function Embedding 5 / 25

SLIDE 6

Introduction State of the art Solution overview Evaluation

Similarity Definition

Definition: Similar Functions Two binary functions are considered similar if they have been compiled from the same source code but possibly using different compilers, different optimizations and/or for different platforms.

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 6 / 25

SLIDE 7

Introduction State of the art Solution overview Evaluation

Similarity Definition

Definition: Similar Functions Two binary functions are considered similar if they have been compiled from the same source code but possibly using different compilers, different optimizations and/or for different platforms. How to compute similarity-preserving embeddings for a binary function?

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 6 / 25

SLIDE 8

Introduction State of the art Solution overview Evaluation

Related Work

Genius [CCS-16] Strand [PLDI-16] Embeddings No-Embeddings Bindiff [STICC-05] Cross platform

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 7 / 25

SLIDE 9

Introduction State of the art Solution overview Evaluation

Genius

GENIUS by 1 showed that a binary function can be represented with a similarity-preserving vector. That is, given two similar functions, their embedding vectors should be similar in terms of cosine similarity. Computing the cosine similarity of two vectors is extremely faster than comparing two graphs. The Binary-Similarity problem has been reduced to the computation of similarity-preserving function embeddings.

• 1Q. Feng, et al. Scalable graph-based bug search for firmware images. In CCS,

2016

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 8 / 25

SLIDE 10

Introduction State of the art Solution overview Evaluation

Related Work

Genius [CCS-16] Gemini [CCS-17] Strand [PLDI-16] Embeddings No-Embeddings Bindiff [STICC-05] Cross platform

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 9 / 25

SLIDE 11

Introduction State of the art Solution overview Evaluation

Gemini

GEMINI by Xiaojun et al.2 proposes a graph embedding deep neural network3 to produce an embedding vector of the annotated control flow graph (ACFG) of a function.

Addr_1: mov eax,10 Addr_2: dec eax Addr_3: mov [base+eax],0 Addr_4: jnz Addr_2 Addr_5: mov eax,ebx

Basic Block Annotation

x1 = (1.3, . . . , 3.1) x2 = (3.3, . . . , 1.1) x3 = (5.1, . . . , 1.2) Features Extraction

CFG ACFG

Structure2vec

~ f = (3.12, . . . , 5.31)

Embedding Model Learned Parameters

2Xiaojun Xu, et al. Neural network-based graph embedding for cross-platform

binary code similarity detection. In CCS, 2017.

3Dai Hanjun, et al. Discriminative embeddings of latent variable models for

structured data. In ICML 2016.

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 10 / 25

SLIDE 12

Introduction State of the art Solution overview Evaluation

Related Work

Genius [CCS-16] Gemini [CCS-17] Asm2Vec [SP-19] Unsupervised feature learning [BAR-19] Strand [PLDI-16] Embeddings No-Embeddings Bindiff [STICC-05] Stripped Binaries Inner eye [NDSS-18] Solves a subproblem Single platform Cross platform

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 11 / 25

SLIDE 13

Introduction State of the art Solution overview Evaluation

Unsupervised features learning

Massarelli et al. 4 proposed a modified version of Gemini where node’s features are learned during training stage.

i2v ~ ι1 = (0.32, . . . , 0.21) ~ ι2 = (0.12, . . . , 0.41)

<latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit><latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit><latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit><latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit>

~ ι3 = (0.22, . . . , 0.62)

<latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit><latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit><latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit><latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit>

~ ι4 = (0.50, . . . , 0.78)

<latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit><latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit><latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit><latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit>

~ ι5 = (0.58, . . . , 0.99)

<latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit><latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit><latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit><latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit>

x1 = (1.3, . . . , 3.1) Instruction Embeddings Aggregation x2 = (3.3, . . . , 1.1) x3 = (5.1, . . . , 1.2)

Features Extraction

ACFG

Structure2vec

~ f = (3.12, . . . , 5.31)

Embedding Model Learned Parameters Addr_1: mov eax,10 Addr_2: dec eax Addr_3: mov [base+eax],0 Addr_4: jnz Addr_2 Addr_5: mov eax,ebx CFG

• 4L. Massarelli et al. Investigating Graph Embedding Neural Networks with

Unsupervised Features Extraction for Binary Analysis, BAR 19

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 12 / 25

SLIDE 14

Introduction State of the art Solution overview Evaluation

Motivations of our work

Exploring efficient way to compute function embeddings Determine if embeddings can be used to afford other tasks

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 13 / 25

SLIDE 15

Introduction State of the art Solution overview Evaluation

Motivations of our work

Exploring efficient way to compute function embeddings Determine if embeddings can be used to afford other tasks Tasks: Binary Similarity: Given two binary functions detect if they have been compiled from the same source code. Function Retrieval: Given a target function retrieve all similar functions from a knowledge base. Semantic Classification: Given a binary function identify the semantic class of the function.

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 13 / 25

SLIDE 16

Introduction State of the art Solution overview Evaluation

Network architecture

i2v

Features Extraction

~ f = (3.12, . . . , 5.31)

Learned Parameters Self Attentive RNN Embedding Model

Addr_1: mov eax,10 Addr_2: dec eax Addr_3: mov [base+eax],0 Addr_4: jnz Addr_2 Addr_5: mov eax,ebx

Function

If

<latexit sha1_base64="KnejVNrQbmN/ruNaNm1+KP5KqXw=">AB+XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTykxIrWl045fzQ3RopsvwRtFDRIVq+hoJ/wTYuIGq0cyudna8SElDtv1pVZW19Y3qpu1re2d3b36/kHXhLEW2BGhCnXfA4NKBtghSQr7kUbwPYU9b3ad+71H1EaGwT3NI3R9mAZyIgVQJvWS21EySdNRvWE37QJ8mTglabAS7VH9azgORexjQEKBMQPHjshNQJMUCtPaMDYgZjBFAcZDcBH4yZF3JSfxAYo5BFqLhUvRPy9kYBvzNz3skf6MEsern4nzeIaXLpJjKIYsJA5IdIKiwOGaFl1gPysdRIBHly5DLgAjQoZYchMjEOCumlvXhLH6/TLpnTcduOnfnjdZV2UyVHbFjdsocdsFa7Ia1WYcJNmNP7Jm9WIn1ar1Z7z+jFavcOWR/YH18A1vrlEM=</latexit><latexit sha1_base64="KnejVNrQbmN/ruNaNm1+KP5KqXw=">AB+XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTykxIrWl045fzQ3RopsvwRtFDRIVq+hoJ/wTYuIGq0cyudna8SElDtv1pVZW19Y3qpu1re2d3b36/kHXhLEW2BGhCnXfA4NKBtghSQr7kUbwPYU9b3ad+71H1EaGwT3NI3R9mAZyIgVQJvWS21EySdNRvWE37QJ8mTglabAS7VH9azgORexjQEKBMQPHjshNQJMUCtPaMDYgZjBFAcZDcBH4yZF3JSfxAYo5BFqLhUvRPy9kYBvzNz3skf6MEsern4nzeIaXLpJjKIYsJA5IdIKiwOGaFl1gPysdRIBHly5DLgAjQoZYchMjEOCumlvXhLH6/TLpnTcduOnfnjdZV2UyVHbFjdsocdsFa7Ia1WYcJNmNP7Jm9WIn1ar1Z7z+jFavcOWR/YH18A1vrlEM=</latexit><latexit sha1_base64="KnejVNrQbmN/ruNaNm1+KP5KqXw=">AB+XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTykxIrWl045fzQ3RopsvwRtFDRIVq+hoJ/wTYuIGq0cyudna8SElDtv1pVZW19Y3qpu1re2d3b36/kHXhLEW2BGhCnXfA4NKBtghSQr7kUbwPYU9b3ad+71H1EaGwT3NI3R9mAZyIgVQJvWS21EySdNRvWE37QJ8mTglabAS7VH9azgORexjQEKBMQPHjshNQJMUCtPaMDYgZjBFAcZDcBH4yZF3JSfxAYo5BFqLhUvRPy9kYBvzNz3skf6MEsern4nzeIaXLpJjKIYsJA5IdIKiwOGaFl1gPysdRIBHly5DLgAjQoZYchMjEOCumlvXhLH6/TLpnTcduOnfnjdZV2UyVHbFjdsocdsFa7Ia1WYcJNmNP7Jm9WIn1ar1Z7z+jFavcOWR/YH18A1vrlEM=</latexit><latexit sha1_base64="KnejVNrQbmN/ruNaNm1+KP5KqXw=">AB+XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTykxIrWl045fzQ3RopsvwRtFDRIVq+hoJ/wTYuIGq0cyudna8SElDtv1pVZW19Y3qpu1re2d3b36/kHXhLEW2BGhCnXfA4NKBtghSQr7kUbwPYU9b3ad+71H1EaGwT3NI3R9mAZyIgVQJvWS21EySdNRvWE37QJ8mTglabAS7VH9azgORexjQEKBMQPHjshNQJMUCtPaMDYgZjBFAcZDcBH4yZF3JSfxAYo5BFqLhUvRPy9kYBvzNz3skf6MEsern4nzeIaXLpJjKIYsJA5IdIKiwOGaFl1gPysdRIBHly5DLgAjQoZYchMjEOCumlvXhLH6/TLpnTcduOnfnjdZV2UyVHbFjdsocdsFa7Ia1WYcJNmNP7Jm9WIn1ar1Z7z+jFavcOWR/YH18A1vrlEM=</latexit>

~ ι1 = (0.32, . . . , 0.21) ~ ι2 = (0.12, . . . , 0.41)

<latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit><latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit><latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit><latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit>

~ ι3 = (0.22, . . . , 0.62)

<latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit><latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit><latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit><latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit>

~ ι4 = (0.50, . . . , 0.78)

<latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit><latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit><latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit><latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit>

~ ι5 = (0.58, . . . , 0.99)

<latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit><latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit><latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit><latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit>

~ If

<latexit sha1_base64="p0rabJDT7lgam0m4Lb5lhZAMVM=">AB/XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTwkx4rOl045Xy27taRIsviK2ihokO0fAsF/4JtXEDCVKOZXe3s+JEUBm3706qsrK6tb1Q3a1vbO7t79f2DrgljzaHDQxnqvs8MSKGgwIl9CMNLPAl9Pzpde73ZqCNCNU9ziPwAjZRYiw4w0xyBzPgye0wGafpsN6wm3YBukyckjRIifaw/jUYhTwOQCGXzBjXsSP0EqZRcAlpbRAbiBifsgm4GVUsAOMlReSUnsSGYUgj0FRIWojweyNhgTHzwM8mA4YPZtHLxf8N8bxpZcIFcUIiueHUEgoDhmuRdYF0JHQgMjy5ECFopxphghaUMZ5JsZObWsD2fx+2XSPWs6dtO5O2+0rspmquSIHJNT4pAL0iI3pE06hJOQPJFn8mI9Wq/Wm/X+M1qxyp1D8gfWxzd8sZYF</latexit><latexit sha1_base64="p0rabJDT7lgam0m4Lb5lhZAMVM=">AB/XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTwkx4rOl045Xy27taRIsviK2ihokO0fAsF/4JtXEDCVKOZXe3s+JEUBm3706qsrK6tb1Q3a1vbO7t79f2DrgljzaHDQxnqvs8MSKGgwIl9CMNLPAl9Pzpde73ZqCNCNU9ziPwAjZRYiw4w0xyBzPgye0wGafpsN6wm3YBukyckjRIifaw/jUYhTwOQCGXzBjXsSP0EqZRcAlpbRAbiBifsgm4GVUsAOMlReSUnsSGYUgj0FRIWojweyNhgTHzwM8mA4YPZtHLxf8N8bxpZcIFcUIiueHUEgoDhmuRdYF0JHQgMjy5ECFopxphghaUMZ5JsZObWsD2fx+2XSPWs6dtO5O2+0rspmquSIHJNT4pAL0iI3pE06hJOQPJFn8mI9Wq/Wm/X+M1qxyp1D8gfWxzd8sZYF</latexit><latexit sha1_base64="p0rabJDT7lgam0m4Lb5lhZAMVM=">AB/XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTwkx4rOl045Xy27taRIsviK2ihokO0fAsF/4JtXEDCVKOZXe3s+JEUBm3706qsrK6tb1Q3a1vbO7t79f2DrgljzaHDQxnqvs8MSKGgwIl9CMNLPAl9Pzpde73ZqCNCNU9ziPwAjZRYiw4w0xyBzPgye0wGafpsN6wm3YBukyckjRIifaw/jUYhTwOQCGXzBjXsSP0EqZRcAlpbRAbiBifsgm4GVUsAOMlReSUnsSGYUgj0FRIWojweyNhgTHzwM8mA4YPZtHLxf8N8bxpZcIFcUIiueHUEgoDhmuRdYF0JHQgMjy5ECFopxphghaUMZ5JsZObWsD2fx+2XSPWs6dtO5O2+0rspmquSIHJNT4pAL0iI3pE06hJOQPJFn8mI9Wq/Wm/X+M1qxyp1D8gfWxzd8sZYF</latexit><latexit sha1_base64="p0rabJDT7lgam0m4Lb5lhZAMVM=">AB/XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTwkx4rOl045Xy27taRIsviK2ihokO0fAsF/4JtXEDCVKOZXe3s+JEUBm3706qsrK6tb1Q3a1vbO7t79f2DrgljzaHDQxnqvs8MSKGgwIl9CMNLPAl9Pzpde73ZqCNCNU9ziPwAjZRYiw4w0xyBzPgye0wGafpsN6wm3YBukyckjRIifaw/jUYhTwOQCGXzBjXsSP0EqZRcAlpbRAbiBifsgm4GVUsAOMlReSUnsSGYUgj0FRIWojweyNhgTHzwM8mA4YPZtHLxf8N8bxpZcIFcUIiueHUEgoDhmuRdYF0JHQgMjy5ECFopxphghaUMZ5JsZObWsD2fx+2XSPWs6dtO5O2+0rspmquSIHJNT4pAL0iI3pE06hJOQPJFn8mI9Wq/Wm/X+M1qxyp1D8gfWxzd8sZYF</latexit>

Each instruction is associated with an embedding vector using the i2v layer; Instructions are fed into a Self-Attentive RNN;

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 14 / 25

SLIDE 17

Introduction State of the art Solution overview Evaluation

Self Attention

Attention model are well studied by NLP community 5. Since they perform very well on different NLP tasks we try to use attention to analyze binary code.

~ ι1 = (0.32, . . . , 0.21) ~ ι2 = (0.12, . . . , 0.41)

<latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit><latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit><latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit><latexit sha1_base64="4Q4RKZijeuqjMQNX52oJ42bCw+w=">ACnicbVBNS8NAEN34WetX1KOX1SJUKCUpBb0IBS8eK9gPaELZbLft0k027E4KJfTsxb/ixYMiXv0F3vw3btsctPXBwO9GWbmBbHgGhzn21pb39jc2s7t5Hf39g8O7aPjpaJoqxBpZCqHRDNBI9YAzgI1o4VI2EgWCsY3c781pgpzWX0AJOY+SEZRLzPKQEjde0zb8xo6nEJpFuZ3hSdslspYU/0JOiSU6l1274JSdOfAqcTNSQBnqXfvL60mahCwCKojWHdeJwU+JAk4Fm+a9RLOY0BEZsI6hEQmZ9tP5K1N8YZQe7ktlKgI8V39PpCTUehIGpjMkMNTL3kz8z+sk0L/2Ux7FCbCILhb1E4FB4lkuMcVoyAmhCquLkV0yFRhIJL29CcJdfXiXNStk1Ad5XCzUniyOHTtE5KiIXaEaukN1EAUPaJn9IrerCfrxXq3Phata1Y2c4L+wPr8AXjvmBs=</latexit>

~ ι3 = (0.22, . . . , 0.62)

<latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit><latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit><latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit><latexit sha1_base64="iFlFixtfsWjeQ3XLX2US08Yt5M=">ACnicbVDLSgNBEJz1GeMr6tHLaBAihGV2FfUiBLx4jGAekA1hdjLRwdmdZaY3EJacvfgrXjwo4tUv8ObfOHkcNLGgoajqprsrTKQwQMi3s7C4tLymlvLr29sbm0XdnbrRqWa8RpTUulmSA2XIuY1ECB5M9GcRqHkjfDhauQ3+lwboeJbGCS8HdG7WPQEo2ClTuEg6HOWBUIB7ZwML0vE9f0yDmRXgSkT98w/7hSKxCVj4HniTUkRTVHtFL6CrmJpxGNgkhrT8kgC7YxqEzyYT5IDU8oe6B3vGVpTCNu2tn4lSE+skoX95S2FQMeq78nMhoZM4hC2xlRuDez3kj8z2ul0LtoZyJOUuAxmyzqpRKDwqNcFdozkAOLKFMC3srZvdUwY2vbwNwZt9eZ7UfdcjrndzWqyQaRw5tI8OUQl56BxV0DWqohpi6BE9o1f05jw5L8678zFpXCmM3voD5zPH4CsmCA=</latexit>

~ ι4 = (0.50, . . . , 0.78)

<latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit><latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit><latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit><latexit sha1_base64="kgb7B0ywC0XYA5rg3a1TuZdVjU=">ACnicbVA9SwNBEN3z2/h1amzGoQI4diTSGyEgI1lBGMCuRD2Nptkyd7tsTsXCEdqG/+KjYUitv4CO/+Nm5hCEx8MPN6bYWZemEhgJAvZ2l5ZXVtfWMzt7W9s7vn7h/cG5VqxmtMSaUbITVcipjXQIDkjURzGoWS18PB9cSvD7k2QsV3MEp4K6K9WHQFo2CltnscDnLAqGAtkvjqwLxLkgRB7KjwBSJV748a7t54pEp8CLxZySPZqi23c+go1ga8RiYpMY0fZJAK6MaBJN8nAtSwxPKBrTHm5bGNOKmlU1fGeNTq3RwV2lbMeCp+nsio5Exoyi0nRGFvpn3JuJ/XjOF7mUrE3GSAo/Zz6JuKjEoPMkFd4TmDOTIEsq0sLdi1qeaMrDp5WwI/vzLi+T+3POJ59+W8hUyi2MDHaETVEA+KqMKukFVEMPaAn9IJenUfn2Xlz3n9al5zZzCH6A+fjG45+mCk=</latexit>

~ ι5 = (0.58, . . . , 0.99)

<latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit><latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit><latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit><latexit sha1_base64="9qG2NW+D/c+IvJsEtOZMe8P3wuQ=">ACnicbVA9SwNBEN2LXzF+RS1tVoOgIMeGEwKQbCxVDBRyIWwt9lLluzdHrtzgXCktvGv2FgoYusvsPfuEmu8OvBwO9GWbmBYkUBgj5dApz8wuLS8Xl0srq2vpGeXOraVSqGW8wJZW+C6jhUsS8AQIkv0s0p1Eg+W0wuJj4t0OujVDxDYwS3o5oLxahYBSs1Cnv+kPOMl8oJ3q+OyAuNXaEfZlV4E5Im69ftgpV4hLpsB/iZeTCspx1Sl/+F3F0ojHwCQ1puWRBNoZ1SCY5OSnxqeUDagPd6yNKYRN+1s+soY71uli0OlbcWAp+r3iYxGxoyiwHZGFPrmtzcR/NaKYS1dibiJAUes9miMJUYFJ7kgrtCcwZyZAlWthbMetTRnY9Eo2BO/3y39J89j1iOtdn1TOSR5HEe2gPXSAPHSKztElukINxNA9ekTP6MV5cJ6cV+dt1lpw8plt9APO+xehOJg1</latexit>

c c c c c Bi-directional Recurrent Neural Network

softmax(Ws2 · tanh(Ws1 · HT ))

<latexit sha1_base64="PTZjAR48nOWmawF0M7zP6rp0Wnw=">ACKHicbZA9TsNAEIX4T/8BShpVkRIpInsCAlKBE1KkAiJFIdovJmEFeu1tTtGRJYvwhE4BS1UdIiChpPgmBQmOrpezOamRfESlpy3XenNDe/sLi0vFJeXVvf2KxsbV/ZKDECWyJSkekEYFJjS2SpLATG4QwUNgObs8mfvsOjZWRvqRxjL0QRloOpQDKUb9ymPp2yG0pBDus4N2P7WNjPtiEBH3CfRNgbzsmzSv08usVutXqm7dLYr/Fd5UVNm0zvuVD38QiSRETUKBtV3PjamXgiEpFGZlP7EYg7iFEXZzqSFE20uL7zK+n1igiMdouFS8gPhzIoXQ2nEY5J0h0I2d9SbwP6+b0PC4l0odJ4RaTBaRVFgsLIPDbkA2mQCaXI5eaCzBAhEZyECKHSZ5jOc/Dm/3+r7hq1D237l0cVk9Op8ks12xw6Yx47YCWuyc9Zigj2wJ/bMXpxH59V5c96/W0vOdGaH/Srn8ws5QKYz</latexit><latexit sha1_base64="PTZjAR48nOWmawF0M7zP6rp0Wnw=">ACKHicbZA9TsNAEIX4T/8BShpVkRIpInsCAlKBE1KkAiJFIdovJmEFeu1tTtGRJYvwhE4BS1UdIiChpPgmBQmOrpezOamRfESlpy3XenNDe/sLi0vFJeXVvf2KxsbV/ZKDECWyJSkekEYFJjS2SpLATG4QwUNgObs8mfvsOjZWRvqRxjL0QRloOpQDKUb9ymPp2yG0pBDus4N2P7WNjPtiEBH3CfRNgbzsmzSv08usVutXqm7dLYr/Fd5UVNm0zvuVD38QiSRETUKBtV3PjamXgiEpFGZlP7EYg7iFEXZzqSFE20uL7zK+n1igiMdouFS8gPhzIoXQ2nEY5J0h0I2d9SbwP6+b0PC4l0odJ4RaTBaRVFgsLIPDbkA2mQCaXI5eaCzBAhEZyECKHSZ5jOc/Dm/3+r7hq1D237l0cVk9Op8ks12xw6Yx47YCWuyc9Zigj2wJ/bMXpxH59V5c96/W0vOdGaH/Srn8ws5QKYz</latexit><latexit sha1_base64="PTZjAR48nOWmawF0M7zP6rp0Wnw=">ACKHicbZA9TsNAEIX4T/8BShpVkRIpInsCAlKBE1KkAiJFIdovJmEFeu1tTtGRJYvwhE4BS1UdIiChpPgmBQmOrpezOamRfESlpy3XenNDe/sLi0vFJeXVvf2KxsbV/ZKDECWyJSkekEYFJjS2SpLATG4QwUNgObs8mfvsOjZWRvqRxjL0QRloOpQDKUb9ymPp2yG0pBDus4N2P7WNjPtiEBH3CfRNgbzsmzSv08usVutXqm7dLYr/Fd5UVNm0zvuVD38QiSRETUKBtV3PjamXgiEpFGZlP7EYg7iFEXZzqSFE20uL7zK+n1igiMdouFS8gPhzIoXQ2nEY5J0h0I2d9SbwP6+b0PC4l0odJ4RaTBaRVFgsLIPDbkA2mQCaXI5eaCzBAhEZyECKHSZ5jOc/Dm/3+r7hq1D237l0cVk9Op8ks12xw6Yx47YCWuyc9Zigj2wJ/bMXpxH59V5c96/W0vOdGaH/Srn8ws5QKYz</latexit><latexit sha1_base64="PTZjAR48nOWmawF0M7zP6rp0Wnw=">ACKHicbZA9TsNAEIX4T/8BShpVkRIpInsCAlKBE1KkAiJFIdovJmEFeu1tTtGRJYvwhE4BS1UdIiChpPgmBQmOrpezOamRfESlpy3XenNDe/sLi0vFJeXVvf2KxsbV/ZKDECWyJSkekEYFJjS2SpLATG4QwUNgObs8mfvsOjZWRvqRxjL0QRloOpQDKUb9ymPp2yG0pBDus4N2P7WNjPtiEBH3CfRNgbzsmzSv08usVutXqm7dLYr/Fd5UVNm0zvuVD38QiSRETUKBtV3PjamXgiEpFGZlP7EYg7iFEXZzqSFE20uL7zK+n1igiMdouFS8gPhzIoXQ2nEY5J0h0I2d9SbwP6+b0PC4l0odJ4RaTBaRVFgsLIPDbkA2mQCaXI5eaCzBAhEZyECKHSZ5jOc/Dm/3+r7hq1D237l0cVk9Op8ks12xw6Yx47YCWuyc9Zigj2wJ/bMXpxH59V5c96/W0vOdGaH/Srn8ws5QKYz</latexit>

Self-Attentive Mechanism

A

<latexit sha1_base64="iHrF6dnSxuKNzdLP9lxXsAcQ28I=">AB83icbVC7TsNAEFzDOEVoKQ5ESFRTZCgjJAQ5lI5CElVnS+bMIp54fu9pCiKF9ACxUdouWDKPgXbOMCEqYazexqZydIlDTkup/Oyura+sZmau8vbO7t185OGyb2GqBLRGrWHcDblDJCFskSWE30cjDQGEnmNxmfucRtZFxdE/TBP2QjyM5koJTKjWvB5WqW3NzsGXiFaQKBRqDyld/GAsbYkRCcWN6npuQP+OapFA4L/etwYSLCR9jL6URD9H4szonJ1awylmCWomFctF/L0x46Ex0zBIJ0NOD2bRy8T/vJ6l0ZU/k1FiCSORHSKpMD9khJZpA8iGUiMRz5IjkxETXHMi1JxIVLRpWU0z68xe+XSfu85rk1r3lRrd8UzZTgGE7gDy4hDrcQNaIADhCZ7hxbHOq/PmvP+MrjFzhH8gfPxDTDQkUo=</latexit><latexit sha1_base64="iHrF6dnSxuKNzdLP9lxXsAcQ28I=">AB83icbVC7TsNAEFzDOEVoKQ5ESFRTZCgjJAQ5lI5CElVnS+bMIp54fu9pCiKF9ACxUdouWDKPgXbOMCEqYazexqZydIlDTkup/Oyura+sZmau8vbO7t185OGyb2GqBLRGrWHcDblDJCFskSWE30cjDQGEnmNxmfucRtZFxdE/TBP2QjyM5koJTKjWvB5WqW3NzsGXiFaQKBRqDyld/GAsbYkRCcWN6npuQP+OapFA4L/etwYSLCR9jL6URD9H4szonJ1awylmCWomFctF/L0x46Ex0zBIJ0NOD2bRy8T/vJ6l0ZU/k1FiCSORHSKpMD9khJZpA8iGUiMRz5IjkxETXHMi1JxIVLRpWU0z68xe+XSfu85rk1r3lRrd8UzZTgGE7gDy4hDrcQNaIADhCZ7hxbHOq/PmvP+MrjFzhH8gfPxDTDQkUo=</latexit><latexit sha1_base64="iHrF6dnSxuKNzdLP9lxXsAcQ28I=">AB83icbVC7TsNAEFzDOEVoKQ5ESFRTZCgjJAQ5lI5CElVnS+bMIp54fu9pCiKF9ACxUdouWDKPgXbOMCEqYazexqZydIlDTkup/Oyura+sZmau8vbO7t185OGyb2GqBLRGrWHcDblDJCFskSWE30cjDQGEnmNxmfucRtZFxdE/TBP2QjyM5koJTKjWvB5WqW3NzsGXiFaQKBRqDyld/GAsbYkRCcWN6npuQP+OapFA4L/etwYSLCR9jL6URD9H4szonJ1awylmCWomFctF/L0x46Ex0zBIJ0NOD2bRy8T/vJ6l0ZU/k1FiCSORHSKpMD9khJZpA8iGUiMRz5IjkxETXHMi1JxIVLRpWU0z68xe+XSfu85rk1r3lRrd8UzZTgGE7gDy4hDrcQNaIADhCZ7hxbHOq/PmvP+MrjFzhH8gfPxDTDQkUo=</latexit><latexit sha1_base64="iHrF6dnSxuKNzdLP9lxXsAcQ28I=">AB83icbVC7TsNAEFzDOEVoKQ5ESFRTZCgjJAQ5lI5CElVnS+bMIp54fu9pCiKF9ACxUdouWDKPgXbOMCEqYazexqZydIlDTkup/Oyura+sZmau8vbO7t185OGyb2GqBLRGrWHcDblDJCFskSWE30cjDQGEnmNxmfucRtZFxdE/TBP2QjyM5koJTKjWvB5WqW3NzsGXiFaQKBRqDyld/GAsbYkRCcWN6npuQP+OapFA4L/etwYSLCR9jL6URD9H4szonJ1awylmCWomFctF/L0x46Ex0zBIJ0NOD2bRy8T/vJ6l0ZU/k1FiCSORHSKpMD9khJZpA8iGUiMRz5IjkxETXHMi1JxIVLRpWU0z68xe+XSfu85rk1r3lRrd8UzZTgGE7gDy4hDrcQNaIADhCZ7hxbHOq/PmvP+MrjFzhH8gfPxDTDQkUo=</latexit>

Attention Matrix .

M

<latexit sha1_base64="OknGFQkULsmJrlL5IoAvPJhkzY=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04kVIwDwgWcLspBOHzD6Y6RFCyBd41ZM38eoHefBf3F3oIl1Kq6eoKEiUNue6ns7K6tr6xWdoqb+/s7u1XDg7bJrZaYEvEKtbdgBtUMsIWSVLYTyMFDYCSY3md95RG1kHN3TNE/5ONIjqTglErNu0Gl6tbcHGyZeAWpQoHGoPLVH8bChiRUNyYnucm5M+4JikUzst9azDhYsLH2EtpxEM0/iwPOmen1nCKWYKaScVyEX9vzHhozDQM0smQ04NZ9DLxP69naXTlz2SUWMJIZIdIKswPGaFl2gCyodRIxLPkyGTEBNecCLVkXIhUtGkl5bQPb/H7ZdI+r3luzWteVOvXRTMlOIYTOAMPLqEOt9CAFghAeIJneHGs8+q8Oe8/oytOsXMEf+B8fANDhJFW</latexit><latexit sha1_base64="OknGFQkULsmJrlL5IoAvPJhkzY=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04kVIwDwgWcLspBOHzD6Y6RFCyBd41ZM38eoHefBf3F3oIl1Kq6eoKEiUNue6ns7K6tr6xWdoqb+/s7u1XDg7bJrZaYEvEKtbdgBtUMsIWSVLYTyMFDYCSY3md95RG1kHN3TNE/5ONIjqTglErNu0Gl6tbcHGyZeAWpQoHGoPLVH8bChiRUNyYnucm5M+4JikUzst9azDhYsLH2EtpxEM0/iwPOmen1nCKWYKaScVyEX9vzHhozDQM0smQ04NZ9DLxP69naXTlz2SUWMJIZIdIKswPGaFl2gCyodRIxLPkyGTEBNecCLVkXIhUtGkl5bQPb/H7ZdI+r3luzWteVOvXRTMlOIYTOAMPLqEOt9CAFghAeIJneHGs8+q8Oe8/oytOsXMEf+B8fANDhJFW</latexit><latexit sha1_base64="OknGFQkULsmJrlL5IoAvPJhkzY=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04kVIwDwgWcLspBOHzD6Y6RFCyBd41ZM38eoHefBf3F3oIl1Kq6eoKEiUNue6ns7K6tr6xWdoqb+/s7u1XDg7bJrZaYEvEKtbdgBtUMsIWSVLYTyMFDYCSY3md95RG1kHN3TNE/5ONIjqTglErNu0Gl6tbcHGyZeAWpQoHGoPLVH8bChiRUNyYnucm5M+4JikUzst9azDhYsLH2EtpxEM0/iwPOmen1nCKWYKaScVyEX9vzHhozDQM0smQ04NZ9DLxP69naXTlz2SUWMJIZIdIKswPGaFl2gCyodRIxLPkyGTEBNecCLVkXIhUtGkl5bQPb/H7ZdI+r3luzWteVOvXRTMlOIYTOAMPLqEOt9CAFghAeIJneHGs8+q8Oe8/oytOsXMEf+B8fANDhJFW</latexit><latexit sha1_base64="OknGFQkULsmJrlL5IoAvPJhkzY=">AB83icbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04kVIwDwgWcLspBOHzD6Y6RFCyBd41ZM38eoHefBf3F3oIl1Kq6eoKEiUNue6ns7K6tr6xWdoqb+/s7u1XDg7bJrZaYEvEKtbdgBtUMsIWSVLYTyMFDYCSY3md95RG1kHN3TNE/5ONIjqTglErNu0Gl6tbcHGyZeAWpQoHGoPLVH8bChiRUNyYnucm5M+4JikUzst9azDhYsLH2EtpxEM0/iwPOmen1nCKWYKaScVyEX9vzHhozDQM0smQ04NZ9DLxP69naXTlz2SUWMJIZIdIKswPGaFl2gCyodRIxLPkyGTEBNecCLVkXIhUtGkl5bQPb/H7ZdI+r3luzWteVOvXRTMlOIYTOAMPLqEOt9CAFghAeIJneHGs8+q8Oe8/oytOsXMEf+B8fANDhJFW</latexit>

Two-layers Fully connected Network with ReLU Activation

~ f

<latexit sha1_base64="RvqXpW0fbrQ19ogDth+NIGO290=">AB+XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNJRBIg8psaL1ZRNOZ+tu3WkyMpH0EJFh2j5Ggr+Bce4gISpRjO72tkJYiUtue6nU1pb39jcKm9Xdnb39g+qh0dtGyVGYEtEKjLdACwqbFkhR2Y4MQBgo7weR24XemaKyM9APNYvRDGs5kgIokzr9KYp0NB9Ua27dzcFXiVeQGivQHFS/+sNIJCFqEgqs7XluTH4KhqRQOK/0E4sxiAmMsZdRDSFaP83jzvlZYoEiHqPhUvFcxN8bKYTWzsIgmwyBHu2ytxD/83oJja79VOo4IdRicYikwvyQFUZmPSAfSoNEsEiOXGouwARGslBiExMsmIqWR/e8verpH1R9y6d39Za9wUzZTZCTtl58xjV6zB7liTtZhgE/bEntmLkzqvzpvz/jNacoqdY/YHzsc3UpuUPQ=</latexit><latexit sha1_base64="RvqXpW0fbrQ19ogDth+NIGO290=">AB+XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNJRBIg8psaL1ZRNOZ+tu3WkyMpH0EJFh2j5Ggr+Bce4gISpRjO72tkJYiUtue6nU1pb39jcKm9Xdnb39g+qh0dtGyVGYEtEKjLdACwqbFkhR2Y4MQBgo7weR24XemaKyM9APNYvRDGs5kgIokzr9KYp0NB9Ua27dzcFXiVeQGivQHFS/+sNIJCFqEgqs7XluTH4KhqRQOK/0E4sxiAmMsZdRDSFaP83jzvlZYoEiHqPhUvFcxN8bKYTWzsIgmwyBHu2ytxD/83oJja79VOo4IdRicYikwvyQFUZmPSAfSoNEsEiOXGouwARGslBiExMsmIqWR/e8verpH1R9y6d39Za9wUzZTZCTtl58xjV6zB7liTtZhgE/bEntmLkzqvzpvz/jNacoqdY/YHzsc3UpuUPQ=</latexit><latexit sha1_base64="RvqXpW0fbrQ19ogDth+NIGO290=">AB+XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNJRBIg8psaL1ZRNOZ+tu3WkyMpH0EJFh2j5Ggr+Bce4gISpRjO72tkJYiUtue6nU1pb39jcKm9Xdnb39g+qh0dtGyVGYEtEKjLdACwqbFkhR2Y4MQBgo7weR24XemaKyM9APNYvRDGs5kgIokzr9KYp0NB9Ua27dzcFXiVeQGivQHFS/+sNIJCFqEgqs7XluTH4KhqRQOK/0E4sxiAmMsZdRDSFaP83jzvlZYoEiHqPhUvFcxN8bKYTWzsIgmwyBHu2ytxD/83oJja79VOo4IdRicYikwvyQFUZmPSAfSoNEsEiOXGouwARGslBiExMsmIqWR/e8verpH1R9y6d39Za9wUzZTZCTtl58xjV6zB7liTtZhgE/bEntmLkzqvzpvz/jNacoqdY/YHzsc3UpuUPQ=</latexit><latexit sha1_base64="RvqXpW0fbrQ19ogDth+NIGO290=">AB+XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNJRBIg8psaL1ZRNOZ+tu3WkyMpH0EJFh2j5Ggr+Bce4gISpRjO72tkJYiUtue6nU1pb39jcKm9Xdnb39g+qh0dtGyVGYEtEKjLdACwqbFkhR2Y4MQBgo7weR24XemaKyM9APNYvRDGs5kgIokzr9KYp0NB9Ua27dzcFXiVeQGivQHFS/+sNIJCFqEgqs7XluTH4KhqRQOK/0E4sxiAmMsZdRDSFaP83jzvlZYoEiHqPhUvFcxN8bKYTWzsIgmwyBHu2ytxD/83oJja79VOo4IdRicYikwvyQFUZmPSAfSoNEsEiOXGouwARGslBiExMsmIqWR/e8verpH1R9y6d39Za9wUzZTZCTtl58xjV6zB7liTtZhgE/bEntmLkzqvzpvz/jNacoqdY/YHzsc3UpuUPQ=</latexit>

~ If

<latexit sha1_base64="p0rabJDT7lgam0m4Lb5lhZAMVM=">AB/XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTwkx4rOl045Xy27taRIsviK2ihokO0fAsF/4JtXEDCVKOZXe3s+JEUBm3706qsrK6tb1Q3a1vbO7t79f2DrgljzaHDQxnqvs8MSKGgwIl9CMNLPAl9Pzpde73ZqCNCNU9ziPwAjZRYiw4w0xyBzPgye0wGafpsN6wm3YBukyckjRIifaw/jUYhTwOQCGXzBjXsSP0EqZRcAlpbRAbiBifsgm4GVUsAOMlReSUnsSGYUgj0FRIWojweyNhgTHzwM8mA4YPZtHLxf8N8bxpZcIFcUIiueHUEgoDhmuRdYF0JHQgMjy5ECFopxphghaUMZ5JsZObWsD2fx+2XSPWs6dtO5O2+0rspmquSIHJNT4pAL0iI3pE06hJOQPJFn8mI9Wq/Wm/X+M1qxyp1D8gfWxzd8sZYF</latexit><latexit sha1_base64="p0rabJDT7lgam0m4Lb5lhZAMVM=">AB/XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTwkx4rOl045Xy27taRIsviK2ihokO0fAsF/4JtXEDCVKOZXe3s+JEUBm3706qsrK6tb1Q3a1vbO7t79f2DrgljzaHDQxnqvs8MSKGgwIl9CMNLPAl9Pzpde73ZqCNCNU9ziPwAjZRYiw4w0xyBzPgye0wGafpsN6wm3YBukyckjRIifaw/jUYhTwOQCGXzBjXsSP0EqZRcAlpbRAbiBifsgm4GVUsAOMlReSUnsSGYUgj0FRIWojweyNhgTHzwM8mA4YPZtHLxf8N8bxpZcIFcUIiueHUEgoDhmuRdYF0JHQgMjy5ECFopxphghaUMZ5JsZObWsD2fx+2XSPWs6dtO5O2+0rspmquSIHJNT4pAL0iI3pE06hJOQPJFn8mI9Wq/Wm/X+M1qxyp1D8gfWxzd8sZYF</latexit><latexit sha1_base64="p0rabJDT7lgam0m4Lb5lhZAMVM=">AB/XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTwkx4rOl045Xy27taRIsviK2ihokO0fAsF/4JtXEDCVKOZXe3s+JEUBm3706qsrK6tb1Q3a1vbO7t79f2DrgljzaHDQxnqvs8MSKGgwIl9CMNLPAl9Pzpde73ZqCNCNU9ziPwAjZRYiw4w0xyBzPgye0wGafpsN6wm3YBukyckjRIifaw/jUYhTwOQCGXzBjXsSP0EqZRcAlpbRAbiBifsgm4GVUsAOMlReSUnsSGYUgj0FRIWojweyNhgTHzwM8mA4YPZtHLxf8N8bxpZcIFcUIiueHUEgoDhmuRdYF0JHQgMjy5ECFopxphghaUMZ5JsZObWsD2fx+2XSPWs6dtO5O2+0rspmquSIHJNT4pAL0iI3pE06hJOQPJFn8mI9Wq/Wm/X+M1qxyp1D8gfWxzd8sZYF</latexit><latexit sha1_base64="p0rabJDT7lgam0m4Lb5lhZAMVM=">AB/XicbVC7TsNAEDyHVwivACXNiQiJKrIREpQRNAFiTwkx4rOl045Xy27taRIsviK2ihokO0fAsF/4JtXEDCVKOZXe3s+JEUBm3706qsrK6tb1Q3a1vbO7t79f2DrgljzaHDQxnqvs8MSKGgwIl9CMNLPAl9Pzpde73ZqCNCNU9ziPwAjZRYiw4w0xyBzPgye0wGafpsN6wm3YBukyckjRIifaw/jUYhTwOQCGXzBjXsSP0EqZRcAlpbRAbiBifsgm4GVUsAOMlReSUnsSGYUgj0FRIWojweyNhgTHzwM8mA4YPZtHLxf8N8bxpZcIFcUIiueHUEgoDhmuRdYF0JHQgMjy5ECFopxphghaUMZ5JsZObWsD2fx+2XSPWs6dtO5O2+0rspmquSIHJNT4pAL0iI3pE06hJOQPJFn8mI9Wq/Wm/X+M1qxyp1D8gfWxzd8sZYF</latexit>

c Vector Concatenation Unit: Matrix Multiplication Unit: . RNN Cell: RNN

<latexit sha1_base64="Q8yhW6zVTfG0xEXQG06gMbjze/o=">AB+3icbVC7TsNAEDzDOEVoKQ5ESFRTZCAlFoqGKAiIPlETR+bIJp9ydrbs1UmT5K2ihokO0fAwF/4JtXEDCVKOZXe3s+KEUFl301laXldWy9tlDe3tnd2K3v7bRtEhkOLBzIwXZ9ZkEJDCwVK6IYGmPIldPzpVeZ3HsFYEeg7nIUwUGyixVhwhql0H/ftmN42GsmwUnVrbg6SLyCVEmB5rDy1R8FPFKgkUtmbc9zQxzEzKDgEpJyP7IQMj5lE+ilVDMFdhDngRN6HFmGAQ3BUCFpLsLvjZgpa2fKTycVwc72Xif14vwvHFIBY6jBA0zw6hkJAfstyItAmgI2EAkWXJgQpNOTMEYygjPNUjNJqymkf3vz3i6R9WvPcmndzVq1fFs2UyCE5IifEI+ekTq5Jk7QIJ4o8kWfy4iTOq/PmvP+MLjnFzgH5A+fjGweElIw=</latexit><latexit sha1_base64="Q8yhW6zVTfG0xEXQG06gMbjze/o=">AB+3icbVC7TsNAEDzDOEVoKQ5ESFRTZCAlFoqGKAiIPlETR+bIJp9ydrbs1UmT5K2ihokO0fAwF/4JtXEDCVKOZXe3s+KEUFl301laXldWy9tlDe3tnd2K3v7bRtEhkOLBzIwXZ9ZkEJDCwVK6IYGmPIldPzpVeZ3HsFYEeg7nIUwUGyixVhwhql0H/ftmN42GsmwUnVrbg6SLyCVEmB5rDy1R8FPFKgkUtmbc9zQxzEzKDgEpJyP7IQMj5lE+ilVDMFdhDngRN6HFmGAQ3BUCFpLsLvjZgpa2fKTycVwc72Xif14vwvHFIBY6jBA0zw6hkJAfstyItAmgI2EAkWXJgQpNOTMEYygjPNUjNJqymkf3vz3i6R9WvPcmndzVq1fFs2UyCE5IifEI+ekTq5Jk7QIJ4o8kWfy4iTOq/PmvP+MLjnFzgH5A+fjGweElIw=</latexit><latexit sha1_base64="Q8yhW6zVTfG0xEXQG06gMbjze/o=">AB+3icbVC7TsNAEDzDOEVoKQ5ESFRTZCAlFoqGKAiIPlETR+bIJp9ydrbs1UmT5K2ihokO0fAwF/4JtXEDCVKOZXe3s+KEUFl301laXldWy9tlDe3tnd2K3v7bRtEhkOLBzIwXZ9ZkEJDCwVK6IYGmPIldPzpVeZ3HsFYEeg7nIUwUGyixVhwhql0H/ftmN42GsmwUnVrbg6SLyCVEmB5rDy1R8FPFKgkUtmbc9zQxzEzKDgEpJyP7IQMj5lE+ilVDMFdhDngRN6HFmGAQ3BUCFpLsLvjZgpa2fKTycVwc72Xif14vwvHFIBY6jBA0zw6hkJAfstyItAmgI2EAkWXJgQpNOTMEYygjPNUjNJqymkf3vz3i6R9WvPcmndzVq1fFs2UyCE5IifEI+ekTq5Jk7QIJ4o8kWfy4iTOq/PmvP+MLjnFzgH5A+fjGweElIw=</latexit><latexit sha1_base64="Q8yhW6zVTfG0xEXQG06gMbjze/o=">AB+3icbVC7TsNAEDzDOEVoKQ5ESFRTZCAlFoqGKAiIPlETR+bIJp9ydrbs1UmT5K2ihokO0fAwF/4JtXEDCVKOZXe3s+KEUFl301laXldWy9tlDe3tnd2K3v7bRtEhkOLBzIwXZ9ZkEJDCwVK6IYGmPIldPzpVeZ3HsFYEeg7nIUwUGyixVhwhql0H/ftmN42GsmwUnVrbg6SLyCVEmB5rDy1R8FPFKgkUtmbc9zQxzEzKDgEpJyP7IQMj5lE+ilVDMFdhDngRN6HFmGAQ3BUCFpLsLvjZgpa2fKTycVwc72Xif14vwvHFIBY6jBA0zw6hkJAfstyItAmgI2EAkWXJgQpNOTMEYygjPNUjNJqymkf3vz3i6R9WvPcmndzVq1fFs2UyCE5IifEI+ekTq5Jk7QIJ4o8kWfy4iTOq/PmvP+MLjnFzgH5A+fjGweElIw=</latexit>

− !

RNN

<latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit>

− !

RNN

<latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit>

− !

RNN

<latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit>

− !

RNN

<latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit>

− !

RNN

<latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit><latexit sha1_base64="sAaCGHPCrYmDmMojpjGcDdQTDs=">ACGHicbVC7TsNAEDzJrwClDQnAhJVZCMkEFUkGioEiCRIcRStj04cb6z7tYgZPkH+AS+ghYqOkRLR8G/YIcUvKYazexqdyZKlHTk+/e2PjE5NT0zGxlbn5hcam6vNJyJrUCm8IoY8jcKikxiZJUnieWIQ4UtiOrg5Kv32N1kmjz+g2wW4MAy37UgAVUq+6EZrSRspCZfTAysElgbXmJs+y0PX56dFRnveqNb/uD8H/kmBEamyE4171I7wIo1Rk1DgXCfwE+pmYEkKhXklTB0mIK5gJ2CaojRdbNhmpxvpg7I8AQtl4oPRfy+kUHs3G0cFZMx0KX7ZXif14npf5eN5M6SQm1KA+RVDg85ISVRQ/IL6RFIig/Ry41F2CBCK3kIEQhpkVvlaKP4Hf6v6S1XQ/8enCyU2vsj5qZYWtsnW2xgO2yBjtkx6zJBLtjD+yRPXn3rP34r1+jY5o51V9gPe2yeW3qF0</latexit>

← −

RNN

<latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit>

← −

RNN

<latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit>

← −

RNN

<latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit>

← −

RNN

<latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit>

← −

RNN

<latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit><latexit sha1_base64="spRMtupkQvNUSYmo+hx7/3HftIM=">ACF3icbVA9SwNBEN3z2/gVtbRZDIJVuBNBsQrYWImK0UAuhLl1Ehf3do/dOUWO+wH+BH+FrVZ2Ymtp4X/xLqbQxFc93nvDzLwoUdKR7396E5NT0zOzc/OVhcWl5ZXq6tqFM6kV2BRGduKwKGSGpskSWErsQhxpPAyujks/ctbtE4afU73CXZi6GvZkwKokLrVWmhKGykLldF9hT0Ca81dnmWh6/Gz4+M8L1J+3R+Aj5NgSGpsiJNu9Su8MiKNUZNQ4Fw78BPqZGBJCoV5JUwdJiBuoI/tgmqI0XWywTM530odkOEJWi4VH4j4eyKD2Ln7OCqSMdC1G/VK8T+vnVJv5NJnaSEWpSLSCocLHLCyqIG5FfSIhGUlyOXmguwQIRWchCiENOitkrRzD6/Ti52KkHfj043a01DobNzLENtsm2WcD2WIMdsRPWZI9sCf2zF68R+/Ve/Pef6IT3nBmnf2B9/ENru2g9w=</latexit>

   hT

1

. . . hT

5

   = H

<latexit sha1_base64="s5ZJupigJMSAyk/e/8DdquXQyAs=">ACYHicbVDLSsNAFJ3EV42vVne6GSyCq5KIohuh6KbLCq0KTSyT6bUOTiZh5kYsIT/pzoUbv8KlaQxaq2d1Oc+T5hIYdB1Xy17YXFpeaW26qytb2xu1Rvb1yZONYc+j2Wsb0NmQAoFfRQo4TbRwKJQwk34eDn1b5AGxGrHk4SCI2VuJecIaFNKxLP4SxUFkYMdTiOXfoDx7usl4+zLyc+v6s7j+NYjRzYlV8MjPBzX6Hnzeceiw3nRbgn6l3gVaZIK3WH9zR/FPI1AIZfMmIHnJhkTKPgEnLHTw0kjD+yMQwKqlgEJsjKVHJ6kBqGMU1AUyFpKcJsR8YiYyZRWFQWNz6YeW8q/ucNUrw/CzKhkhRB8ekiFBLKRYZrUcQNdCQ0ILp5UCFopxphghaUMZ5IaZF/k6Rhzf/V9yfdTy3JZ3dxsX1TJ1Mge2SeHxCOnpE06pEv6hJMX8mFZlm292zV7y258ldpW1bNDfsHe/Q2KzL</latexit><latexit sha1_base64="s5ZJupigJMSAyk/e/8DdquXQyAs=">ACYHicbVDLSsNAFJ3EV42vVne6GSyCq5KIohuh6KbLCq0KTSyT6bUOTiZh5kYsIT/pzoUbv8KlaQxaq2d1Oc+T5hIYdB1Xy17YXFpeaW26qytb2xu1Rvb1yZONYc+j2Wsb0NmQAoFfRQo4TbRwKJQwk34eDn1b5AGxGrHk4SCI2VuJecIaFNKxLP4SxUFkYMdTiOXfoDx7usl4+zLyc+v6s7j+NYjRzYlV8MjPBzX6Hnzeceiw3nRbgn6l3gVaZIK3WH9zR/FPI1AIZfMmIHnJhkTKPgEnLHTw0kjD+yMQwKqlgEJsjKVHJ6kBqGMU1AUyFpKcJsR8YiYyZRWFQWNz6YeW8q/ucNUrw/CzKhkhRB8ekiFBLKRYZrUcQNdCQ0ILp5UCFopxphghaUMZ5IaZF/k6Rhzf/V9yfdTy3JZ3dxsX1TJ1Mge2SeHxCOnpE06pEv6hJMX8mFZlm292zV7y258ldpW1bNDfsHe/Q2KzL</latexit><latexit sha1_base64="s5ZJupigJMSAyk/e/8DdquXQyAs=">ACYHicbVDLSsNAFJ3EV42vVne6GSyCq5KIohuh6KbLCq0KTSyT6bUOTiZh5kYsIT/pzoUbv8KlaQxaq2d1Oc+T5hIYdB1Xy17YXFpeaW26qytb2xu1Rvb1yZONYc+j2Wsb0NmQAoFfRQo4TbRwKJQwk34eDn1b5AGxGrHk4SCI2VuJecIaFNKxLP4SxUFkYMdTiOXfoDx7usl4+zLyc+v6s7j+NYjRzYlV8MjPBzX6Hnzeceiw3nRbgn6l3gVaZIK3WH9zR/FPI1AIZfMmIHnJhkTKPgEnLHTw0kjD+yMQwKqlgEJsjKVHJ6kBqGMU1AUyFpKcJsR8YiYyZRWFQWNz6YeW8q/ucNUrw/CzKhkhRB8ekiFBLKRYZrUcQNdCQ0ILp5UCFopxphghaUMZ5IaZF/k6Rhzf/V9yfdTy3JZ3dxsX1TJ1Mge2SeHxCOnpE06pEv6hJMX8mFZlm292zV7y258ldpW1bNDfsHe/Q2KzL</latexit><latexit sha1_base64="s5ZJupigJMSAyk/e/8DdquXQyAs=">ACYHicbVDLSsNAFJ3EV42vVne6GSyCq5KIohuh6KbLCq0KTSyT6bUOTiZh5kYsIT/pzoUbv8KlaQxaq2d1Oc+T5hIYdB1Xy17YXFpeaW26qytb2xu1Rvb1yZONYc+j2Wsb0NmQAoFfRQo4TbRwKJQwk34eDn1b5AGxGrHk4SCI2VuJecIaFNKxLP4SxUFkYMdTiOXfoDx7usl4+zLyc+v6s7j+NYjRzYlV8MjPBzX6Hnzeceiw3nRbgn6l3gVaZIK3WH9zR/FPI1AIZfMmIHnJhkTKPgEnLHTw0kjD+yMQwKqlgEJsjKVHJ6kBqGMU1AUyFpKcJsR8YiYyZRWFQWNz6YeW8q/ucNUrw/CzKhkhRB8ekiFBLKRYZrUcQNdCQ0ILp5UCFopxphghaUMZ5IaZF/k6Rhzf/V9yfdTy3JZ3dxsX1TJ1Mge2SeHxCOnpE06pEv6hJMX8mFZlm292zV7y258ldpW1bNDfsHe/Q2KzL</latexit>

5Lin, Zhouhan, et al. ”A structured self-attentive sentence embedding.” arXiv

preprint arXiv:1703.03130 (2017).

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 15 / 25

SLIDE 18

Introduction State of the art Solution overview Evaluation

Datasets

AMD64ARMOpenSSL Dataset: 95535 functions, Two versions of OpenSSL6 compiled for ARM and X86 using gcc. AMD64multipleCompilers Dataset: 452598 functions, Nine different open source projects7 compiled for X86 platform. AMD64PostgreSQL Dataset: 581640 functions, PostgreSQL 9.6.0 compiled with 12 different compilers8 for X86 platform. Semantic Dataset: 15158 functions, implementations of different 443 c functions that have been manually annotated as implementing algorithms in one of the 4 classes: Encryption, Sorting, String Manipulation, Mathematical.

6v1.0.1f - v1.0.1u 7binutils-2.30, ccv0.7, coreutils-8.29, curl- 7.61.0, gsl-2.5, libhttpd-2.0,

• penmpi-3.1.1, openssl- 1.1.1-pre8, valgrind-3.13.0

8gcc- 3.4, gcc-4.7, gcc-4.8, gcc-4.9, gcc-5.4, gcc-6, gcc-7, clang-3.8, clang-3.9,

clang-4.0, clang-5.0, clang-6.0

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 16 / 25

SLIDE 19

Introduction State of the art Solution overview Evaluation

Task 1: Binary Similarity

Evaluation of embedding quality using the Receiver Operating Characteristic (ROC) curve on the test set.

0.0 0.2 0.4 0.6 0.8 1.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate SAFE Gemini

AMD64ARMOpenSSL Dataset. SAFE AUC: 0.99, GEMINI AUC: 0.95

0.0 0.2 0.4 0.6 0.8 1.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate SAFE Gemini

AMD64multipleCompilers Dataset. SAFE AUC: 0.99, GEMINI AUC: 0.93

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 17 / 25

SLIDE 20

Introduction State of the art Solution overview Evaluation

Task 2: Functions Retrieval

We test SAFE single platform trained model as a function search engine: Given the embedding of a query function we want to retrieve all functions similar from a knowledge base. Evaluation metrics: Precision @ k: Precision over the first k retrieved functions. Recall @ k: Recall over the first k retrieved functions. Normalized Discounted Cumulative Gain: nDCG(R

f ) =

k

i=1 isSimilar(ri, f ) log(1+i)

IdealDCGk

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 18 / 25

SLIDE 21

Introduction State of the art Solution overview Evaluation

Task 2: Functions Retrieval

Varying k we evalutated the three metrics over the AMD64PostgreSQL Dataset.

10 20 30 40 50 Number of Nearest Results 0.0 0.2 0.4 0.6 0.8 1.0 precision

SAFE Gemini

Precision @ k

25 50 75 100 125 150 175 200 Number of Nearest Results 0.0 0.2 0.4 0.6 0.8 1.0 recall

SAFE Gemini

Recall @ k

10 20 30 40 50 Number of Nearest Results 0.0 0.2 0.4 0.6 0.8 1.0 normalized DCG

SAFE Gemini

nDCG @ k

SAFE GEMINI Precision @ 10 84% 74% Recall @ 10 21% 18% nDCG @ 10 85% 75%

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 19 / 25

SLIDE 22

Introduction State of the art Solution overview Evaluation

Task 3: Functions Semantic Classification

Using SAFE and GEMINI embeddings we try to classify the semantic of the functions in the Semantic Dataset. Embeddings are classified using a linear SVM in four semantic classes.

Confusion matrix with GEMINI embeddings Accuracy: 0.89 Confusion matrix with SAFE embeddings Accuracy: 0.95

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 20 / 25

SLIDE 23

Introduction State of the art Solution overview Evaluation

Task 3: Functions Semantic Classification

Projecting SAFE embeddings of Semantic Dataset functions it is possible to identify also semantic clusters.

(S) Sorting (E) Encryption (SM) String Manipulation (M) Math

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 21 / 25

SLIDE 24

Introduction State of the art Solution overview Evaluation

Main Takeaways

We show that using a Recurrent Neural Network with self attention permits to compute function embeddings with several advantages:

1 Precision, since SAFE embedding capture better similarity

between functions;

2 Efficiency, since it is not required the computation of the

CFG;

3 Semantic Similarity, since it is possible to capture the

semantic similarity between functions;

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 22 / 25

SLIDE 25

Introduction State of the art Solution overview Evaluation

Future Directions

We plan to continue this research following two different strategies: Increment SAFE performances: Recent works like Asm2Vec9 show that information on the CFG can be taken into accounts considering paths on the graph. Extensive study on semantic classification: We want to deeply study this problem defining a larger dataset and evaluating new training strategies and new model to tackle this tasks.

• 9S. H. Ding, B. C. Fung, and P. Charland, “Asm2vec: Boosting static

representation robustness for binary clone search against code obfuscation and compiler optimization,” in Proceedings of 40th Symposium on Security and Privacy, (SP), 2019.

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 23 / 25

SLIDE 26

Introduction State of the art Solution overview Evaluation

https://github.com/gadiluna/SAFE

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 24 / 25

SLIDE 27

Introduction State of the art Solution overview Evaluation

https://github.com/gadiluna/SAFE

Thanks for your attention!

Mail: massarelli@diag.uniroma1.it

Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 24 / 25

SLIDE 28

Introduction State of the art Solution overview Evaluation Massarelli, Di Luna, Petroni, Querzoni, Baldoni SAFE: Self Attentive Function Embedding 25 / 25