Verification of RNN-Based Neural Agent-Environment Systems Michael Akintunde , Andreea Kevorchian, Alessio Lomuscio, Edoardo Pirovano Imperial College London, UK VNN 2019, Stanford, California
This work We introduce Recurrent Neural Agent-Environment Systems to formalise RNN-based agents interacting with an environment with non-linear dynamics. We define and study various verification problems for these systems. We define two methods to solve said verification problems. We present an implementation and report experimental results. The paper builds upon work from previous work (KR’18)
Recurrent Neural Networks (RNNs) Many approaches already exist to perform verification on single FFNNs and closed-loop systems with FFNN-based agents. RNNs, equipped with a state that evolves over time, are designed to process sequences of data o t − 1 o t o t +1 o W ( h → o ) W ( h → o ) W ( h → o ) W ( h → o ) W ( h → h ) W ( h → h ) W ( h → h ) W ( h → h ) W ( h → h ) h Unroll h t − 1 h t h t +1 W ( i → h ) W ( i → h ) W ( i → h ) W ( i → h ) x t − 1 x t x t +1 x
Single-Layer Recurrent Neural Networks (RNNs) Definition A single-layer recurrent neural network (RNN) R with h hidden units and input size i and output size o is a neural network associated with the weight matrices W ( i → h ) ∈ R i × h , W ( h → h ) ∈ R h × h and W ( h → o ) ∈ R h × o , and the two activation functions σ : R h → R h and σ ′ : R o → R o . Here we assume the activation functions σ = σ ′ = ReLU.
Function Computed by an RNN Definition (Function computed by RNN) For an RNN R with weight matrices W ( i → h ) , W ( h → h ) and W ( h → o ) , let x ∈ ( R k ) n denote an input sequence of length n where each element of the sequence is a vector of size k , with x t denoting the t -th vector of x . We define h x 0 = 0 as a vector of 0 s. For each time step 1 ≤ t ≤ n , we define: h x t = σ ( W ( h → h ) h x t − 1 + W ( i → h ) x t ) . Then, the output of the RNN is given by f ( x ) = σ ′ ( W ( h → o ) h x n ) .
Recurrent Neural Agent-Environment Systems Definition (RNN-AES) A Recurrent Neural Agent-Environment System (RNN-AES) is a tuple AES = ( Ag, E, I ) where: Ag is a recurrent neural agent with action function act : O ∗ → Act , E = ( S, O, o, t E ) is an environment with state space S ⊆ R m , observation space O ⊆ R m ′ , observation function o : S → O and transition function t E : S × Act → S , I ⊆ S is a set of initial states. Paths are sequences of env state observations determined by the transition function t E from an initial state. We assume linearly definable AES (both t E and I ).
Bounded Specifications Definition (Specifications) For an environment with state space S ⊆ R m , we consider a fragment of LTL given by the following BNF: φ ::= X k C | C U ≤ k C C ::= C ∨ C | ( i ) op ( j ) | ( i ) op x where op ∈ { <, ≤ , = , � = , ≥ , > } , i, j ∈ { 1 , . . . , m } , x ∈ R , k ∈ N .
Satisfaction Satisfaction relation | = is defined as follows: Definition (Satisfaction) Given a path ρ ∈ Π on an RNN-AES and a formula φ : ρ | = ( i ) op ( j ) iff ρ (0) .i op ρ (0) .j holds; ρ | = C 1 ∨ C 2 iff ρ | = C 1 or ρ | = C 2 ; = X k C ρ | iff ρ ( k ) | = C ; = C 1 U ≤ k C 2 ρ | iff there is some i ≤ k such that ρ ( i ) | = C 2 and ρ ( j ) | = C 1 for all 0 ≤ j < i .
Verification problem We say that an agent-environment system AES satisfies a specification φ if it is the case that every path originating from an initial state i ∈ I satisfies φ , denoted AES | = φ . This is the basis of the verification problem: Definition (Verification problem) Determine if given an RNN-AES AES and a formula φ , it is the case that AES | = φ .
Approach: Unrolling RNNs to FFNNs Example : How to construct an FFNN from an RNN with input sequence of length 4, input size of 2, 3 hidden units and output size 1 (single output)? o W ( h → o ) W ( h → h ) W ( h → h ) W ( h → h ) W ( h → h ) h 0 h 1 h 2 h 3 h 4 W ( i → h ) W ( i → h ) W ( i → h ) W ( i → h ) x 1 x 2 x 3 x 4
Approach: Unrolling RNNs to FFNNs Input on Start (IOS) Scale input values according to the weights of W ( i → h ) . At each time step when the input is needed, pass it unchanged to the corresponding hidden layer of the FFNN. o x 11 x 12 x 21 x 22 x 31 x 32 x 41 x 42 FFNN constructed from RNN with length 4 input sequence, input size of 2, 3 hidden units and output size 1.
Approach: Unrolling RNNs to FFNNs Input on Demand (IOD) At the time step when the input term is needed, scale the input (on demand) and pass to the corresponding hidden layer of the FFNN, otherwise propogate the term’s original value. o x 11 x 12 x 21 x 22 x 31 x 32 x 41 x 42 FFNN constructed from RNN with length 4 input sequence, input size of 2, 3 hidden units and output size 1.
Equivalences Theorem For an RNN-AES AES and a specification φ k , = φ k iff IOD ( AES ) | = φ k iff IOS ( AES ) | = φ k . AES | Verification on bounded specifications of RNN-AES can be recast as FFNN-AES verification . See paper for further details of the unrolling methods. Verification for FFNN-AES addressed in KR’18 paper.
MILP Encoding for ReLU-FFNN Maganti & Lomuscio, 2017, Cheng, Nührenberg & Ruess, 2017 ReLU activation function � � x ( i ) 0 , W ( i ) j x ( i − 1) + b ( i ) j = 1 · · · | L ( i ) | = max , j j Active phase: x ( i ) = W ( i ) j x ( i − 1) + b ( i ) δ ( i ) (set ¯ = 0 ) j j j Inactive phase: x ( i ) δ ( i ) = 0 (set ¯ = 1 ) j j Value of ¯ δ j forces two of the four constraints to become vacuously true, and the other two correspond exactly to inactive/active phase of neuron: j x ( i − 1) + b ( i ) x ( i ) ≥ W ( i ) j j j x ( i − 1) + b ( i ) x ( i ) ≤ W ( i ) + M ¯ δ ( i ) j j j x ( i ) ≥ 0 j x ( i ) δ ( i ) ≤ M (1 − ¯ j ) j
Verifying RNN-AESs via MILP Theorem x (1) = ¯ x ( m ) = ¯ The MILP P FFNN is feasible for ¯ x, ¯ y iff f NN (¯ x ) = ¯ y . Verification problem can be solved via MILP by considering the linear programming problem defined on the unrolled RNN truncated by the bound on the spec. Theorem Verification of RNN-AESs against bounded specifications is coNP-complete.
Verification Procedure Environment E � �� � Goal : Take RNN-AES AES = ( Ag N , ( S, O, o, t E ) , I ) and a specification φ . Return whether φ is satisfied on the system. For X k C : For each step n from 0 → k , add constraints corresponding to the observation function, the unrolling of length n of the RNN and the transition function of the environment Check whether ¯ C can be satisfied in any of the states possible after k steps, and return result accordingly.
Verification Procedure For C 1 U ≤ k C 2 : For each n from 0 to k , check whether C 2 is always satisfied in valid paths of length n that have not already had C 2 satisfied earlier on. If so, return True . Otherwise, continue from states not satisfying C 2 . Check if not all of these satisfy C 1 . If so, return False . Otherwise, we’re on a valid path. Continue to add the constraints corresponding to the observation function, the unrolling of length n of the RNN and the transition function of the environment. Iterate to n + 1 . If reached n = k without a result returned, there must exist a path of length k along which C 2 is never satisfied, and so we return False .
RNSVerify Experimental toolkit produced, solving desired verification problems. Takes as input an RNN-AES, property φ and produces associated MILP problem. Fed to Gurobi 7.5.2 to solve. If output is False , counterexample in the form of a trace is shown.
Example: OpenAI Pendulum Brockman et. al, 2016 Example (Pendulum) OpenAI Gym task Pendulum-v0 : System composed of a pendulum and an agent which can apply a force to the pendulum. Agent can observe the current angle θ of the pendulum ( θ = 0 indicates that it is perfectly vertical) and the pendulum’s angular velocity ˙ θ . Agent chooses a small torque to be applied to the pendulum at each time step. Aim : Learn how to keep the pendulum upright by applying torque at each time step.
Example: OpenAI Pendulum Brockman et. al, 2016
Evaluation: OpenAI Pendulum Agent observes the angle and angular velocity and applies a torque to keep it vertical. Encoded as a RNN-AES: agent-environment system, non-linear transition function, and sequence of env state observations. Agent’s policy synthesised using Q-Learning on a ReLU-RNN. Env approximated from data (since env is non linear). RNSVerify found several bugs in the synthesised agent, e.g., the agent would apply the torque incorrectly in some situations.
Verification Results Input on Start – Evaluation on Pendulum [OpenAI, 2018] Check the property X n ( θ f > − ε ) for different values of n and ε using IOS. Fix ( θ i , ˙ θ i ) ∈ [0 , π/ 64] × [0 , 0 . 3] . ε π/ 10 π/ 30 π/ 50 π/ 70 1 0.056s 0.067s 0.011s 0.014s 2 0.052s 0.179s 0.138s 0.197s 3 0.372s 0.904s 5.794s 0.552s 4 2.578s 7.222s 0.378s 0.368s n 5 20.57s 31.07s 0.748s 0.663s 6 73.97s 3.264s 31.07s 23.99s 7 54.30s 96.54s 116.8s 207.8s 8 693.2s 294.9s 239.8s 243.3s Greyed areas denote False result, hence insufficiently trained system.
Recommend
More recommend
