Neural Program Synthesis with Priority Queue Training Daniel A. - - PowerPoint PPT Presentation

Neural Program Synthesis with Priority Queue Training

Daniel A. Abolafia, Mohammad Norouzi, Jonathan Shen, Rui Zhao, Quoc V. Le

https://arxiv.org/abs/1801.03526

Why Program Synthesis?

• One of the hard AI reasoning domains
• A tool for planning in robotics
• Increased interpretability (human can read code more easily than NN weights)

Deep Reinforcement Learning

https://becominghuman.ai/the-very-basics-of-reinforcement-learning-154f28a79071

Value based RL e.g. Q-learning Policy based RL e.g. policy gradient

Agent

Deep RL for Combinatorial Optimization

• Neural Architecture Search with Reinforcement Learning

Deep RL for Combinatorial Optimization

• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with

Weak Supervision

Deep RL for Combinatorial Optimization

• Neural Combinatorial Optimization with Reinforcement Learning

"Fundamental" Program Synthesis

• Focus on algorithmic coding problems.
• No ground-truth program solutions.
• Simple Turing-complete language.

++++++++ Set Cell #0 to 8 [ >++++ Add 4 to Cell #1; this will always set Cell #1 to 4 [ as the cell will be cleared by the loop >++ Add 2 to Cell #2 >+++ Add 3 to Cell #3 >+++ Add 3 to Cell #4 >+ Add 1 to Cell #5 <<<<- Decrement the loop counter in Cell #1 ] Loop till Cell #1 is zero; number of iterations is 4 >+ Add 1 to Cell #2 >+ Add 1 to Cell #3 >- Subtract 1 from Cell #4 >>+ Add 1 to Cell #6 [<] Move back to the first zero cell you find; this will be Cell #1 which was cleared by the previous loop <- Decrement the loop Counter in Cell #0 ] Loop till Cell #0 is zero; number of iterations is 8 The result of this is: Cell No : 0 1 2 3 4 5 6 Contents: 0 0 72 104 88 32 8 Pointer : ^ >>. Cell #2 has value 72 which is 'H' >---. Subtract 3 from Cell #3 to get 101 which is 'e' +++++++..+++. Likewise for 'llo' from Cell #3 >>. Cell #5 is 32 for the space <-. Subtract 1 from Cell #4 for 87 to give a 'W' <. Cell #3 was set to 'o' from the end of 'Hello' +++.------.--------. Cell #3 for 'rl' and 'd' >>+. Add 1 to Cell #5 gives us an exclamation point >++. And finally a newline from Cell #6

HelloWorld.bf

++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>- >>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.++ +.------.--------.>>+.>++. https://en.wikipedia.org/wiki/Brainfuck

Anatomy of BF

Turing complete! https://esolangs.org/wiki/Brainfuck#Computational_class

BF Execution Demo: Reverse a list

Why BF

• Turing complete, and suitable for any algorithmic task in theory.
• Many algorithms have surprisingly elegant BF implementations.
• No syntax errors (with minor adjustment to interpreter).
• No names (variables, functions).

Training Setup

RNN Code Reward

Inference Gradient Update

Reward Function

Test cases BF interpreter

code inputs

• utputs

Scoring Function

reward

Training Setup: Reward Function

Reward = ∑

S(P(I), Y*)

Test case

input

• utput

program program output X = [1, 2, 3, 4, 0] Y* = [4, 3, 2, 1, 0] P = ,>,.<. P(X) = [2, 1]

Score

S(Y, Y*) = d(∅, Y*) - d(Y, Y*) d(Y, Y*) = variable length Hamming distance Y* = [4, 3, 2, 1, 0] Y = [2, 1] d = ∑ 2 2 B B B base B = 256 Y* = [4, 3, 2, 1, 0] ∅ = [] d = ∑ B B B B B

Problems with policy gradient (REINFORCE)

• Catastrophic forgetting and unstable learning
• Sample inefficient

Solution: Priority Queue Training (PQT)

Max-Unique Priority Queue training targets Reward Function code sampled from RNN rewards

Results

⋯

Fixed Length Programs

remove

<>[,[<+,.],<<]],[[-[+.>>][]>[>[>[[<+>>.+<>]>]<<>]],]>+-++--,>[+[[<----].->+]->]]]-[,.]+>>,-,,-]><,,]

reverse

,[[>,<>]]-]+[<[.,++,<]<>[->.,+,[<+]<-]<,,<<>>[[[<+<[],.>->]>,<-]<]<>,-<,,[+>,<,><.[.<-+,+-<]+<[,+-<>

add

,>,[-<+>][,],>]<]-<.+,,+,<.,>]>,[><<-,][+-[.[[+<[.>]],>.[]-<,],+,[,->]>>->+,[+[>]-,-]--,.,>+-<<]]<,+

length 100

Synthesized vs "Ground Truth"

,[>,]+[,<.] ,-[+.,-]+[,.] ,[-[>]>+<<<<,]>. ,[+>,<<->],<.,. ,+>,<[,>],<+<. ++++++++.---.+++++++..+++. ,.,[.>.-<,[[[.+,>+[-.>]..<]>+<<]>+<<]] ,[.,.[.,.[..,[....,[.....,[.>]<]].]] ,>,[.,]<.>. ,[>,]<.,<<<<<.[>.]

• [,>,[.,>,]<[>,]<.]

,>,[<.>>,]. >,<,>>,[<.,[<.[>]],]. ,[.,>,]<<<<.[>.] ,[>+<,]>. ,[,]-[,.] ,-[->-[,]<]-[,.] >,[>,]<[.<]. ,[-[+.[-]],]. >,[-[<->[-]]<+>,]<. ,>,<[->+<]>. ??? ++++++++.---.+++++++..+++. ,[.>[->+>.<<]>+[-<+>]<<,] ,>>+<<[>>[-<+>]<[->+<<.>]>+<<,]. ,>,[.,]<.,. >,[>,]<.[-]<[<]>[.>]. >,[>,[.[-]],]<[.<]. ,>,[[<.[-]>[-<+>]],]. ,>,>,[[<<.[-]>[-<+>]>[-<+>]],]. >,[.,>,]<<[<]>[.>]. >+>,[[<]>+[>],]<[<]>-. ,[,],[.,]. ,-[->,[,]<],[.,]. reverse remove count-char add bool-logic print zero-cascade cascade shift-left shift-right unriffle remove-last remove-last-two echo-alternating length echo-second-seq echo-nth-seq Synthesized Experimenter's best solution

What's next?

Scale up to harder coding problems, and more complex programming languages.

• Augment RL with supervised training on a large corpus of programs.
• Give the code synthesizer access to auxiliary information, such as stack traces

and program execution internals.

• Data augmentation techniques, such as Hindsight experience replay.
• Few-shot learning techniques can help with generalization issues, e.g. MAML.

Thank you! Questions?

Thank you to my coauthors: Mohammad Norouzi, Jonathan Shen, Rui Zhao, Quoc

• V. Le.

Prior Work

• Algorithm induction

○ Neural Programmer, A. Neelakantan, et al. ○ Neural Programmer-Interpreters, S. Reed, et al.

• Domain specific languages

○ RobustFill, J. Devlin, et al. ○ DeepCoder, M. Balog, et al. ○ TerpreT, A. Gaunt, et al.

• Precursors to PQT

○ Noisy Cross-Entropy Method, I. Szita, et al. ○ Neural Symbolic Machines, C. Liang, et al.