Proving Unrealizability for Syntax-Guided Synthesis Qinheping Hu , - - PowerPoint PPT Presentation

Proving Unrealizability for Syntax-Guided Synthesis

Qinheping Hu , Jason Breck , John Cyphert , Loris D'Antoni , Thomas Reps

Proving Unrealizability for Syntax-Guided Synthesis

1

Start → +(Start, Start) | 𝐽𝑈𝐹(BExpr, Start, Start) 𝑦 𝑧 0 1 BExpr → 𝑂𝑝𝑢(BExpr) | > (Start, Start) |𝐵𝑜𝑒(BExpr, BExpr)

Syntax-Guided Synthesis (SyGuS)

SyGuS Solver Specification Solution Program Search space 𝐻:

𝜒 𝑛𝑏𝑦(𝑦, 𝑧), 𝑦, 𝑧 :

𝑛𝑏𝑦 𝑦, 𝑧 ≥ 𝑦 ∧ 𝑛𝑏𝑦 𝑦, 𝑧 ≥ 𝑧 ∧ (𝑛𝑏𝑦 𝑦, 𝑧 = 𝑦 ∨ 𝑛𝑏𝑦 𝑦, 𝑧 = 𝑧)

𝑓 ∈ 𝑀(𝐻) such that ∀𝑦, 𝑧. 𝜒 𝑓, 𝑦, 𝑧 max 𝑦, 𝑧 = 𝐽𝑈𝐹(> (𝑦, 𝑧), 𝑦, 𝑧)

Syntax-Guided Synthesis (SyGuS)

• Goal: find a program 𝑓 ∈ 𝑀(𝐻) such that ∀𝑦, 𝑧. 𝜒 𝑓, 𝑦, 𝑧
• SyGuS-Competition
• SyGuS Solvers: CVC4, EUSolver, Euphony, DryadSynth,

LoopInvGen, E3Solver, Esolver

What if there doesn’t exist 𝑓 ∈ 𝑀(𝐻) such that ∀𝑦, 𝑧. 𝜒 𝑓, 𝑦, 𝑧 (Unrealizable)

Proving Unrealizability for Syntax-Guided Synthesis

1 2

Example of Unrealizable SyGuS Problems

∀𝑦, 𝑧. max 𝑦, 𝑧 ≥ 𝑦 ∧ max 𝑦, 𝑧 ≥ 𝑧 ∧ (max 𝑦, 𝑧 = 𝑦 ∨ max 𝑦, 𝑧 = 𝑧) max 𝑦, 𝑧 = 𝐽𝑈𝐹(> (𝑦, 𝑧), 𝑦, 𝑧) Start = +(Start, Start) | 𝐽𝑈𝐹(BExpr, Start, Start) 𝑦 𝑧 0 1 BExpr = 𝑂𝑝𝑢(BExpr) | > (Start, Start) |𝐵𝑜𝑒(BExpr, BExpr)

Specification Search space

Example of Unrealizable SyGuS Problems

∀𝑦, 𝑧. max 𝑦, 𝑧 ≥ 𝑦 ∧ max 𝑦, 𝑧 ≥ 𝑧 ∧ (max 𝑦, 𝑧 = 𝑦 ∨ max 𝑦, 𝑧 = 𝑧) Start = +(Start, Start) 𝑦 𝑧 0 1

Specification Search space

No Solution

Proving Unrealizability for Syntax-Guided Synthesis

1 2 3

Examples E: (x0,y0)=(0,0) CEGIS Verifier Synthesizer 𝑇𝑧𝐹

φ: 𝑔 𝑦, 𝑧 ≥ 𝑦 ∧ 𝑔 𝑦, 𝑧 ≥ 𝑧 ∧ (𝑔 𝑦, 𝑧 = 𝑦 ∨ 𝑔 𝑦, 𝑧 = 𝑧) 𝑇𝑧 ≔ G: Start = +(Start, Start) x 𝑧 1

Examples E: (x0,y0)=(0,0) Synthesizer 𝑇𝑧𝐹

𝑇𝑧𝐹: ∧ 𝑦,𝑧 ∈𝐹 φ(𝑔, 𝑦, 𝑧)

Examples E: (x0,y0)=(0,0) CEGIS Verifier Synthesizer 𝑇𝑧𝐹 𝑔 𝑦, 𝑧 = 0 new ce (0,1)

φ: 𝑔 𝑦, 𝑧 ≥ 𝑦 ∧ 𝑔 𝑦, 𝑧 ≥ 𝑧 ∧ (𝑔 𝑦, 𝑧 = 𝑦 ∨ 𝑔 𝑦, 𝑧 = 𝑧) 𝑇𝑧 ≔ G: Start = +(Start, Start) x 𝑧 1

Examples E: (x0,y0)=(0,0) (x1,y1)=(0,1) CEGIS Verifier Synthesizer 𝑇𝑧𝐹 𝑔 𝑦, 𝑧 = 𝑧 new ce (1,0)

φ: 𝑔 𝑦, 𝑧 ≥ 𝑦 ∧ 𝑔 𝑦, 𝑧 ≥ 𝑧 ∧ (𝑔 𝑦, 𝑧 = 𝑦 ∨ 𝑔 𝑦, 𝑧 = 𝑧) 𝑇𝑧 ≔ G: Start = +(Start, Start) x 𝑧 1

Examples E: (x0,y0)=(0,0) (x1,y1)=(0,1) (x2,y2)=(1,0) CEGIS Verifier Synthesizer 𝑇𝑧𝐹 𝑔 𝑦, 𝑧 = 1 new ce (2,0)

φ: 𝑔 𝑦, 𝑧 ≥ 𝑦 ∧ 𝑔 𝑦, 𝑧 ≥ 𝑧 ∧ (𝑔 𝑦, 𝑧 = 𝑦 ∨ 𝑔 𝑦, 𝑧 = 𝑧) 𝑇𝑧 ≔ G: Start = +(Start, Start) x 𝑧 1

Examples E: (x0,y0)=(0,0) (x1,y1)=(0,1) (x2,y2)=(1,0) (x3,y3)=(2,0) CEGIS Verifier Synthesizer 𝑇𝑧𝐹

Unrealizable! φ: 𝑔 𝑦, 𝑧 ≥ 𝑦 ∧ 𝑔 𝑦, 𝑧 ≥ 𝑧 ∧ (𝑔 𝑦, 𝑧 = 𝑦 ∨ 𝑔 𝑦, 𝑧 = 𝑧) 𝑇𝑧 ≔ G: Start = +(Start, Start) x 𝑧 1

𝑡𝑧

is unrealizable

𝑡𝑧𝐹

is unrealizable

No solution over 𝐹 No solution over all inputs

From SyGuS over Examples to a Reachability Problem

Reachability Problem

void main(){ int x = 0; while(nd()){ x++; } assert(x<0) } Goal: can the assert be falsified?

Reachability solver: CPA-checker Uautomizer Seahorn …

Non-deterministic choice

Overview

SyGuS over examples Reachability problem

𝑡𝑧𝐹 𝑆𝑓𝐹

(φ𝐹, 𝐻) void main(){ … assert(…)}

𝑡𝑧𝐹is unrealizable assert cannot be falsified

𝑇𝑧𝐹 to 𝑆𝑓𝐹

Ԧ 𝑦 ← 𝐹 Ԧ 𝑝 ← 𝑔

𝐻( Ԧ

𝑦)

𝑔

𝐻 is non-deterministically drawn from 𝑀(𝐻)

Set input to 𝐹

assert(¬ٿ 𝑦𝑗 ∈ 𝐹. φ(𝑝𝑗, 𝑦𝑗))

Check if Ԧ 𝑝 doesn’t satisfy φ 𝑔

𝐻( Ԧ

𝑦) satisfy φ on 𝐹 𝑇𝑧𝐹 is unrealizable

Ԧ 𝑦 ← 𝐹

Set input to 𝐹

x0 = 0; y0 = 0; x1 = 0; y1 = 1;

Examples E: (x0,y0)=(0,0) (x1,y1)=(0,1)

𝑇𝑧𝐹 to 𝑆𝑓𝐹

Ԧ 𝑦 ← 𝐹 Ԧ 𝑝 ← 𝑔

𝐻( Ԧ

𝑦)

𝑔

𝐻 is non-deterministically drawn from 𝑀(𝐻)

Set input to 𝐹

assert(¬ٿ 𝑦𝑗 ∈ 𝐹. φ(𝑝𝑗, 𝑦𝑗))

Check if Ԧ 𝑝 doesn’t satisfy φ

assert(¬ٿ 𝑦𝑗 ∈ 𝐹. φ(𝑝𝑗, 𝑦𝑗))

φ 𝑔 𝑦, 𝑧 ≔ 𝑔 𝑦, 𝑧 ≥ 𝑦 ∧ 𝑔 𝑦, 𝑧 ≥ 𝑧 ∧ (𝑔 𝑦, 𝑧 = 𝑦 ∨ 𝑔 𝑦, 𝑧 = 𝑧)

void main(){ … assert(!(spec(x0,y0,o0)&&spec(x1,y1,o1))); } bool spec(x,y,o){ return (o>=x)&&(o>=y)&&(o==x||o==y); } Check if Ԧ 𝑝 doesn’t satisfy φ

𝑇𝑧𝐹 to 𝑆𝑓𝐹

Ԧ 𝑦 ← 𝐹 Ԧ 𝑝 ← 𝑔

𝐻( Ԧ

𝑦)

𝑔

𝐻 is non-deterministically drawn from 𝑀(𝐻)

Set input to 𝐹

assert(¬ٿ 𝑦𝑗 ∈ 𝐹. φ(𝑝𝑗, 𝑦𝑗))

Check if Ԧ 𝑝 doesn’t satisfy φ

Ԧ 𝑝 ← 𝑔

𝐻( Ԧ

𝑦)

𝑔

𝐻 is non-deterministically drawn from 𝑀(𝐻)

int fStart(x0,y0){ if(nd()){ return 0;} if(nd()){ return 1;} if(nd()){ return x0;} if(nd()}{ return y0;} if(nd()){ left = fStart(x0,y0); right = fStart(x0,y0); return left + right;} }

• 0 = fStart(x0,y0);

\\ Start -> 0 \\ Start -> 1 \\ Start -> x \\ Start -> y \\ Start -> +(Start,Start)

• 0=fStart(x0,y0);
• 0 is 𝑔

𝐻 x0,y0 for some 𝑔 𝐻 in 𝑀 𝐻

• 1=fStart(x1,y1);
• 1 is 𝑔

𝐻 x1,y1 for some 𝑔 𝐻 in 𝑀 𝐻

Can be different

Ԧ 𝑝 ← 𝑔

𝐻( Ԧ

𝑦)

𝑔

𝐻 is non-deterministically drawn from 𝑀(𝐻)

<int,int> fStart(x0,y0,x1,y1){ if(nd()){ return (0,0);} if(nd()){ return (1,1);} if(nd()){ return (x0,x1);} if(nd()}{ return (y0,y1);} if(nd()){ (a0,a1) = Start(x0,y0,x1,y1); (b0,b1) = Start(x0,y0,x1,y1); return (a0+b0,a1+b1);} } \\ Start -> 0 \\ Start -> 1 \\ Start -> x \\ Start -> y \\ Start -> +(Start,Start) (o0,o1) = Start(x0,y0);

𝑡𝑧

unrealizable

𝑡𝑧𝐹 unrealizable assert cannot be falsified

Evaluation

The tool NOPE

Example Set 𝐹

Nope

Veifier Z3 ESolver

𝑇𝑧𝐹

Unreachable

UNREALIZABLE

𝑄 UNSAT

𝑄 is a solution

SAT new example e

Seahorn Reduction 𝑆𝑓𝐹

Application

QSyGUS[cav18]

Optimal?

QSyGuS SyGuS Weighted 𝑋 Specification 𝜚 Minimize 𝑑𝑝𝑡𝑢 𝜚, 𝐻 𝜚, 𝐻<𝑑1

Solution Cost 1

⋯

Solution cost 𝐷1

𝜚, 𝐻<1

𝑁𝑗𝑜𝑗𝑛𝑗𝑨𝑓 # 𝐽𝑈𝐹 (𝜒, 𝐻<1) is unrealizable max 𝑦, 𝑧 = 𝐽𝑈𝐹(> (𝑦, 𝑧), 𝑦, 𝑧)

Benchmarks

60 SyGuS benchmarks

QSyGuS

132 SyGuS benchmarks which should be unrealizable

Overall performance of NOPE

132 variants of benchmarks taken from SyGuS

• 1. bounded number of if-operators
• 2. bounded number of plus-operators
• 3. restricted range of constants

Solved 13/57 1/30 45/45 59/132

Limitation 1 of NOPE: number of examples

Limitation 2 of NOPE: size of grammars

Large sized reachability problem

Conclusion

Example Set 𝐹

Nope

Verifier Z3 ESolver

𝑇𝑧𝐹

Unreachable

UNREALIZABLE

𝑄 UNSAT

𝑄 is a solution

SAT new example e

Seahorn Reudction 𝑆𝑓𝐹

Open questions:

• 1. reachability problem with large number of functions
• 2. beyond SyGuS

CEGIS may not Terminate

𝜒 𝑔(𝑦), 𝑦 = 𝑔 𝑦 > 𝑦

Start → +(Start, Start) 0 1

Example Set 𝐹

𝑔 𝑦 = max E + 1

Non Single-invocation Specification