SymbiYosys Formal Hardware Verification with OpenSource Tools - - PowerPoint PPT Presentation

SymbiYosys – Formal Hardware Verification with OpenSource Tools Clifford Wolf Symbiotic EDA

Yosys, Yosys-SMTBMC, SymbiYosys

• Yosys

– FOSS Verilog Synthesis tool and more – highly flexible, customizable using scripts

• Formal Verification (Safety Properties, Liveness Properties, Equivalence, Coverage)
• FPGA Synthesis for iCE40 (Project IceStorm), Xilinx 7-series (Vivado for P&R),

GreenPAK4 (OpenFPGA), Xilinx Coolrunner-II, Gowin Semi FPGAs, MAX10, …

• ASIC Synthesis (full FOSS flows: Qflow, Coriolis2)
• Yosys-SMTBMC

– A flow with focus on verification of safety properties using BMC and k-

induction, using SMT2 circuit descriptions generated by Yosys

• SymbiYosys

– A unified front-end for many Yosys-based formal verification flows

SymbiYosys Features

• Bounded verification of safety properties
• Unbounded verification of safety properties
• Generation of test benches from cover statements
• Verification of liveness properties
• Formal equivalence checking [TBD]
• Reactive Synthesis [TBD]

Solvers:

– SMT2

• Yices, Boolector, Z3, CVC4, Mathsat
• easy to extend to any SMT2 solver with QF_AUFBV, QF_ABV, QF_BV, or QF_UFBV support

– AIGER

• super_prove, Avy, everything in ABC (including pdr)
• easy to extend to any AIGER solver for safety and/or liveness properties

– BTOR2 [TBD]

• new word-level HW model checking format (see CAV 2018 paper)

Types of Properties Supported in SymbiYosys

• Safety properties

– Verilog assume(…) and assert(…) statements – a CEX trace satisfies all assumptions and violates at least one assertion

• Fairness and Liveness

– Fairness: if (…) assume property (s_eventually …); – Liveness: if (…) assert property (s_eventually …); – a CEX trace contains a loop that satisfies all fairness properties (and all assumptions)

and violates at least one liveness property

• Cover

– Verilog cover(…) statements – Produces a trace for each cover statement that satisfies that cover statement.

Availability of various EDA tools for students, hobbyists, enthusiasts

• FPGA Synthesis

– Free to use:

• Xilinx Vivado WebPack, etc.

– Free and Open Source:

• Yosys + Project IceStorm
• VTR (Odin II + VPR)
• HDL Simulation

– Free to use:

• Xilinx XSIM, etc.

– Free and Open Source:

• Icarus Verilog, Verilator, etc.
• Formal Verification

– Free to use:

• ???

– Free and Open Source:

• ???

.. and people in the industry are complaining they can't find any verification experts to hire!

“Formal first” vs. traditional use

• f formal methods

Cost of (fixing) a bug Time Development Verification / Testing Production Traditional use-case for formal Number of found new bugs Formal first Most formal tools are priced and advertised for the traditional use case.

Formal First → designing better digital circuits faster and cheaper

• Formal First is a set of design methodologies focusing on using formal methods during

development, as early as possible.

– Target user base is design engineers, not verification engineers

• Not necessarily for creating complete correctness proofs. Instead run simple BMC for “low hanging

fruits” safety properties, such as

– standard bus interfaces like AXI/APB/etc. – simple data flow analysis to catch reset issues and/or pipeline interlocking problems – use cover() statements to replace hard-to-write one-off test benches for trying things with the design under test

• Can be as simple as: always @(posedge i_clk) cover(o_wb_ack);
• Formal methods can help to find a vast range of bugs sooner and produces shorter (and thus easier

to analyze) counter example traces.

• Let’s not limit our thinking to “formal is for XYZ”! Formal is a set of fairly generic technologies that

have applications everywhere in the design process!

– But we cannot unleash the full potential formal has to offer unless we make sure that every digital design

and/or verification engineer has access to formal tools. (Like each of those people has access to HDL simulators.)

Formal First

• Here are a few example use cases for formal tools during the development phase of a new

circuit:

– Verification of embedded “sanity check” assertions

• E.g. “write and read pointers never point to the same element after reset”

– Verification of standardized interface using standardized “off-the-shelf” formal properties

• E.g. standardized bus interfaces such as AXI.

– Using cover statements to create test benches quickly.

• E.g. cover “done signal goes high (some time after reset)”

– Using cover statements during debugging to make sense of trace data from FPGA based test runs.

• E.g. cover “done signal goes high while NAK is active”
• Or assert “done signal never goes high while NAK is active”

– Note that this are the same techniques that are employed in the traditional use case for formal. – This is similar to how simulators are used by design and verification engineers alike. – Nobody would claim that simulators are “only for verification (of few very special designs)”.

HDL features in Yosys (Open Source) and Symbiotic EDA Suite (Commercial)

• Yosys

– Verilog 2005 – Memories / Arrays – Immediate assert(),

assume(), and cover()

– checkers, rand [const] regs – Special attributes:

• anyconst, anyseq, allconst, allseq, gclk
• Symbiotic EDA Suite

– Everything in Yosys

+ SystemVerilog 2012 + VHDL 2008 + Concurrent assert(), assume(), and cover() + SVA Properties

Safety Properties: Are the bad states reachable from the initial states ?

reachable states (implicit) assertions (explicit) unreachable non- bad states. many of those usually implies a difficult proof.

Bounded Model Check (BMC)

In steps 1 .. k-1: UNSAT → next step SAT → FAIL In step k: UNSAT → PASS SAT → FAIL

1 2 3 k-1 1 1 1 2 2 3

Step 1: Step 2: Step 3: Step 4: Step k: BMC proves that no bad state is reachable within k cycles.

k-Induction

k-1 k k-2 k-3 k-1 k k-2 k-3 k-1 k k-2 k-1 k k

Step 1: Step 2: Step 3: Step 4: Step k+1: In steps 1 .. k: SAT → next step UNSAT → PASS In step k+1: SAT → UNKNOWN UNSAT → PASS k-induction proves that a sequence of k non-bad states is always followed by another non-bad state. The k used for induction must be ≤ the k used in BMC for a valid complete proof.

SymbiYosys flow with Yosys-SMTBMC

Verilog Design Verilog Asserts Yosys SMT-LIB2 Code Constraints File Yosys-SMTBMC PASS / FAIL VCD File Verilog Testbench Constraints File

Trace / counterexample formats

SMT-LIB2 Solver

SymbiYosys flow with AIGER model checker

Verilog Design Verilog Asserts Yosys AIGER Model Checker (e.g. pdr, avy) AIGER witness SMT-LIB code Yosys-SMTBMC Counter Example SMT-LIB2 Solver

unoptimized word-level representation, good for creating human readable counter examples

• ptimized bit-level model

Yosys-SMTBMC is only used here as a post- processor, turning the AIGER witness into a useful human readable counter example (e.g. VCD). PASS/FAIL

Custom SMT-LIB Flows

Verilog Design Verilog Asserts Yosys SMT-LIB2 Code

? ? ?

SMT-LIB2 Solver Options for writing custom proofs:

• Hand-written SMT2 code
• Custom python script using

smtio.py (the python lib implementing most of yosys- smtbmc)

• Any other app using any SMT-

LIB2 solver (e.g. using C/C++ API for proofs that involve many (check-sat) calls.

Hello World

module hello ( input clk, rst,

• utput [3:0] cnt

); reg [3:0] cnt = 0; always @(posedge clk) begin if (rst) cnt <= 0; else cnt <= cnt + 1; end `ifdef FORMAL always @* assume (cnt != 10); always @* assert (cnt != 15); `endif endmodule hello.sv [options] mode prove depth 10 [engines] smtbmc z3 [script] read_verilog -formal hello.sv prep -top hello [files] hello.sv hello.sby

Hello World

$ sby -f hello.sby SBY 14:45:35 [hello] Removing direcory 'hello'. SBY 14:45:35 [hello] Copy 'hello.sv' to 'hello/src/hello.sv'. SBY 14:45:35 [hello] engine_0: smtbmc z3 … … … SBY 14:45:35 [hello] engine_0.induction: finished (returncode=0) SBY 14:45:35 [hello] engine_0: Status returned by engine for induction: PASS SBY 14:45:36 [hello] engine_0.basecase: finished (returncode=0) SBY 14:45:36 [hello] engine_0: Status returned by engine for basecase: PASS SBY 14:45:36 [hello] summary: Elapsed clock time [H:MM:SS (secs)]: 0:00:00 (0) SBY 14:45:36 [hello] summary: Elapsed process time [H:MM:SS (secs)]: 0:00:00 (0) SBY 14:45:36 [hello] summary: engine_0 (smtbmc z3) returned PASS for induction SBY 14:45:36 [hello] summary: engine_0 (smtbmc z3) returned PASS for basecase SBY 14:45:36 [hello] summary: successful proof by k-induction. SBY 14:45:36 [hello] DONE (PASS, rc=0)

• The sby option -f causes sby to remove the output directory if it already exists.
• The output directory contains all relevant information, including copies of the HDL design files.

fib.sv

module fib ( input clk, pause, start, input [3:0] n,

• utput reg busy, done,
• utput reg [9:0] f

); reg [3:0] count; reg [9:0] q; initial begin done = 0; busy = 0; end always @(posedge clk) begin done <= 0; if (!pause) begin if (!busy) begin if (start) busy <= 1; count <= 0; q <= 1; f <= 0; end else begin q <= f; f <= f + q; count <= count + 1; if (count == n) begin busy <= 0; done <= 1; end end end end `ifdef FORMAL always @(posedge clk) begin if (busy) begin assume (!start); assume ($stable(n)); end if (done) begin case ($past(n)) 0: assert (f == 1); 1: assert (f == 1); 2: assert (f == 2); 3: assert (f == 3); 4: assert (f == 5); 5: assert (f == 8); endcase cover (f == 13); cover (f == 144); cover ($past(n) == 15); end assume property (s_eventually !pause); if (start && !pause) assert property (s_eventually done); end `endif endmodule

fib_{prove,live,cover}.sby

[options] mode prove [engines] abc pdr [script] read_verilog -formal fib.sv prep -top fib [files] fib.sv fib_prove.sby [options] mode live [engines] aiger suprove [script] read_verilog -formal fib.sv prep -top fib [files] fib.sv fib_live.sby [options] mode cover append 10 [engines] smtbmc yices [script] read_verilog -formal fib.sv prep -top fib [files] fib.sv fib_cover.sby

Prove safety properties in fib.v using IC3 (pdr). Prove liveness properties in fib.v. This assumes that safety properties are already proven. Create a trace for each cover statement in the design (and check asserts for that trace). Add 10 additional time steps after the cover statement has been reached.

parcase.sv

module parcase (input clk, A, B, C, D, E, BUG, output reg Y); always @(posedge clk) begin Y <= 0; if (A != B || BUG) begin (* parallel_case *) case (C) A: Y <= D; B: Y <= E; endcase end end endmodule

[script] read_verilog -formal parcase.v prep -top parcase assertpmux $ sby -f parcase.sby … … Assert failed in parcase: parcase.v:6 … SBY [parcase] DONE (FAIL)

memcmp.sv

module memory1 ( input clk, input [3:0] wstrb, input [15:0] waddr, input [15:0] raddr, input [31:0] wdata,

• utput [31:0] rdata

); reg [31:0] mem [0:2**16-1]; reg [15:0] buffered_raddr; // "transparent" read assign rdata = mem[buffered_raddr]; always @(posedge clk) begin if (wstrb[3]) mem[waddr][31:24] <= wdata[31:24]; if (wstrb[2]) mem[waddr][23:16] <= wdata[23:16]; if (wstrb[1]) mem[waddr][15: 8] <= wdata[15: 8]; if (wstrb[0]) mem[waddr][ 7: 0] <= wdata[ 7: 0]; buffered_raddr <= raddr; end endmodule

memcmp.sv

module memory2 ( input clk, input [3:0] wstrb, input [15:0] waddr, input [15:0] raddr, input [31:0] wdata,

• utput [31:0] rdata

); reg [31:0] mem [0:2**16-1]; reg [31:0] buffered_wdata; reg [31:0] buffered_rdata; reg [3:0] buffered_wstrb; reg waddr_is_not_raddr; wire [31:0] expanded_wstrb = {{8{wstrb[3]}}, {8{wstrb[2]}}, {8{wstrb[1]}}, {8{wstrb[0]}}}; wire [31:0] expanded_buffered_wstrb = {{8{buffered_wstrb[3]}}, {8{buffered_wstrb[2]}}, {8{buffered_wstrb[1]}}, {8{buffered_wstrb[0]}}}; assign rdata = waddr_is_not_raddr ? buffered_rdata : (buffered_wdata & expanded_buffered_wstrb) | (buffered_rdata & ~expanded_buffered_wstrb); always @(posedge clk) begin mem[waddr] <= (wdata & expanded_wstrb) | (mem[waddr] & ~expanded_wstrb); buffered_wstrb <= wstrb; buffered_wdata <= wdata; buffered_rdata <= mem[raddr]; waddr_is_not_raddr <= waddr != raddr; end endmodule

memcmp.sv

module memcmp ( input clk, input [3:0] wstrb, input [15:0] waddr, input [15:0] raddr, input [31:0] wdata,

• utput [31:0] rdata1,
• utput [31:0] rdata2

); memory1 mem1 ( .clk (clk ), .wstrb(wstrb ), .waddr(waddr ), .raddr(raddr ), .wdata(wdata ), .rdata(rdata1) ); memory2 mem2 ( .clk (clk ), .wstrb(wstrb ), .waddr(waddr ), .raddr(raddr ), .wdata(wdata ), .rdata(rdata2) ); endmodule

initial assume (= [mem1.mem] [mem2.mem]) always 1 assert (= [mem1.mem] [mem2.mem]) assert (= [rdata1] [rdata2])

memcmp.smtc [options] mode prove smtc memcmp.smtc depth 10 [script] read_verilog -formal memcmp.sv prep -nordff -top memcmp ... memcmp.sby

memcheck.sv

module memory ( input clk, we, input [31:0] addr, input [7:0] wdata,

• utput reg [7:0] rdata

); reg [7:0] bank_0 [0:2**30-1]; reg [7:0] bank_1 [0:2**30-1]; reg [7:0] bank_2 [0:2**30-1]; reg [7:0] bank_3 [0:2**30-1]; always @(posedge clk) begin case (addr[1:0]) 2'b 00: begin rdata <= bank_0[addr >> 2]; if (we) bank_0[addr >> 2] <= wdata; end 2'b 01: begin rdata <= bank_1[addr >> 2]; if (we) bank_1[addr >> 2] <= wdata; end 2'b 10: begin rdata <= bank_2[addr >> 1]; // <- BUG if (we) bank_2[addr >> 2] <= wdata; end 2'b 11: begin rdata <= bank_3[addr >> 2]; if (we) bank_3[addr >> 2] <= wdata; end endcase end endmodule

memcheck.sv

module memcheck ( input clk, we, input [31:0] addr, input [7:0] wdata,

• utput [7:0] rdata

); memory uut ( .clk (clk ), .we (we ), .addr (addr ), .wdata(wdata), .rdata(rdata) ); reg monitor_valid = 0; (* anyconst *) wire [31:0] monitor_addr; reg [7:0] monitor_data; always @(posedge clk) begin if ((addr == monitor_addr) && we) begin monitor_valid <= 1; monitor_data <= wdata; end if (($past(addr) == monitor_addr) && monitor_valid && $past(monitor_valid)) begin assert (rdata == $past(monitor_data)); end end endmodule

[options] mode bmc expect fail depth 10 ...

memcheck.sby

$ make memcheck sby -f memcheck.sby SBY 14:54:59 [memcheck] Removing direcory 'memcheck'. SBY 14:54:59 [memcheck] Copy 'memcheck.sv' to 'memcheck/src/memcheck.sv'. SBY 14:54:59 [memcheck] engine_0: smtbmc z3 ... SBY 14:54:59 [memcheck] engine_0: ## Solver: z3 SBY 14:54:59 [memcheck] engine_0: ## Checking assumptions in step 0.. SBY 14:54:59 [memcheck] engine_0: ## Checking assertions in step 0.. SBY 14:54:59 [memcheck] engine_0: ## Checking assumptions in step 1.. SBY 14:54:59 [memcheck] engine_0: ## Checking assertions in step 1.. SBY 14:54:59 [memcheck] engine_0: ## Checking assumptions in step 2.. SBY 14:54:59 [memcheck] engine_0: ## Checking assertions in step 2.. SBY 14:54:59 [memcheck] engine_0: ## Checking assumptions in step 3.. SBY 14:54:59 [memcheck] engine_0: ## Checking assertions in step 3.. SBY 14:54:59 [memcheck] engine_0: ## BMC failed! SBY 14:54:59 [memcheck] engine_0: ## Value for anyconst memcheck.monitor_addr (memcheck.sv:16): 66 SBY 14:54:59 [memcheck] engine_0: ## Assert failed in memcheck: memcheck.sv:26 SBY 14:54:59 [memcheck] engine_0: ## Writing trace to VCD file: engine_0/trace.vcd SBY 14:54:59 [memcheck] engine_0: ## Writing trace to Verilog testbench: engine_0/trace_tb.v SBY 14:54:59 [memcheck] engine_0: ## Writing trace to constraints file: engine_0/trace.smtc SBY 14:54:59 [memcheck] engine_0: ## Status: FAILED (!) SBY 14:54:59 [memcheck] engine_0: finished (returncode=1) SBY 14:54:59 [memcheck] engine_0: Status returned by engine: FAIL SBY 14:54:59 [memcheck] summary: Elapsed clock time [H:MM:SS (secs)]: 0:00:00 (0) SBY 14:54:59 [memcheck] summary: Elapsed process time [H:MM:SS (secs)]: 0:00:00 (0) SBY 14:54:59 [memcheck] summary: engine_0 (smtbmc z3) returned FAIL SBY 14:54:59 [memcheck] summary: counterexample trace: memcheck/engine_0/trace.vcd SBY 14:54:59 [memcheck] DONE (FAIL, rc=0)

multiclk.sv

module multiclk(input clk, output [3:0] counter_a, counter_b); reg [3:0] counter_a = 0; reg [3:0] counter_b = 0; always @(posedge clk) counter_a <= counter_a + 1; always @(posedge clk) counter_b[0] <= !counter_b[0]; always @(negedge counter_b[0]) counter_b[1] <= !counter_b[1]; always @(negedge counter_b[1]) counter_b[2] <= !counter_b[2]; always @(negedge counter_b[2]) counter_b[3] <= !counter_b[3]; always @* assert (counter_a == counter_b); endmodule

... [options] ... multiclock on ... ...

multiclk.sby

setreset.sv

module setreset(input clk, input set, rst, d, output q1, q2); reg q1 = 0; always @(posedge clk, posedge set, posedge rst) if (rst) q1 <= 0; else if (set) q1 <= 1; else q1 <= d; reg q2_s = 0, q2_r = 0, q2_l; wire q2 = q2_l ? q2_s : q2_r; always @(posedge clk, posedge set) if (set) q2_s <= 1; else q2_s <= d; always @(posedge clk, posedge rst) if (rst) q2_r <= 0; else q2_r <= d; always @* begin if (rst) q2_l <= 0; else if (set) q2_l <= 1; always @* assert (q1 == q2); endmodule

... [options] ... multiclock on ... ...

setreset.sby

Free Variables

• (* anyseq *) wire [7:0] cmd;

– Behaves like an additional primary input

• (* anyconst *) wire [7:0] cmd;

– Behaves like an additional primary input that is latched in the first

cycle.

• rand reg [7:0] cmd;
• rand const reg [7:0] cmd;

– For improved SV compatibility (only valid SV in checker …

endchecker block, Yosys supports it everywhere)

Special “forall” Variables

• (* allseq *) wire [7:0] cmd;

– Find a trace so that assumptions are satisfied for all possible sequences

• f values in cmd. Only one of those values needs to violate an assertion

for a CEX.

• (* allconst *) wire [7:0] cmd;

– Find a trace so that assumptions are satisfied for all possible constant

values in cmd. Only one of those values needs to violate an assertion for a CEX.

• This features can be used to construct exists-forall (

) SMT2 ∃∀ problems using Verilog code. (Requires SMT solver with quantifier support, such as Z3.)

∃∀ Example: Find a prime

module primegen; (* anyconst *) wire [31:0] prime; (* allconst *) wire [15:0] factor; always @* begin if (1 < factor && factor < prime) assume((prime % factor) != 0); assume(prime > 1000000000); cover(1); end endmodule Z3 after 35 seconds: 2359012091

∃∀ Example: Find two primes with a prime gap of 500

module primes; parameter [8:0] gap = 500; (* anyconst *) wire [8:0] prime1; wire [9:0] prime2 = prime1 + gap; (* allconst *) wire [4:0] factor; always @* begin if (1 < factor && factor < prime1) assume((prime1 % factor) != 0); if (1 < factor && factor < prime2) assume((prime2 % factor) != 0); assume(1 < prime1); cover(1); end endmodule Z3 after 2 seconds: 173 and 673

Usability Study: riscv-formal

• riscv-formal is a framework for formal verification of RISC-V

Processor Cores against ISA spec using SymbiYosys.

• riscv-formal is verifying real-world RISC-V processors cores (Rocket,

VexRiscv, PicoRV32) using the SymbiYosys FOSS flow.

• Processors must implement RVFI (RISC-V Formal Interface) to be

verifiable using riscv-formal.

• riscv-formal spec is formally verified for equivalence against riscv-

isa-sim (aka Spike), the official reference simulator (C++).

• It is also formally verified against riscv-semantics, aka “the RISC-V

MIT Model” and upcoming official RISC-V formal spec written in Haskell.

• Currently supported ISAs: User-Mode RV32IMC, RV64IMC
• Currently in development: M-Mode, S-Mode, F/D/Q-Extension

Future Work: Formal Testbench Fuzzer

• We are building a “test bench fuzzer” to assess the quality of test benches using a

mutation testing approach:

– We automatically find small changes to a design that have externally observable

• consequences. (aka “bugs”)

– Formal methods are deployed to find those changes. – We then verify that the test bench will find those bugs. – The percentage of modified designs that the test bench can find is a metric for the quality of

the test bench.

• This is an improvement over traditional test coverage metrics:

– Coverage only checks that the test bench utilizes each line of code. – Our approach checks that the consequence of each line of code is actually checked by the

test bench.

– No support from simulator is needed, thus this works to verify any verification strategy as

long as the verification system can process the modified Verilog designs.

References

• Bounded Model Checking, Armin Biere, Handbook of Satisfiability. Armin Biere, Marijn Heule,

Hans von Maaren and Toby Walsh (Eds.), pages 457-481

• Satisfiability Modulo Theories, Clark Barrett, Roberto Sebastiani, Sanjit A. Seshia and Cesare
• Tinelli. Handbook of Satisfiability. Armin Biere, Marijn Heule, Hans von Maaren and Toby Walsh

(Eds.), pages 852-885

• Temporal Induction by Incremental SAT Solving. Niklas Een, Niklas Sörensson, BMC 2003.
• The SMT-LIB Standard: Version 2.5, by Clark Barrett, Pascal Fontaine, and Cesare Tinelli.
• Boolector 2.0. Aina Niemetz, Mathias Preiner, Armin Biere. Journal of Satisfiability, Boolean

Modeling and Computation (JSAT), vol. 9, 2015, pages 53-58.

• Yices 2.2. Bruno Dutertre. CAV'2014.
• These slides: http://www.clifford.at/papers/2018/sby-formal/
• Yosys: http://www.clifford.at/yosys/
• SymbiYosys: https://symbiyosys.readthedocs.io/
• riscv-formal: https://github.com/cliffordwolf/riscv-formal/