iCoq : Regression Proof Selection for Large-Scale Verification - - PowerPoint PPT Presentation

▶
icoq regression proof selection for large scale
SMART_READER_LITE
LIVE PREVIEW

iCoq : Regression Proof Selection for Large-Scale Verification - - PowerPoint PPT Presentation

Oct 11, 2023 384 likes •837 views

iCoq : Regression Proof Selection for Large-Scale Verification Projects iCoq : Regression Proof Selection for Large-Scale Verification Projects Karl Palmskog University of Illinois at Urbana-Champaign Joint work with Ahmet Celik and Milos

slide-1
SLIDE 1

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Karl Palmskog University of Illinois at Urbana-Champaign Joint work with Ahmet Celik and Milos Gligoric at UT Austin

1 / 21

slide-2
SLIDE 2

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Verification Using Proof Assistants

1 encode definitions in (higher-order) formalism 2 prove propositions interactively using powerful tactics 3 check soundness of every low-level step user logic engine proof checker proof assistant tactics subgoals proof

examples: Coq, HOL4, HOL Light, Isabelle/HOL, Lean, Nuprl, ...

2 / 21

slide-3
SLIDE 3

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Some Large-Scale Proof Assistant Projects

Project Year Assistant Check Time LOC 4-Color Theorem 2005 Coq hours 60k Odd Order Theorem 2012 Coq hours 150k Kepler Conjecture 2015 HOL Light days 500k CompCert 2009 Coq tens of mins 40k seL4 2009 Isabelle/HOL hours 200k Verdi Raft 2016 Coq tens of mins 50k problem: long proof checking times

3 / 21

slide-4
SLIDE 4

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Problem: Regression Proving in Evolving Projects

Typical proving scenario:

1 change definition or lemma statement 2 begin process of re-checking all proofs 3 checking fails hours later (for seemingly unrelated proof)

4 / 21

slide-5
SLIDE 5

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Problem: Regression Proving in Evolving Projects

Typical proving scenario:

1 change definition or lemma statement 2 begin process of re-checking all proofs 3 checking fails hours later (for seemingly unrelated proof)

Typical testing scenario:

1 change method statements or method signature 2 begin process of re-running all tests 3 testing fails hours later (for seemingly unrelated test)

4 / 21

slide-6
SLIDE 6

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Regression Test Selection (RTS)

A regression test selection technique chooses, from an existing test set, tests that are deemed necessary to validate modified software. Rothermel and Harrold, ACM TOSEM 6, 2 ’97

5 / 21

slide-7
SLIDE 7

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Regression Test Selection (RTS)

A regression test selection technique chooses, from an existing test set, tests that are deemed necessary to validate modified software. Rothermel and Harrold, ACM TOSEM 6, 2 ’97 A regression proof selection technique chooses, from an existing proof set, proofs that are deemed necessary to verify modified theories.

5 / 21

slide-8
SLIDE 8

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Our Contribution

1 propose a regression proof selection (RPS) technique 2 implement RPS technique in tool, iCoq, which supports Coq 3 evaluate iCoq on revision histories of large Coq projects

6 / 21

slide-9
SLIDE 9

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Coq in a Nutshell

based on a constructive dependent type theory (CiC) definitions + proofs programmed in Gallina/Ltac (.v files) coqc tool processes .v files, outputs terms (.vo files)

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

Const Lambda Name l1 App App Ind list Ind nat Const list ind

AltLem.vo

7 / 21

slide-10
SLIDE 10

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Coq in a Nutshell

based on a constructive dependent type theory (CiC) definitions + proofs programmed in Gallina/Ltac (.v files) coqc tool processes .v files, outputs terms (.vo files)

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

Const Lambda Name l1 App App Ind list Ind nat Const list ind

AltLem.vo

7 / 21

slide-11
SLIDE 11

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Coq in a Nutshell

based on a constructive dependent type theory (CiC) definitions + proofs programmed in Gallina/Ltac (.v files) coqc tool processes .v files, outputs terms (.vo files)

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

Const Lambda Name l1 App App Ind list Ind nat Const list ind

AltLem.vo

7 / 21

slide-12
SLIDE 12

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Coq in a Nutshell

based on a constructive dependent type theory (CiC) definitions + proofs programmed in Gallina/Ltac (.v files) coqc tool processes .v files, outputs terms (.vo files)

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

Const Lambda Name l1 App App Ind list Ind nat Const list ind

AltLem.vo

7 / 21

slide-13
SLIDE 13

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Coq v8.5 Asynchronous Proof-Checking Toolchain

newly-added toolchain can produce .vio files without proofs .vio files contain proof tasks, checked asynchronously

AltLem.v coqc -quick AltLem.vio coqc -check-vio-tasks alt exists proof script alt exists proof task

8 / 21

slide-14
SLIDE 14

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Regression Proof Selection Technique

Three phases:

1 analysis: locate proofs affected by changes 2 execution: emit and run proof-checking commands 3 collection: find dependencies of modified definitions and

lemmas+proofs Key idea: maintain file and identifier dependency graphs

9 / 21

slide-15
SLIDE 15

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 1

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1 with | [] ⇒ l2 | h1 :: t1 ⇒ match l2 with | [] ⇒ h1 :: t1 | h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

10 / 21

slide-16
SLIDE 16

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 1

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1 with | [] ⇒ l2 | h1 :: t1 ⇒ match l2 with | [] ⇒ h1 :: t1 | h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

List.v Alternate.v AltLem.v File dependency graph

10 / 21

slide-17
SLIDE 17

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 1

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1 with | [] ⇒ l2 | h1 :: t1 ⇒ match l2 with | [] ⇒ h1 :: t1 | h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

# coqc -quick Alternate.v # coqc -quick AltLem.v # coqc -check-vio-tasks 0 Alternate.vio # coqc -check-vio-tasks 0 AltLem.vio

10 / 21

slide-18
SLIDE 18

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 1

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1 with | [] ⇒ l2 | h1 :: t1 ⇒ match l2 with | [] ⇒ h1 :: t1 | h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

alternate alt alt alternate alt exists Identifier dependency graph

10 / 21

slide-19
SLIDE 19

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 1

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1 with | [] ⇒ l2 | h1 :: t1 ⇒ match l2 with | [] ⇒ h1 :: t1 | h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

Programmer now changes alternate.

10 / 21

slide-20
SLIDE 20

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 2

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1, l2 with | [], _ ⇒ l2 | _, [] ⇒ l1 | h1 :: t1, h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

Change creates new revision.

11 / 21

slide-21
SLIDE 21

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 2

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1, l2 with | [], _ ⇒ l2 | _, [] ⇒ l1 | h1 :: t1, h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

List.v Alternate.v AltLem.v File dependency graph

11 / 21

slide-22
SLIDE 22

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 2

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1, l2 with | [], _ ⇒ l2 | _, [] ⇒ l1 | h1 :: t1, h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

List.v Alternate.v AltLem.v File dependency graph

11 / 21

slide-23
SLIDE 23

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 2

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1, l2 with | [], _ ⇒ l2 | _, [] ⇒ l1 | h1 :: t1, h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

# coqc -quick Alternate.v

11 / 21

slide-24
SLIDE 24

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 2

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1, l2 with | [], _ ⇒ l2 | _, [] ⇒ l1 | h1 :: t1, h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

alternate alt alt alternate alt exists Identifier dependency graph

11 / 21

slide-25
SLIDE 25

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 2

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1, l2 with | [], _ ⇒ l2 | _, [] ⇒ l1 | h1 :: t1, h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

alternate alt alt alternate alt exists Identifier dependency graph

11 / 21

slide-26
SLIDE 26

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Example, revision 2

Require Export List. Export ListNotations. Fixpoint alternate l1 l2 : list nat := match l1, l2 with | [], _ ⇒ l2 | _, [] ⇒ l1 | h1 :: t1, h2 :: t2 ⇒ h1 :: h2 :: alternate t1 t2 end. Inductive alt : list nat → list nat → list nat → Prop := | alt_nil : forall l, alt [] l l | alt_step : forall a l t1 t2, alt l t1 t2 → alt (a :: t1) l (a :: t2). Lemma alt_alternate : forall l1 l2 l3, alt l1 l2 l3 → alternate l1 l2 = l3. Proof. (* ... omitted proof script ... *) Qed.

Alternate.v

Require Import Alternate. Lemma alt_exists : forall l1 l2, exists l3, alt l1 l2 l3. Proof. induction l1; intros; destruct l2.

  • exists []. apply alt_nil.
  • exists (n :: l2). apply alt_nil.
  • exists (a :: l1). apply alt_step.

apply alt_nil.

  • specialize(IHl1 l2).

destruct IHl1. exists (a :: n :: x). repeat apply alt_step. auto. Qed.

AltLem.v

# coqc -check-vio-tasks 0 Alternate.vio

11 / 21

slide-27
SLIDE 27

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Components

Component Lang. Description coq-ast OCaml compute digests of term ASTs coq-depends OCaml extract dependencies from term ASTs coq-digest OCaml compute digests of proof scripts coqc extension OCaml proof-checking dependency extraction graph builder Java construct/maintain dependency graphs proof runner bash execute proofs

12 / 21

slide-28
SLIDE 28

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-29
SLIDE 29

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-30
SLIDE 30

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-31
SLIDE 31

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-32
SLIDE 32

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-33
SLIDE 33

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-34
SLIDE 34

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-35
SLIDE 35

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-36
SLIDE 36

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-37
SLIDE 37

iCoq: Regression Proof Selection for Large-Scale Verification Projects

iCoq Workflow

file dep graph new revision identifier dep graph impacted .v files .v file check- sums impacted proofs .vio files identifier check- sums checking com- mands identifier deps new identifier dep graph

13 / 21

slide-38
SLIDE 38

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Evaluation Research Questions

RQ1 How effective is iCoq in terms of reduction in the number of checked proofs? RQ2 How effective is iCoq in terms of the proof checking time inside a CIS environment? RQ3 How effective is iCoq in terms of the proof checking time

  • utside a CIS environment?

CIS: Continuous Integration Service (e.g., Travis CI)

14 / 21

slide-39
SLIDE 39

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Evaluation: Open Source Git-Based Projects

Project LOC #Revisions CTLTCTL 0.6k 10 InfSeqExt 1.8k 10 StructTact 2.5k 10 WeakUpTo 1.8k 10 Flocq 25k 24 UniMath 43k 24 Verdi 54k 24

15 / 21

slide-40
SLIDE 40

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Evaluation Environments

1 CI-Env uses a CIS to check proofs (from scratch) 2 LO-Env uses local machine to check proofs (incrementally)

Experiments run on Intel Core i7-6700 CPU @ 3.40GHz with 16GB

  • f RAM, running Ubuntu 14.04 LTS.

Dependency data always persisted between revision (cached).

16 / 21

slide-41
SLIDE 41

iCoq: Regression Proof Selection for Large-Scale Verification Projects

RQ1: reduction in #proofs to check using iCoq

Project Proofs iCoq Total Psel Flocq

  • 2164

22482 N/A Avg. 90.16 936.75 9.62 UniMath

  • 853

17754 N/A Avg. 35.54 739.75 4.85 Verdi

  • 4458

65413 N/A Avg. 185.75 2725.54 6.80 Psel: proof selection percentage

17 / 21

slide-42
SLIDE 42

iCoq: Regression Proof Selection for Large-Scale Verification Projects

RQ2: reduction in checking time using iCoq in CI-Env

Project CI-Env Time [s] coq makefile iCoq Flocq

  • 888.36

303.71 Avg. 37.01 12.65 UniMath

  • 12882.46

3742.88 Avg. 536.76 155.95 Verdi

  • 32528.57

3379.37 Avg. 1355.35 140.80 end-to-end time in seconds, including all phases

18 / 21

slide-43
SLIDE 43

iCoq: Regression Proof Selection for Large-Scale Verification Projects

RQ3: reduction in checking time using iCoq in LO-Env

Project LO-Env Time [s] coq makefile iCoq Flocq

  • 297.97

261.62 Avg. 12.41 10.90 UniMath

  • 3783.52

1692.33 Avg. 157.64 70.51 Verdi

  • 8157.45

3130.96 Avg. 339.89 130.45 end-to-end time in seconds, including all phases

19 / 21

slide-44
SLIDE 44

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Ratios of Total Times

Project CI-Env LO-Env Flocq 2.92 1.13 UniMath 3.44 2.23 Verdi 9.62 2.60

20 / 21

slide-45
SLIDE 45

iCoq: Regression Proof Selection for Large-Scale Verification Projects

Conclusion

technique, regression proof selection, implemented in tool, iCoq, for Coq projects, giving speedup of up to 10× in CIS, 3× locally Contact us: Ahmet Celik (ahmetcelik@utexas.edu) Karl Palmskog (palmskog@illinois.edu) Milos Gligoric (gligoric@utexas.edu) Resources: download iCoq: http://cozy.ece.utexas.edu/icoq Coq components: https://github.com/proofengineering This work was partially supported by the US National Science Foundation under Grants Nos. CCF-1438982, CCF-1566363, and CCF-1652517, and a Google Faculty Research Award.

21 / 21