Predictive Coding: The g Future of eDiscovery presenters - - PowerPoint PPT Presentation

▶
predictive coding the g future of ediscovery
SMART_READER_LITE
LIVE PREVIEW

Predictive Coding: The g Future of eDiscovery presenters - - PowerPoint PPT Presentation

Feb 03, 2023 227 likes •565 views

Predictive Coding: The g Future of eDiscovery presenters Stephanie A. Tess Blair Scott A. Milner May 15th, 2012 Introduction Pl Please note that any advice contained in this presentation is not intended or t th t d i t i d i thi

slide-1
SLIDE 1

Predictive Coding: The g Future of eDiscovery

presenters

Stephanie A. “Tess” Blair Scott A. Milner May 15th, 2012

slide-2
SLIDE 2

Introduction

Pl t th t d i t i d i thi t ti i t i t d d Please note that any advice contained in this presentation is not intended or written to be used, and should not be used, as legal advice.

slide-3
SLIDE 3

Overview Overview

  • The eDiscovery Problem
  • Evolution of a Solution
  • Predictive Coding
  • Defensibility
  • Getting Started
  • Early Results

3

slide-4
SLIDE 4

The eDiscovery Problem The eDiscovery Problem

4

slide-5
SLIDE 5

The eDiscovery Problem The eDiscovery Problem

V l

  • Volume

– The Digital Universe doubles

every 18 months every 18 months

– Corporate data volumes

increasing

– 98% of all information

generated today is stored electronically

– 2010: 988 Exabytes

(1 Exabyte = 1 trillion books) (1 Exabyte 1 trillion books)

5

slide-6
SLIDE 6

The eDiscovery Problem The eDiscovery Problem

  • Expense
  • eDiscovery market expected to hit

y p $1.5 billion by 2013

  • eDiscovery can consume 75% or

more of litigation budget

  • Primary cost driver is volume of

information subject to discovery

6

slide-7
SLIDE 7

Evolution of a Solution Evolution of a Solution

  • Early focus on driving down

Early focus on driving down cost of labor

  • Traditional Associates $$$
  • Contract Attorneys $$
  • Contract Attorneys $$
  • LPO $
  • Current focus on driving down

g volume of data subject to discovery

  • Key words

Key words

  • Analytics
  • Predictive Coding

7

slide-8
SLIDE 8

Evolution of a Solution Evolution of a Solution

Linear Review Linear Review Lim ited Lim ited Relevance/ Priority Relevance/ Priority- Linear Review Linear Review

Traditional Model

  • Custodian driven

NonLinear NonLinear Review Review

2 nd-Generation Model

  • Keyword/ topic driven

/ y / y Centric Review Centric Review

3 rd-Generation Model

  • Substance driven;

computer expedited Expensive

  • False positives
  • Lack of context
  • Manual - slow

Less Expensive

  • Docs/ hr improved
  • Limited context
  • Mostly manual - faster

computer expedited Least Expensive

  • Predictive Analytics™
  • Domain & relevance
  • Technology assisted -

a ua s o

  • Keyword driven
  • No prioritization
  • Multipass required

Mostly manual faster

  • Keyword focused
  • No prioritization
  • Multipass still required
  • Technology assisted

fastest

  • Meaning based
  • Docs prioritized
  • Multipass optional

Unnecessary Risk

  • Many false negatives
  • Many false positives
  • No consistency
  • Contract attorneys

Unnecessary Risk

  • Many false negatives
  • Many false positives
  • Limited consistency
  • No learning

Lim its Risk

  • Identifies false negatives
  • Identifies false positives
  • Maximum consistency

E t d i

8

  • Contract attorneys
  • No learning
  • Expert driven

8

slide-9
SLIDE 9

Predictive Coding Defined Predictive Coding Defined

9

slide-10
SLIDE 10

Predictive Coding Defined Predictive Coding Defined

  • What it is NOT:
  • Artificial intelligence
  • The end of attorneys reviewing documents
  • Perfect but it is far superior to human only linear
  • Perfect, but it is far superior to human-only, linear

review

10

slide-11
SLIDE 11

Predictive Coding Defined Predictive Coding Defined

  • It is also NOT:
  • Keyword or search-term filtering
  • Near duplicates, email threading
  • “Clustering”
  • Clustering
  • Concept groups
  • Relevancy ratings

11

slide-12
SLIDE 12

Predictive Coding Defined Predictive Coding Defined

  • So, what is it?
  • Computer-Assisted Review
  • Iterative, Smart, Prioritized Review
  • Faster
  • Faster
  • More Accurate
  • Less Expensive

12

slide-13
SLIDE 13

Predictive Coding Defined Predictive Coding Defined

  • Other Benefits
  • ECA
  • Quality Control
  • Privilege Analysis
  • Privilege Analysis
  • Inbound Productions

13

slide-14
SLIDE 14

Predictive Coding Workflow

Step 3

Predictive Coding Workflow

Step 2 Step 1 Step 4 p

Human Review of Computer Suggested

p

System Training on Relevant Documents Computer Suggested

p

Predictive Analytics™ to Create Review Sets Human Review Adaptive ID Cycles (Train, Suggest, Review)

p

Statistical Quality- Control Validation 14

slide-15
SLIDE 15

Iteration Tracking: Wh A W D ? When Are We Done?

T i i It ti A l i

100%

Training Iteration Analysis

60% 80% 20% 40% 0% 20% 1 2 3 4 5 6 7 8 9 10 11 12

Percent Relevant Percent NonRelevant

15

slide-16
SLIDE 16

Hypothetical: Human Review vs. P di ti C di Predictive Coding

Linear Review Predictive Coding g

2,000,000 Documents 2,000,000 Documents 227 Days 81 Days* Days Cost Days Cost* Cost $1,636,364 Cost $582,568

Predictive Coding Savings $1 053 796

*Required only 35% of the

$1,053,796

Required only 35% of the collection to be reviewed.

16

slide-17
SLIDE 17

Defensibility Defensibility

17

slide-18
SLIDE 18

Defensibility Defensibility

D f ibilit

  • Defensibility
  • Predictive coding not at issue – Humans review and determine

relevancy of computer-suggested documents assisted by Predictive C di N “bl k b ” Coding – No “black box”

  • For documents not reviewed – Issue is sampling
  • Statistical sampling widely accepted – scientific method supported by

Statistical sampling widely accepted scientific method supported by expert testimony

  • Disclosure
  • Split emerging

ithin profession on disclos re

  • Split emerging within profession on disclosure
  • Whether and when to disclose use of Predictive Coding
  • What to disclose

What to disclose

18

slide-19
SLIDE 19

Defensibility Defensibility

D f ibilit ( t )

  • Defensibility (cont.)

– Case law growing on the use of sampling techniques

  • Zubulake v. UBS Warburg, LLC, 217 F.R.D. 309 (S.D.N.Y. 2003)
  • Court accepted the use of sampling due to the prospect of having to restore

thousands of archived data tapes.

  • Mt. Hawley Ins. Co. v. Felman Prod. Inc. 2010 WL 1990555 (S.D. W.Va. May

18, 2010)

  • “Sampling is a critical quality control process that should be conducted

throughout the review.”

  • In re Seroquel Prods. Liab. Litig., 244 F.R.D. 650 (M.D. Fla. 2007)
  • Court instructed “common sense dictates that sampling and other quality

assurance techniques must be employed to meet requirements of completeness ” completeness.

19

slide-20
SLIDE 20

Defensibility Defensibility

D f ibilit ( t )

  • Defensibility (cont.)
  • Endorsement by legal community (Legal Tech 2012, NYC)
  • Judge Andrew Peck and judicial endorsement
  • October 2011 LTN Article
  • Order in Da Silva Moore v. Publicas Groupe et al. (S.D.N.Y 2011)

20

slide-21
SLIDE 21

Getting Started Getting Started

21

slide-22
SLIDE 22

Key Ingredients Key Ingredients

  • Predictive Coding requires:
  • People
  • Process
  • Technology
  • Technology

22

slide-23
SLIDE 23

People People

  • People:
  • Experienced litigators to create and QC seed set
  • Experienced discovery attorneys to drive the

predictive coding workflow, gather metrics, and measure results

  • Technicians to run the technology and manage

gy g the data

23

slide-24
SLIDE 24

Process Process

  • Process
  • Documented workflow
  • Process capable of being repeated
  • Quality control by attorneys
  • Quality control by attorneys
  • Process for gathering appropriate metrics
  • Level of confidence supported by statistics

24

slide-25
SLIDE 25

Technology Technology

  • Technology
  • Few software vendors offer true “predictive

coding” capability

  • Many are claiming they have this technology, but

are just repackaging existing technologies with new buzzwords

  • Buyer beware

25

slide-26
SLIDE 26

Earl Res lts Early Results

26

slide-27
SLIDE 27

How Morgan Lewis Uses Predictive Coding How Morgan Lewis Uses Predictive Coding

  • Increase Quality
  • Error rate reduction
  • Confidence intervals
  • Enhance Service Delivery
  • Enhance Service Delivery
  • Cost certainty
  • Time certainty
  • Demonstrate Real Value
  • Early Case Assessment
  • Discovery cost equal to value received
  • Competitive Advantage
  • Dedicated technical and legal team with expertise in predictive coding
  • Pricing competitive with all other market segments, including offshore

g p g , g

27

slide-28
SLIDE 28

Case Studies

Reduction in Volume

Review and Production of ESI

552 871 t t l d t

Review and Production of ESI

552,871 total documents

Case Study 1

  • Coded by computer = 57%

(317,000 docs)

  • Confidence interval = 95%

Confidence interval 95%

  • Defect rate = .79% or less

57% coded by computer

28

slide-29
SLIDE 29

Case Studies

Reduction in Volume (cont.)

Review and Production of ESI

254 720 t t l d t

Review and Production of ESI

254,720 total documents

Case Study 2

  • Coded by computer = 75%

(192,000 docs)

  • Confidence Interval = 95%

Confidence Interval 95%

  • Defect rate = 5% or less

75% coded by computer

29

slide-30
SLIDE 30

Case Studies

Reduction in Volume (cont.)

Review and Production of ESI

242 974 t t l d t

Review and Production of ESI

Case Study 3

242,974 total documents

  • Coded by computer = 85%

(206,000 docs)

  • Confidence Interval= 95%

Confidence Interval 95%

  • Defect rate = 5% or less

85% coded by computer

30

slide-31
SLIDE 31

Contacts Contacts

T Bl i Tess Blair Partner, Morgan, Lewis & Bockius LLP eData Practice Group eData Practice Group 215.963.5161 sblair@morganlewis.com Scott Milner Partner, Morgan, Lewis & Bockius LLP g eData Practice Group 215.963.5016 il @ l i

31

smilner@morganlewis.com

slide-32
SLIDE 32

Participants Participants

Stephanie A. Blair

Partner Morgan Lewis

Scott A. Milner

Partner Morgan Lewis Morgan Lewis P: 215.963.5161 E: sblair@morganlewis.com Morgan Lewis P: 215.963.5016 E: smilner@morganlewis.com

32

slide-33
SLIDE 33

international presence

33

Beijing Boston Brussels Chicago Dallas Frankfurt Harrisburg Houston Irvine London Los Angeles Miami New York Palo Alto Paris Philadelphia Pittsburgh Princeton San Francisco Tokyo Washington Wilmington