Operational Trials: Objective and Design Issues Wendy Bergerud - - PowerPoint PPT Presentation

Operational Trials:

Objective and Design Issues

Wendy Bergerud

Research Branch BC Min. of Forests

March 2003

WAB

Adaptive Management Cycle

Assess Evaluate Adjust Implement

Design

Monitor

WAB

Research trials vs Operational trials

Operational trials must be designed and implemented with great care and forethought if results are to be accurate and useful in making better management decisions. Operational trials are not the research trials’ “poor cousin”.

WAB

Research trials vs Operational trials

Operational trials are designed to test if treatments “work” when applied under

• perational conditions.

Research trials are designed to find treatment differences while controlling or accounting for as many other sources of variation as possible.

WAB

“Statistical analysis and interpretation are the least critical aspects of experimentation, in that if purely statistical or interpretative errors are made, the data can be reanalyzed. On the other hand, the only complete remedy for design or execution errors is repetition of the experiment.” (Hurlbert, 1984, p 189)

WAB

Trial Life Cycle

Steps: 1) Identify need for study 2) Design 3) Establish 4) Collect data and maintain site(s) 5) Analyse data 6) Communicate results 7) Wrap-up Documentation: Problem Analysis Working Plan Establishment Report Project Diary Progress and/or Final Report(s) and Extension

WAB

The Role of Statistics and Statisticians

The statistician’s expertise is particularly relevant during:

Design of the operational trial (step 2) Analysis and interpretation of collected data

(steps 5 and 6)

Research scientists can often provide this expertise at a basic level.

WAB

Questions we ask about design:

What questions are we trying to answer with this trial? What is the objective? How can we design this trial to answer these questions and yet do so within the given logical and resource restraints? What internal and external threats could undermine our confidence in the final results? How can we mitigate against these threats?

WAB

Questions we ask about analysis and interpretation:

How do we analyse this data to answer the questions? What assumptions are necessary to do this analysis and make these conclusions? And how well do we know these conclusions? (e.g. confidence limits around a mean).

WAB

“Statistical designs always involve compromises between the desirable and the possible”.

• Leslie Kish

Some assumptions and simplifications must always be made. We must understand their consequences:

Will others agree with them? How do they weaken our ability to

generalize the results?

How do they weaken any cause and effect

statements that we might like to make?

WAB

Trial Objective

Should be specific and detailed. Should describe the population under study. If looking for differences, should describe the minimum difference of practical importance. Consider a range of possible outcomes and what that means for the design and data analysis and interpretation.

Q1

WAB

Population Definition

To what population do we want our trial results to be relevant?

That is, what material do we want to study?

What population CAN we study? How does this limit our study results?

Q2

WAB

Populations

WAB

What are the treatments and what controls will we have?

Controls are very important for determining if treatments have had an effect.

Consider if controls for spatial and/or

temporal variation are required.

Q3

WAB

Effect Size

How big must the difference between the treatments or treatment and control be in

• rder for us to change our management

practice?

If our trial can’t detect a difference this small

then our study can’t provide the required evidence to change management practice [sample size requirements and power].

Q4

WAB

Consider possible outcomes

What are some of the possible outcomes and how do they affect the design and possible data analysis and interpretation?

Q5

WAB

Nuisance variables

What are some of the ‘nuisance’ variables that will affect our results? For each one, we need to determine whether to:

randomize it out; or include it in the design as a factor in the study;

• r

include it as an independent continuous

variable in the statistical model.

Q6

WAB

Study Unit Definition and Selection

How are we going to select members from the population to study? These members are our study units.

Can we select them in a random and/or

unbiased manner??

Q7

WAB

Replication

How many study units will we use? Do

• ur resources limit the amount of

replication possible? Know how to recognize pseudo- replication.

Q8

WAB

Treatment Assignment

How will the treatment and ‘nuisance’ factors be assigned to the study units?

Can we assign them in a random and

unbiased manner? or

Are they characteristics of the study units

that can only be observed (observational factors)?

Q9

WAB

Response Variables

What variables measure the response we are interested in and how will we measure them

Can the variables we are interested in be

directly measured or must we use a proxy or surrogate measure?

Q10

WAB

Subsampling of study units

Must we subsample the study units to

• btain a study unit response or can we

directly get a response for the whole unit?

If subsampling is required can we use simple

random sampling within each study unit? Or do we need some more complicated sampling scheme?

Q11

WAB

Proposed Analysis

What form of analysis do we expect to run on the data?

What design requirements does this analysis

have?

Q12

WAB

Example: Objective

Objective is to compare site preparation with duff-planting. Seems like a simple and straightforward

• bjective, but is it?

WAB

Example: Setting

Forest company must manage a large number of cutblocks that need to be planted each year (the population). Duff-planting is cheaper than site-prep. Company would prefer to use site-prep

• nly when necessary.

WAB

Example: Setting

Various attributes of cutblocks in the population are known (Factor C). Based on these attributes, we know that some cutblocks will do just fine with duff-planting and that others will need site-prep. The question applies to those cutblocks in the ‘gray area’.

WAB

Possible Outcomes: for C = 1

C =1 C=2 C=3 C=4 C=5 Factor C Growth or Survival

Duff-planting Site-prep No real difference in response

WAB

Possible Outcomes: for C = 5

C =1 C=2 C=3 C=4 C=5 Factor C Growth or Survival

Duff-planting Site-prep Gain using site-prep

WAB

Possible Outcomes: for other C

C =1 C=2 C=3 C=4 C=5 Factor C Growth or Survival

• Question is:

What happens in this gray area?

WAB

Example: Objective

How large must the gain be for management to switch practice from duff-planting to site-prep? You must determine threshold values. The question we are really interested in is: Can we use some cutblock attributes to predict when the difference in response between duff-planting and site-prep is large enough that site-prep is preferred?

WAB

Example: Decision Risks

Study can help increase percent of ‘right’ decisions, but 100% ‘right’ decisions are unattainable. Industry might prefer to err by duff- planting unless really necessary. Ministry might prefer to err by requiring site-prep unless loss due to duff-planting is demonstrated to be negligible.

WAB

Abbreviations:

RB - Randomized Block CR - Completely Randomized FRD - Factor Relationship Diagram df - Degrees of freedom

WAB

RB Design - 5 “Different” Sites

Factor C: C=1 C=2 C=3 C=4 C=5

Site 1 Site 2 Site 3 Site 4 Site 5 Design 1

WAB

RB Design - 5 “Different” Sites

Treatment is replicated. Factor C is not replicated and is confounded with site/blocks. A good “screening” design if nothing is known about the levels of Factor C.

1 1

A

1

B

2

2 2

A

3

B

4

3 3

A

5

B

6

4 4

A

7

B

8

5 5

A

9

B

10

Plot Treatment Site/Block Factor C

Design 1

WAB

Source df Error Source df Test?

Factor C B(C) Site/Block B(C) 4 Plots(CBT) Block B 4

• Treatment T

1 T x B(C) Treatment T 1 Yes T x C T x B(C) T x B(C) 4 Plots(CBT) T x B 4

• Plots(CBT)
• Analysis of Variance Table

Design 1

Many sources cannot be estimated but the most interesting test is available. Matches the FRD Final Report

WAB

RB Design - 5 “Similar” Sites

Factor C: C=3

Site 1 Site 2 Site 3 Site 4 Site 5 Design 2

WAB

RB Design - 5 “Similar” sites

Treatments are replicated within one level of Factor C. No information on different levels of Factor C - inference is limited.

3 1

A

1

B

2

2

A

3

B

4

3

A

5

B

6

4

A

7

B

8

5

A

9

B

10

Plot Treatment Site/Block Factor C

Design 2

WAB

Source df Error Source df Test?

Factor C B(C) Site/Block B(C) 4 Plots(CBT) Block B 4

• Treatment T

1 T x B(C) Treatment T 1 Yes T x C T x B(C) T x B(C) 4 Plots(CBT) T x B 4

• Plots(CBT)
• Analysis of Variance Table

Design 2

Looks like Design 1 -- but inferences apply to only one level of Factor C. Matches the FRD Final Report

WAB

RB Design - one site

Factor C: C=3

Block 2 Block 1 Block 4 Block 5 Block 3 Design 3

WAB

RB Design - one site

Pseudo-replicated - treatments are repeated only within one site. Treatment application is replicated but little else.

3 1 1

A

1

B

2

2

A

3

B

4

3

A

5

B

6

4

A

7

B

8

5

A

9

B

10

Plot Treatment Block Site Factor C

Design 3

WAB

Source df Error Source df Test?

Factor C S(C) Site S(C) B(SC) Block B(SC) 4 Plots(CSBT) Block B 4

• Treatment T

1 T x S(C) Treatment T 1 ?Yes? T x C T x S(C) T x S(C) T x B(SC) T x B(SC) 4 Plots(CSBT) T x B 4

• Plots(CSBT)
• Analysis of Variance Table

Design 3

Looks like Designs 1 & 2 -- but inferences are much more limited! Final Report Matches the FRD

WAB

Split-plot Design (CR)

Factor C: C=2 C=3

Site 1 Site 2 Site 3 Site 4 Site 5 Design 4

WAB

Split-plot Design (CR)

Factor C and treatment are replicated, but have different study units. Equal replication is ‘nice’ but not essential.

2 1

A

1

B

2

2

A

3

B

4

3

A

5

B

6

3 4

A

7

B

8

5

A

9

B

10

Plot Treatment Site/Block Factor C

Design 4

WAB

Source df Error Source df Test?

Factor C 1 B(C) Factor C 1 Yes Site/Block B(C) 3 Plots(CBT) Block B(C) 3

• Treatment T

1 T x B(C) Treatment T 1 Yes T x C 1 T x B(C) T x C 1 Yes T x B(C) 3 Plots(CBT) T x B(C) 3

• Plots(CBT)
• Analysis of Variance Table

Design 4

More interesting tests are now available. Final Report Matching FRD

WAB

Blocked Study Units

Previous designs have arranged the treatments together into ‘blocks’. This is desirable if:

Treatments can be applied to portions of

cutblocks; and

Within cutblock variability is less than

between cutblock variability so that treatment comparisons will be more precise.

WAB

Separate Study Units

Treatments may be applied to whole cutblocks if:

Operationally, treatments can only be

applied to large areas; and/or

Within cutblock variability is at least as

great as between cutblock variability (not a common assumption!).

This leads us to Completely Randomized (CR) Designs.

WAB

CR Design - 10 “Different” Sites

C=1 C=2 C=3 C=4 C=5

Sites 3 & 4 Sites 7 & 8 Sites 1 & 2 Sites 5 & 6 Sites 9 & 10

Factor C:

Design 5

WAB

CR Design - 10 “Different” sites

Treatments are replicated. Factor C is minimally replicated. Analysis may need to assume no interaction between the treatment and C

Design 5

1 A

1

1

B

2

2

2 A

3

3

B

4

4

3 A

5

5

B

6

6

4 A

7

7

B

8

8

5 A

9

9

B

10

10

Plot Site Treatment Factor C

WAB

Source df Error Source df Test?

Factor C 4 S(CT) Factor C 4 Yes? Treatment T 1 S(CT) Treatment T 1 Yes? T x C S(CT) Site S(CT) 4 Plots(CBT) Site S(CT) 4

• Plots(CBT)
• Analysis of Variance Table

Design 5

Test for Factor C assumes no interaction between the treatment and C. Test for treatment okay if Factor C is ‘random’. Final Report Matching FRD

WAB

CR Design - 10 “Similar” Sites

Factor C: C=3

Sites 3, 4 & 5 Sites 8, 9 & 10 Sites 1 & 2 Sites 6 & 7 Design 6

WAB

CR Design - 10 “Similar” Sites

Treatment is replicated. No information on different levels of Factor C - inference is limited.

Design 6

3 A

1

1

2

2

3

3

4

4

5

5

B

6

6

7

7

8

8

9

9

10

10

Plot Site Treatment Factor C

WAB

Source df Error Source df Test?

Factor C S(CT) Treatment T 1 S(CT) Treatment T 1 Yes T x C S(CT) Site S(CT) 8 Plots(CBT) Site S(CT) 8

• Plots(CBT)
• Analysis of Variance Table

Design 6

Simple One-way Design, but inferences apply to only one level of Factor C. Final Report Matching FRD

WAB

CR Design - one site

Factor C: C=3

Design 7

WAB

CR Design - one site

Pseudo-replicated - treatments are repeated only within one site. Treatment application is replicated but little else.

3 1

A

1 2 3 4 5

B

6 7 8 9 10

Plot Treatment Site/Block Factor C

Design 7

WAB

Analysis of Variance Table

Design 7

Appears to be a simple One-way Design but uses pseudo-replication. Final Report Matching FRD

Source df Error Source df Test?

Factor C S(C) Site S(C) Plots(CBT) Treatment T 1 T x S(C) Treatment T 1 Yes T x C T x S(C) T x S(C) 8 Plots(CBT) Site S(CT) 8

• Plots(CBT)

WAB

CR Design

Sites 3, 4, 5 & 6 Sites 7, 8, 9 & 10 Sites 1 & 2

Factor C: C=2 C=3

Design 8

WAB

CR Design

Factor C and treatment are replicated. Equal replication is ‘nice’ but not essential.

Design 8

2 A

1

1

2

2

3

3

B

4

4

5

5

6

6

3 A

7

7

8

8

9

9

B

10

10

Plot Site Treatment Factor C

WAB

Analysis of Variance Table

Design 8

Factorial Two-way Design with real replication. Many interesting tests. Final Report Matching FRD

Source df Error Source df Test?

Factor C 1 S(CT) Factor C 1 Yes Treatment T 1 S(CT) Treatment T 1 Yes T x C 1 S(CT) T x C 1 Yes Site S(CT) 6 Plots(CST) Site S(CT) 6

• Plots(CST)

WAB

Conclusions

The objective and design provide the foundation for the trial - if poorly done, the trial has little chance of succeeding. There is no one design that is always “best”. The “best” design depends upon the objective.

WAB

Conclusions

Get good statistical advice early, during the design. Consider how the assumptions and simplifications you’ve had to make affect the strength and applicability of your trial’s conclusions.

WAB