Data Visualization/Reporting and Valid Accountability System Design - - PowerPoint PPT Presentation

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References

Data Visualization/Reporting and Valid Accountability System Design Damian W. Betebenner

National Center for the Improvement of Educational Assessment Dover, NH

CCSSO National Conference on Student Assessment Orlando, Florida June 21st, 2011

Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References

Topics of Today’s Presentation

Validity vis-à-vis Accountability Systems. Data Quality vis-à-vis Data Use. Data Information. The Medium is the Message.

Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References

Validity & Accountability (Braun 2008)

Validity as a scientific enterprise Validity is an ongoing argument that seeks to clarify what a measurement means and to understand the limitations of each score interpretation. (Cronbach,1988) Validity as consideration of consequences Validity is an overall evaluative judgment, founded on empirical evidence and theoretical rationales, of the adequacy and appropriateness of inferences and actions based on test

• scores. (Messick, 1989)

Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References

Systemic Validity (Braun, 2008)

“Assessment practices and systems of accountability are systemically valid if they generate useful information and constructive responses that support one or more policy goals (Access, Quality, Equity, Efficiency) within an education system, without causing undue deterioration with respect to other goals.” “In validating an accountability system, the theory-of-action plays the same role as does the construct in test validation.” “Consequential validity is the ultimate criterion by which we should judge an accountability system.”

Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References

Theory of Action

What is the theory of action that turns data into information that increases educational efficacy? Articulating that theory of action is essential to validating the teacher evaluation system.

Example 1: Publishing VAM scores in the LA Times. Example 2: Incentivizing behavior through monetary compensation (see http://www.youtube.com/watch?v=u6XAPnuFjJc).

Perhaps this is what Box was referring to when he said, “All models are wrong, some are useful.”

Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References

Creating a Virtuous Cycle

Assessment Data Use Assessment Data Quality

Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References

The Data to Knowledge Continuum

Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References

From Paper to Interactive Media

We are witnessing a revolution in the way in which humans interact with media. From static (paper) to interactive (Web 2.0) Confirmatory versus Exploratory data analysis. Causal Attribution (man as researcher, intra-psychic) versus Causal Explanation (conversational, inter-psychic). Rendering verdicts about education/educator quality or promoting conversations about education/educator quality. UX (User Experience) Design: What use cases do we want to promote? (Telling versus Involving).

Tell me and I will forget, show me and I will remember, involve me and I will understand. (Confucius)

Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References

References

Braun, H. I. (2008). Viccissitudes of the validators. Presentation made at the 2008 Reidy Interactive Lecture Series, Portsmouth, NH, September, 2008.

Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design