GRADUATE NYC! Academy for Leaders in the Field of College - - PDF document

5/4/2012 1

GRADUATE NYC!

Academy for Leaders in the Field of College Transition

Asking the Right Questions

Graduate NYC! Academy for Leaders in the Field of College Transition

Taking a deep look at your data collection techniques and how the choices you make impact data quality and program implementation practices.

Revising Day 1: A Recap

Graduate NYC! Academy for Leaders in the Field of College Transition

Program Management & Evaluation/Assessment Data Overview: What do we collect? Why do we collect it? Introduction to a Logic Model Opportunity for Feedback & Reflection

5/4/2012 2

Day 2: What to Expect

Graduate NYC! Academy for Leaders in the Field of College Transition

Asking the Right Questions: A Case Study Data Types & Collection Strategies In-depth Look at Program/Administrative Data Data Quality, Ethics, and Security

Academy Overview

Graduate NYC! Academy for Leaders in the Field of College Transition

Day 1: Introduction to Data & Logic Models Day 2: Answering Questions with Administrative Data Day 3: The Role of Surveys, Focus Groups, Interviews, & Existing Publicly Available Data Day 4: Data Management & Analysis Day 5: Communicating through Data

A Case Study

Graduate NYC! Academy for Leaders in the Field of College Transition

What data do you need to collect for this program and why?

5/4/2012 3

What do we collect and how?

• Data Types
• Demographic Data
• Participants’ background

information

• Program Data
• Begin and end dates of

program participation, level of services provided, attendance

• Outcomes Data
• College matriculation, post-

participation employment, civic involvement

• Collection Methods
• Administrative Records
• Information/Registration Forms
• Surveys
• Interviews/Focus Groups
• Other Methods?

Graduate NYC! Academy for Leaders in the Field of College Transition Graduate NYC! Academy for Leaders in the Field of College Transition

Deciding on a Method: Considerations

• Staff resources
• Time, expertise
• Costs
• Participation incentives, dedicated data collection/entry staff
• Time constraints
• Urgent data request
• Frequency of collection
• One-time data request
• Ongoing evaluation
• Purpose of data
• Funder request
• Internal research

A Framework for Data Collection & Use:

• Utility
• Do the data address the questions that are important to your program?
• Is any of the information unnecessary or redundant?

Graduate NYC! Academy for Leaders in the Field of College Transition

5/4/2012 4

A Framework for Data Collection & Use:

• Timeliness
• Have you discussed appropriate timelines for data collection and data

entry?

• Are data collection processes designed to give you data quickly enough

to make program decisions?

Graduate NYC! Academy for Leaders in the Field of College Transition

A Framework for Data Collection & Use:

• Quality
• Are standards of quality established and communicated for all phases of

research process?

Graduate NYC! Academy for Leaders in the Field of College Transition

A Framework for Data Collection & Use:

• Security
• Is student confidentiality enforced?
• When appropriate, are students de-identified?
• Have you obtained consent and assent?

Graduate NYC! Academy for Leaders in the Field of College Transition

5/4/2012 5

A Framework for Data Collection:

Utility Timeliness Quality Security

Graduate NYC! Academy for Leaders in the Field of College Transition

Dilemmas of Practice

Utility Timeliness Quality Security

Graduate NYC! Academy for Leaders in the Field of College Transition

Example

Graduate NYC! Academy for Leaders in the Field of College Transition

Your funder requires that you submit evidence that your students are program eligible based on their enrollment in a civics class in high school. What process would you use to determine the best data collection method for this request?

5/4/2012 6

Administrative and Program Data

Graduate NYC! Academy for Leaders in the Field of College Transition

• What information?
• Existing administrative data
• Transcript data
• Program information/registration forms
• Other data collection
• Staff records the number of advising sessions a student attends
• Others
• Why collect it?
• What is the utility of the data?
• How will the data be used?
• Who will see the results?

Obtaining & Using Existing Administrative Data

Advantages

• Cost effective
• Data are already available
• Easy to store (if electronic)
• Historical data
• Construction of comparison

group

• Compelling to funders and
• utside audiences
• Tracking participants
• Data Verification

Challenges

• Lack of control over data

elements

• Varying degrees of

completeness

• Limited access
• Data agreements
• Large data files
• Confusing data codes

Graduate NYC! Academy for Leaders in the Field of College Transition

Collecting Program Data from Students

• How do you get information about your students?
• Information/Registration forms (Example: MSP)
• Utility
• Is your “instrument” answering the questions that you need to know?
• Timeliness
• How much time lapses between collection and entry?
• Are data collected at one time, or is collection spread out over a period of

time?

• Challenges
• Student-reported data
• Informal data collection
• Mixed methods
• Decentralized data collection

Graduate NYC! Academy for Leaders in the Field of College Transition

5/4/2012 7

Data Storage

• Excel or Flat file data storage
• Organized around a single table
• Easy to sort data and do analyses using functions
• Especially useful for numeric data
• Good if interested in only one type of data (eg. academic coursework)
• Difficult to manage large datasets
• Data duplication – Can have implications on data quality
• Access or Relational database
• Organized around the relationships between tables
• Keeping track of several different types of data (eg. demographic, academic

activities, non-academic activities)

• Easy to run complex queries; which combine the data from the various tables in

the database

• Other database options
• Online options

Graduate NYC! Academy for Leaders in the Field of College Transition

Data Storage Tips

• Employ uniform field layouts and formats
• Use consistent naming conventions
• Format consistently (dates, numbers, and text fields)
• Insert drop-down menus to minimize data entry error
• Restrict maximum number of characters
• Employ accountability measures
• Include fields indicating when and by whom records are added or updated
• Prevent data redundancy
• If something appears in one table, it doesn’t need to appear in another

table

• Prevent duplicate records
• Create primary keys
• Back up your data

Graduate NYC! Academy for Leaders in the Field of College Transition

Ensuring Data Quality

• Establishing a culture of quality
• Beliefs
• Good data are an integral part of teaching and learning
• Everyone involved in a program is responsible for quality data
• It is possible to create orderly information from disorderly settings
• Components
• Policies and Regulations
• Standards and Guidelines
• Training and Professional Development
• Timelines and Calendars
• Technology
• Data Entry Environment

National Forum on Education Statistics

Graduate NYC! Academy for Leaders in the Field of College Transition

5/4/2012 8

Quality Data Culture

Roles in a Quality Data Culture Importance of a Quality Data Culture Factors Affecting a Quality Data Culture Components of a Quality Data Culture

Graduate NYC! Academy for Leaders in the Field of College Transition

Data Quality Strategies

• Getting buy-in: How to get everyone excited about data?!
• Everyone should get a sense of his/her role in the big picture
• Feedback loop
• Create an open dialogue about the processes
• Encourage data entry staff to openly share what works and what doesn’t
• Share the exciting results of the research with data entry staff!
• Data checklists
• Are the data complete?
• Can you obtain missing data?
• Are data reasonable (no zeros where zero is an impossible response)?

Graduate NYC! Academy for Leaders in the Field of College Transition

Feedback Loop

Graduate NYC! Academy for Leaders in the Field of College Transition

Share findings with data entry staff Identify concerns during data entry process

Data Entry Staff Administrators/ Supervisors

5/4/2012 9

Bad Data: Why Does it Matter?

Graduate NYC! Academy for Leaders in the Field of College Transition

20 25 30 35 40 45 50 2007 2008 2009 2010

Percentage of Male Students in College Program

Bad Data: Why Does it Matter?

Graduate NYC! Academy for Leaders in the Field of College Transition

10 20 30 40 50 60 70 80 90 100 2007 2008 2009 2010

Percentage of Male Students in College Program

Bad Data: Why Does it Matter?

Graduate NYC! Academy for Leaders in the Field of College Transition

10 20 30 40 50 60 70 80 90 100 2007 2008 2009 2010

Percentage of Male Students in College Program

Female Male

5/4/2012 10

Bad Data: Why Does it Matter?

Graduate NYC! Academy for Leaders in the Field of College Transition

10 20 30 40 50 60 70 80 90 100 2007 2008 2009 2010

Percentage of Male Students in College Program

Data Ethics

(see National Forum on Educational Statistics handout)

• Integrity

1.

Demonstrate honesty, integrity, and professionalism at all times.

2.

Appreciate that, while data may represent attributes of real people, they do not describe the whole person.

3.

Be aware of applicable statutes, regulations, practices, and ethical standards governing data collection and reporting.

4.

Report information accurately and without bias.

5.

Be accountable, and hold others accountable, for ethical use

• f data.

Data Ethics

(see National Forum on Educational Statistics handout)

• Data Quality

1.

Promote data quality by adhering to best practices and

• perating standards.

2.

Provide all relevant data, definitions, and documentation to promote comprehensive understanding and accurate analysis.

• Security

1.

Treat data systems as valuable organizational assets.

2.

Safeguard sensitive data to guarantee privacy and confidentiality.

5/4/2012 11

Data Privacy

• Privacy
• What information can you collect/say
• What information can you NOT collect/say
• FERPA
• Data agreements
• IRB
• What is your role?
• Improving data security
• Student ID numbers
• Sharing sensitive data safely

Recap of the Day…