NAEA Museum Education Division Research Marathon Practical - - PowerPoint PPT Presentation

▶
naea museum education division research marathon
SMART_READER_LITE
LIVE PREVIEW

NAEA Museum Education Division Research Marathon Practical - - PowerPoint PPT Presentation

Sep 23, 2022 343 likes •1.36k views

NAEA Museum Education Division Research Marathon Practical Strategies for Getting Started and Making it Happen #NAEAMusEd16 @NAEAMusEd Developing a Shared Vocabulary Visitor Studies The interdisciplinary study of human experiences within

slide-1
SLIDE 1

NAEA Museum Education Division Research Marathon

Practical Strategies for Getting Started and Making it Happen

#NAEAMusEd16 @NAEAMusEd

slide-2
SLIDE 2

Developing a Shared Vocabulary

Evaluation

  • Tests feasibility, effectiveness, impact,
  • r value
  • Seeks to inform decisions, policies, or

program development Uses social science research methods

Research

  • Tests hypotheses
  • Seeks to add knowledge to/develop

theory in a given field

  • May have an experimental –

“treatment-control” - design

Visitor Studies

“The interdisciplinary study of human experiences within informal learning environments. Visitor studies. . .

  • Follow rigorous methods that adhere to the standards of the social sciences
  • Draw from and contribute to the theory and practice of social science
  • Are designed to improve the practices of learning in informal environments.”*

* Visitor Studies Association, http://www.visitorstudies.org/glossary-of-terms, and

The Definitions Project, http://www.definitionsproject.com/definitions/media/definitions_list_0107.pdf

slide-3
SLIDE 3

Presenters

Lynn Courtney Head of Planning & Evaluation, Museum of Fine Arts Boston Cecilia Garibay Principal, Garibay Group Jennifer Wild Czajkowski VP Learning & Interpretation, Detroit Institute of Arts

slide-4
SLIDE 4

Session Goals

Participants will...

  • understand a variety of research / evaluation models for a

variety of situations

  • feel confident and inspired to engage in next-level

conversations about research and evaluation in their institutions

  • be intrigued to attend the rest of the marathon!
slide-5
SLIDE 5

Lynn Courtney

Head of Planning and Evaluation

#NAEAMusEd16 @NAEAMusEd

slide-6
SLIDE 6

Developing a Shared Vocabulary

Evaluation

  • Tests feasibility, effectiveness, impact,
  • r value
  • Seeks to inform decisions, policies, or

program development Uses social science research methods

Research

  • Tests hypotheses
  • Seeks to add knowledge to/develop

theory in a given field

  • May have an experimental –

“treatment-control” - design

Visitor Studies

“The interdisciplinary study of human experiences within informal learning environments. Visitor studies. . .

  • Follow rigorous methods that adhere to the standards of the social sciences
  • Draw from and contribute to the theory and practice of social science
  • Are designed to improve the practices of learning in informal environments.”*

* Visitor Studies Association, http://www.visitorstudies.org/glossary-of-terms, and

The Definitions Project, http://www.definitionsproject.com/definitions/media/definitions_list_0107.pdf

slide-7
SLIDE 7

In-House Evaluation from Soup to Nuts

7

A Case Study:

Meeting with the “clients” Evaluation questions: ➲How satisfied are Playdates families? ➲ What works? What doesn’t? ➲ What are toddlers learning? ➲ What should we change?

slide-8
SLIDE 8

In-House Evaluation from Soup to Nuts

8

A Case Study:

Designing the study Exit Interviews? Who? Online surveys? Observations Survey Cards? When? How many?

slide-9
SLIDE 9

In-House Evaluation from Soup to Nuts

9

A Case Study:

Designing the study Parents and caregivers Online surveys Observations Twice a month At least 75

slide-10
SLIDE 10

In-House Evaluation from Soup to Nuts

10

A Case Study:

Collecting the Data ➲Onsite recruitment ➲ Online survey link sent within 1 week of visit ➲ 2 follow-up emails

slide-11
SLIDE 11

In-House Evaluation from Soup to Nuts

11

A Case Study:

Collecting the Data

Timeline

Planning & survey design Data collection Data analysis Final report September October April May June

slide-12
SLIDE 12

In-House Evaluation from Soup to Nuts

12

A Case Study:

Data analysis & reporting: what did we learn? ➲69% of Playdates children are 2 years old or younger

slide-13
SLIDE 13

In-House Evaluation from Soup to Nuts

13

A Case Study:

Data analysis & reporting: what did we learn? ➲High satisfaction levels ➲But. . .Younger children were not always engaged; elements were not always well-integrated; parents want more help with art making

slide-14
SLIDE 14

In-House Evaluation from Soup to Nuts

14

A Case Study:

Data Analysis & Reporting: what did we learn? ➲We identified indicators of learning in young children

“My daughter STILL talks about the crown she made at an MFA Playdates last summer. She thinks the Chihuly sculpture looks like celery.” “She's young, but she was looking and pointing at the artwork.” “The next day my daughter sang every line of a song the instructor sang!”

slide-15
SLIDE 15

In-House Evaluation from Soup to Nuts

15

A Case Study:

Disseminating Results ➲Meet with program staff: give them usable results ➲Evaluation Highlights ➲Present at Museum staff meetings

slide-16
SLIDE 16

In-House Evaluation from Soup to Nuts

16

A Case Study: What did we change? ➲Professional development for gallery educators ➲Design for 2-year-olds and adapt for older children ➲ Work with the guards ➲ Stronger connections: theme, art, gallery activity, art making

slide-17
SLIDE 17

Resources and Capacity

17

With… ➲ 1 full-time evaluator ➲ 2 work-study students at 14 hours per week ➲ 1 paid summer intern at 14 hours per week ➲ 1 summer Evaluation Assistant at 14 hours per week Each year we conduct… ➲ 2 – 4 major studies ➲ 4 – 6 smaller studies ➲ 1 major planning process ➲ Various ad hoc projects

slide-18
SLIDE 18

When to Call in an External Evaluator

18

➲When there is funding ➲When you need specialized skills ➲ To boost credibility ➲When there is a large, ongoing project that is beyond your capacity

slide-19
SLIDE 19

Expanding Uses of Evaluation

Cecilia Garibay

Principal, Garibay Group

#NAEAMusEd16 @NAEAMusEd

slide-20
SLIDE 20

Front-End Formative Remedial Summative

Concept Preview Launch Development Post-Opening Full Implementation

slide-21
SLIDE 21

Listening to Community Voices

  • Evaluation as a process

for creating accessible, relevant places and experiences

  • Evaluation as deeply

collaborative process

slide-22
SLIDE 22
  • How will what we learn help you make better decisions

for the specific project/your organization?

  • How will this study help you better serve your visitors

and community members?

slide-23
SLIDE 23
slide-24
SLIDE 24
slide-25
SLIDE 25
  • Offered deeper interpretation and historical context.
  • Made changes to placement of artwork from original plan.
  • Added response/reflection areas.
  • Issues raised by participants became talking points in
  • interpretation. (A shift in who gets to tell the story!)
slide-26
SLIDE 26
slide-27
SLIDE 27

.

  • Examine processes to

identify factors that lead to

  • utcomes
  • Articulate interrelationships
  • Develop models
slide-28
SLIDE 28

“It was a wonderful experience. We can talk about one picture with other classmates and we can try to discuss pictures with the vocabulary we had…. Somehow I feel very connected to each

  • ther.”

“When you have friends around you it helps to motivate [you] to talk about the the picture… sometimes you want to add something…and you also learn from what

  • ther people say.”

“I found that when you are sharing ideas about art it doesn’t matter the right or wrong because it’s your feelings and it’s your opinion. And through this program— I think we feel free—I don’t worry about right or wrong.” Empowerment Sharing English Skills Community

slide-29
SLIDE 29
slide-30
SLIDE 30

Interesting Uninteresting Familiar Unfamiliar

I don’t know what this is but want to know more.

I know what this is and want

to know more. I don’t know what this is but don’t want to know more.

I know what this is and don’t want to

know more.

slide-31
SLIDE 31

Characteristics that afford engagement

  • Scale
  • Evidence of age
  • Details
  • Intriguing mysteries
  • Tangible expression of a time
  • Relationship to a real person
  • Readily identifiable, but unique
  • Representative of a specific Chicago place
  • Connection to a well-known event/story

Characteristics that impede engagement

  • Ordinariness
  • Dense, small type (documents)
  • Archaic type (documents)
  • Formal language or jargon
  • Small size
  • Difficult topics

Select opportunities based

  • n characteristics
  • Discuss conservation process
  • Help explore details
  • Discuss how to solve mysteries
  • Create a feeling of time

Select strategies to

  • vercome impediments
  • Emphasize uniqueness of ordinary objects
  • Highlight important parts of document
  • Translate formal language
  • Call attention to small items
  • Provide interpretation related to difficult topics
slide-32
SLIDE 32
  • These kinds of evaluations are collaborative endeavors

between museum team, external evaluators, visitors, and community members.

  • These uses of evaluation provide significant opportunities

to deepen organizational learning beyond one program/exhibit.

  • They require a willingness to engage in a deep, reflective

process.

slide-33
SLIDE 33

Jennifer Wild Czajkowski

VP Learning & Interpretation

#NAEAMusEd16 @NAEAMusEd

slide-34
SLIDE 34

Research with Universities

Possible partnership formats:

  • find academic partner who teaches a research class
  • course buyout: pay university to keep instructor out of class
slide-35
SLIDE 35

Research with Universities: Finding a Partner

NAEA resources for museum / higher ed partnerships:

  • Professional Learning Through Research working group
  • Higher Ed leadership/membership
  • Research Commission Interactive Cafe

Future projects?

  • NAEA database of higher ed members with research interests
slide-36
SLIDE 36

Action Research

  • inquiry and research
  • focused efforts to improve performance
  • designed, conducted, analyzed by practitioners

Reflective practice for continual improvement

slide-37
SLIDE 37
slide-38
SLIDE 38

Action Research at the DIA

10 gallery teachers giving tours for 25,000 students

  • rapid learning curve
  • culture of collaboration

and coaching

  • student engagement goals
slide-39
SLIDE 39

Action Research at the DIA

  • 1. Plan: ‘pair and share’ will involve

more students, result in richer interpretation

  • a variety of ideas shared
  • changing direction or scaffolding
  • continued talking as we leave the object
slide-40
SLIDE 40

Action Research at the DIA

  • 2. Act: implement ‘pair & share’
  • try different prompts
  • try with different objects
  • try with different ages
slide-41
SLIDE 41

Action Research at the DIA

  • 3. Observe ‘pair and share’
  • a coach observes, takes notes
  • videorecording
slide-42
SLIDE 42

Action Research at the DIA

  • 4. Reflect on hypothesis

What happened?

  • journaling
  • reflection forms
  • group review sessions
  • peer coaching
  • impromptu conversations
slide-43
SLIDE 43

Action Research at the DIA

  • 5. Plan Again

What does this look like?

  • adopt success
  • try a new activity
slide-44
SLIDE 44

Action Research at the DIA

The team’s learning curve:

  • started with reflection forms
  • then journaling
  • formal group reviews
  • coaching
  • informal conversation; fluid, ongoing practice
slide-45
SLIDE 45

Action Research

Challenges:

  • can get too loose; requires coordination and leadership
  • requires self-directed, disciplined staff who buy-in

Advantages:

  • flexible, scalable
  • individual or group project
  • builds staff skills, continuous learning, empowers staff
  • pen-ended
slide-46
SLIDE 46

Questions? Contact Information

Lynn Courtney lcourtney@mfa.org 617/369-3376 Jennifer Wild Czajkowski czajkowski@dia.org 313/833-5989 @jwildczaj Cecilia Garibay cgaribay@garibaygroup.com 773/327-9006 @garibaygroup

slide-47
SLIDE 47

Where to Find Post-Session Online E-Resource http://bit.ly/1pjcR12

#NAEAMusEd16 @NAEAMusEd

slide-48
SLIDE 48

NAEA Museum Education Division Research Marathon

What Can Research Do For You?

# NAEAMusEd16 @NAEAMusEd

slide-49
SLIDE 49

Developing a Shared Vocabulary

Evaluation

  • Tests feasibility, effectiveness, impact,
  • r value
  • Seeks to inform decisions, policies, or

program development Uses social science research methods

Research

  • Tests hypotheses
  • Seeks to add knowledge to/develop

theory in a given field

  • May have an experimental –

“treatment-control” - design

Visitor Studies

“The interdisciplinary study of human experiences within informal learning environments. Visitor studies. . .

  • Follow rigorous methods that adhere to the standards of the social sciences
  • Draw from and contribute to the theory and practice of social science
  • Are designed to improve the practices of learning in informal environments.”*

* Visitor Studies Association, http://www.visitorstudies.org/glossary-of-terms, and

The Definitions Project, http://www.definitionsproject.com/definitions/media/definitions_list_0107.pdf

slide-50
SLIDE 50

Deborah Randolph Curator of Education, Southeastern Center for Contemporary Art, Winston-Salem, NC Paula Lynn Manager of Planning & Evaluation, National Gallery of Art Jennifer Beradino Object-based Learning Manager, Museum of Fine Arts Houston

Presenters

slide-51
SLIDE 51

SESSION GOALS

Participants will…

  • gain practical tips for conducting research
  • feel confident engaging with various steps of doing research
  • be inspired to incorporate research into their practice and

use research as advocacy

slide-52
SLIDE 52

Literature Review

Deborah Randolph

Curator of Education Southeastern Center for Contemporary Art (SECCA)

#NAEAMusEd16 @NAEAMusEd

slide-53
SLIDE 53

Literature Review

What is a literature review?

▪ A literature review surveys books, scholarly articles and other sources relevant to a particular subject or research problem.

Why is a literature review important to my research?

▪ Places your research within the context

  • f existing literature.

▪ Reviewing and writing throughout the process helps articulate your ideas.

slide-54
SLIDE 54

Literature Review

Where do I look? Resources for a quick search

▪ ArtsEd Search (Arts Education Partnership) ▪ University databases ▪ JSTOR membership

How do I search?

▪ Key word searches ▪ Authors (experts in the field)

slide-55
SLIDE 55

Literature Review

What are you reading?

▪ Journals ▪ Newspapers and magazines ▪ Non-fiction books

What are they reading?

▪ Bibliographies ▪ In-text references ▪ Literature reviews

slide-56
SLIDE 56

Literature Review

How can I complete this review?

▪ Narrow your search ▪ Limit your review to current studies (3-5 years) ▪ Stay on topic (no bird walking) ▪ Take notes

What and how many do I include?

▪ Focus on relevance ▪ Choose quality over quantity in selecting resources ▪ Scale to type: Action research, long-term

slide-57
SLIDE 57

Literature Review

What do I do now?

▪ Organize your references ▪ Write your review ▪ Place your research in context ▪ Tell your story

slide-58
SLIDE 58

Strategies For Engaging With Your Data

PAULA LYNN

Manager of Planning & Evaluation National Gallery of Art

Paula Lynn

Manager of Planning & Evaluation, National Gallery of Art

#NAEAMusEd16 @NAEAMusEd

slide-59
SLIDE 59

Preparing for Data Collection

Learn from your literature review Tune your instrument

slide-60
SLIDE 60

Getting Started with Data

Create a plan to organize Check in with your data

slide-61
SLIDE 61

Working with Data - Big Picture

Read early... The whole and its parts Know the code

slide-62
SLIDE 62

Working with Data - Fine Grain

...read often Emergent vs. predetermined codes “Data, I have a question.”

slide-63
SLIDE 63

How Much Time* Does It Take?

*depends on sample size, methods, etc.

Preparing Collecting Reading and coding Asking questions

slide-64
SLIDE 64
slide-65
SLIDE 65

Using Your Findings To Inform Your Work Jennifer Beradino

Object-based Learning Manager Museum of Fine Arts, Houston

#NAEAMusEd16 @NAEAMusEd

slide-66
SLIDE 66

Using Your Findings To Inform Your Work

Roadmap to Success

  • Advocating Internally
  • Communicating with External

Stakeholders

  • Designing your Daily Practice
slide-67
SLIDE 67

Using Your Findings To Inform Your Work

Advocating Internally

  • Acting as your most powerful advocate
  • Harnessing the research to lead decision

making

  • Lead the charge!
slide-68
SLIDE 68

Using Your Findings To Inform Your Work

Communicating with Stakeholders

Funders, Parents, School Administration, Community Partners ▪ Know the goals and expertise of your

stakeholders

▪ Sharing Evidence through a variety of

lenses

▪ Research Participants as avenues to

external stakeholders

slide-69
SLIDE 69

Using Your Findings To Inform Your Work

Designing your Daily Practice

▪ Be open to change ▪ Research is ongoing ▪ Sharing your research

slide-70
SLIDE 70

Questions? Contact Information

Emily Jennings ejennings@famsf.org 415-750-3522 Sara Egan segan@isgm.org 618-278-5162 @SaraBethEgan Deborah Randolph Deborah.Randolph@ncdcr.gov 336-397-2108 @SECCAcontempart Paula Lynn P-Lynn@nga.gov 202-842-6027 @MuseumBug Jennifer Beradino jberadino@mfah.org 713-639-7588 @MFAH

slide-71
SLIDE 71

Where to Find Post-Session Online E-Resource http://bit.ly/1pjcR12

#NAEAMusEd16 @NAEAMusEd

slide-72
SLIDE 72

NAEA Museum Education Division Research Marathon

Developing a Museum-Wide Culture of Learning

·

#NAEAMusEd16 @NAEAMusEd

slide-73
SLIDE 73

Developing a Shared Vocabulary

Evaluation

  • Tests feasibility, effectiveness, impact,
  • r value
  • Seeks to inform decisions, policies, or

program development Uses social science research methods

Research

  • Tests hypotheses
  • Seeks to add knowledge to/develop

theory in a given field

  • May have an experimental –

“treatment-control” - design

Visitor Studies

“The interdisciplinary study of human experiences within informal learning environments. Visitor studies. . .

  • Follow rigorous methods that adhere to the standards of the social sciences
  • Draw from and contribute to the theory and practice of social science
  • Are designed to improve the practices of learning in informal environments.”*

* Visitor Studies Association, http://www.visitorstudies.org/glossary-of-terms, and

The Definitions Project, http://www.definitionsproject.com/definitions/media/definitions_list_0107.pdf

slide-74
SLIDE 74

Presenters

Wendy Wolf

Deputy Director for Learning and Visitor Experience, Vizcaya Museum and Gardens

Jessica Sack

Jan and Frederick Mayer Senior Associate Curator of Public Education, Yale University Art Gallery

Elizabeth Williams

John Walsh Senior Fellow in Education, Yale University Art Gallery

slide-75
SLIDE 75

SESSION GOALS

Participants will…

  • explore how to develop or leverage interdepartmental

collaboration

  • identify and examine strategic questions
  • consider what a culture of learning might look like
  • discuss the capacities and infrastructure that sustain a

culture of learning

slide-76
SLIDE 76

Vizcaya Museum and Gardens Institutional Research & Reflective Practice

Wendy Wolf

Deputy Director for Learning and Visitor Experience, Vizcaya Museum and Gardens

#NAEAMusEd16 @NAEAMusEd

slide-77
SLIDE 77

Developing a Shared Vocabulary

Evaluation

  • Tests feasibility, effectiveness, impact,
  • r value
  • Seeks to inform decisions, policies, or

program development Uses social science research methods

Research

  • Tests hypotheses
  • Seeks to add knowledge to/develop

theory in a given field

  • May have an experimental –

“treatment-control” - design

Visitor Studies

“The interdisciplinary study of human experiences within informal learning environments. Visitor studies. . .

  • Follow rigorous methods that adhere to the standards of the social sciences
  • Draw from and contribute to the theory and practice of social science
  • Are designed to improve the practices of learning in informal environments.”*

* Visitor Studies Association, http://www.visitorstudies.org/glossary-of-terms, and

The Definitions Project, http://www.definitionsproject.com/definitions/media/definitions_list_0107.pdf

slide-78
SLIDE 78

Vizcaya Museum and Gardens Institutional Research & Reflective Practice

Visitor Studies Through Evaluation

  • Staff work in partnership with external consultants to develop visitor surveys,

facilitate Think Tanks (with staff and external stakeholders), facilitate Focus Groups, analyze data and submit reports on process and findings.

  • External consultant provided a qualitative analysis of 5 years of program and visitor

surveys.

  • External consultant provided a qualitative analysis of 5 years of written student

feedback to assess cognitive and affective outcomes.

slide-79
SLIDE 79

Visitor Studies

slide-80
SLIDE 80

We are doing a great job. Yes we are sure.

slide-81
SLIDE 81

How do we get residents to return? Or become members?

slide-82
SLIDE 82

Visitation has doubled. Yes we are sure.

slide-83
SLIDE 83

Why has visitation doubled?

slide-84
SLIDE 84

Infrastructure. We have it.

slide-85
SLIDE 85

Collaboration. We have it.

slide-86
SLIDE 86

Develop an approach and tools to guide us in learning more about

  • ur visitors and their experiences

so that we can make informed decisions and achieve our strategic plan goals

slide-87
SLIDE 87

Where will I find the time?

slide-88
SLIDE 88

We will pace ourselves.

slide-89
SLIDE 89

A small planning group will take a close look.

slide-90
SLIDE 90

Discussions at regularly scheduled meetings.

slide-91
SLIDE 91

Success will look like this…

slide-92
SLIDE 92

TEACHING LAB at the YALE UNIVERSITY ART GALLERY

Jessica Sack

Jan and Frederick Mayer Senior Associate Curator of Public Education, Yale University Art Gallery

Elizabeth Williams

John Walsh Senior Fellow in Education, Yale University Art Gallery

#NAEAMusEd16 @NAEAMusEd

slide-93
SLIDE 93

Yale University Art Gallery Institutional Research & Reflective Practice

Visitor Studies through Research and Reflection

  • Teaching Lab: An interdepartmental platform for reflecting,

experimenting and collaborating on issues and ideas related to teaching and learning.

  • Sessions present a big idea to the group for conversation and

workshopping

  • This collaborative framework has resulted in more formal evaluation

projects as well as plans to expand to entire Museum

slide-94
SLIDE 94
slide-95
SLIDE 95
slide-96
SLIDE 96
slide-97
SLIDE 97

Critical Conversations & Key Essential Items

  • What big ideas do you want to explore?
  • What overarching questions have surfaced (in your

institution/work) that affect multiple departments?

  • For small group discussions

1. Review opportunities: consider actual and perceived barriers 2. What three action steps could you take to advance thinking about this in your institution, to advance this goal in your institution or to scale up current approaches? 3. How would you assess these action steps? How will you know it’s working?

slide-98
SLIDE 98

Discoveries

  • What new ideas or new tools did you discover?
  • Did any patterns emerge?
slide-99
SLIDE 99

Questions? Contact Information

Jessica Sack jessica.sack@yale.edu Elizabeth Williams elizabeth.j.williams@yale.edu Wendy Wolf wendy.wolf@vizaya.org

slide-100
SLIDE 100

Where to Find Post-Session Online E-Resource http://bit.ly/1pjcR12

#NAEAMusEd16 @NAEAMusEd