A Longitudinal Case-Study Project A presentation by Dr. Christine - PowerPoint PPT Presentation

‘Learning from the Learners’ Drawing on Experience of Researchers A Longitudinal Case-Study Project A presentation by Dr. Christine Rivers QUIC Node, University of Surrey, UK

Overview  Goals and Background of Project  Method and Analyses Descriptive Analysis using QDA Miner Conceptual Analysis using MAXQDA  Discussion and Conclusion

“…user experience must be studied and analysed to provide optimum solutions to meet pedagogical needs of Goals students and teachers.” (Machado and Tao, 2007) Understanding how users learn Tracking users over time Developing online materials Improving software training provision Contributing to software learning research

Background of LP 09/2009 07/2011 Recruiting Users Collecting data Analysis 2-Day course 1month 6 months 12 months 5 CAQDAS Wave1 Wave2 Wave3 packages 23 users 17 responses 16 resp. So far 9 resp. Significant decline in responses

Method and Analysis Descriptive Analysis using QDA Miner (Aug 2010 – Feb 2011) Conceptual Analysis using MAXQDA (Feb 2011 – ongoing)

Descriptive Analysis using • Research Questions How do people use CAQDAS software after having an introductory 2-day training workshop and what can we learn from their experiences? – How useful are CAQDAS packages and which tools are best? – What analytical challenges and usability issues arise? – What workarounds or suggestions do users give? – How can we improve our training to better address common issues? • Thematic approach – Identification of themes & recognizing patterns across the packages and time – Coding Co-occurrences, Coding Frequencies, Coding by Variable • Analysis using QDA Miner : (open-ended questions) • Textual analysis tool for qualitative analysis and quantitative output • Presented at Qualitative Computing Conference in Turkey, Feb 2011

Conceptual Analysis using • Research Questions What can we learn from the users’ experiences to develop online materials and improve software training? – What software related issues to users point out in the OeQ? – What suggestions to users give? – What online materials can be developed based on the findings? – How can we improve our training to better address common issues? • Thematic approach – Identification of themes & recognizing patterns across the packages and time – Coding Co-occurrences, Coding Frequencies, Coding by Variable • Analysis using MAXQDA: (open-ended questions and closed questions) • Textual analysis tool for qualitative analysis and quantitative output • Will be published in special issue FQS 2012 together with descriptive analysis

Descriptive Analysis using

Identifying Themes Feelings Learning Aspects of Experience Usability DATA ANALYSIS Open-ended questions Software Use Software Tools

Identifying Themes: Feelings Negative Feelings Positive Feelings Feels Frustrated when using it Helps analytical thinking process Feels confused when using it Just want to play around and familiarize Thinks he or she needs Feels it will be very useful or refreshing is already

Identifying Themes: Software Use Stage of Use Value of Software Use Starting within days after Increases analytical thinking training Starting within weeks New insights after training Frequent use Project management Rare use Literature review Have not started yet Use of field notes

Identifying Themes: Software Tools Coding Linking & Output Coding Structure Creating Relationships Open Coding Linking different types of data Thematic Coding Models Use of Color Networks

Recognizing Patterns Feelings, Value of Software Use & THEMES Software Tools Combined with VARIABLES Software Wave e.g. Coding Frequency, Co-occurrences (based on case similarity analysis), Coding by Variable

Recognizing Patterns • Focus on Research Question: – What can we learn from the data? • learning experience (feelings), analytical aspects (software use), usability (software tools) • Patterns related to: Feelings, Wave, Software 1. Changing of Feelings per Wave 2. Feelings towards Software 3. Changing of Feelings per Wave and Software

Recognizing Patterns Pattern 1: Feelings and Wave increase increase decrease

Recognizing Patterns Pattern 1: Feelings and Software

Recognizing Patterns Pattern 1: Feelings, Wave and Software

Recognizing Patterns Code-Occurrences in Wave3 and Software Confusion

Recognizing Patterns Code-Occurrences in Wave3 and Software Frustration

Recognizing Patterns Frequency of Code Co-occurrences Frustration Confusion Linking Tool (3) Tools best (3) Tools best (3) Output (3) Difficulties (2) Technical problems (3) Output (2) Need refresh (3)

Recognizing Patterns Frequency of Code Co-occurrences Linking Tool Tools Best Models (7) Coding (6) Helps analytical thinking Output (4) process (7) Linking different types of Linking Tools (3) data (6) Using Memos (4) Frustration (3)

Summary: What can we learn? • The learning process is ongoing – Later-used (more sophisticated) tools can be as challenging as getting started • In general software viewed as useful especially – helping analytical thinking – project management aspects (linking data) • But there are certain tools and aspects of use which cause confusion and frustration – Linking tools in particular seem to be problematic • Valued in terms of integrating data • Confusing in terms of linking concepts – Output options also criticised

Conceptual Analysis using

Frequency of Conceptual Codes

Code Co-occurrence of Conceptual Codes Conceptual Codes 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Pos understanding data 2 3 6 3 2 5 2 5 3 2 1 experimenting with functions 2 2 2 2 2 2 2 differences in software functions 3 too much output data 4 3 2 4 limited use 3 4 2 2 2 5 identified additional use of SW 6 interesting suggestions to improve training 2 4 2 4 2 2 7 linking findings 8 SW main tool 2 2 9 unsure about functions 6 2 4 4 4 2 2 4 8 10 6 8 4 10 trained someone else to use it 11 stopped using sw 12 presentation of findings 13 software changes practice with data 3 2 2 2 4 2 2 4 8 14 difficulties applying coding structure 2 2 2 2 2 2 15 specific method/ approach 4 2 2 2 2 2 16 Mutlimedia data 5 2 4 4 2 2 2 2 2 2 6 17 back up project 2 2 2 2 18 missing functions 2 8 2 2 3 3 3 2 19 statistical measurements 20 Gaining insight via training 2 2 21 autocoding in mind 2 22 mixed methods 23 analysing text 24 exploring data further 2 3 10 2 3 7 25 keeping record via output files 5 2 2 6 4 2 2 3 4 2 26 codes new meaning 3 2 2 8 8 2 2 2 3 7 4 27 saving time by organising data 28 organizing and preparing data 29 textual data 2 2 2 4 2 2 6 2 2 30

Conceptual Codes and OeQs Pos OeQ Conceptual Codes/ OeQs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 understanding data 2 4 18 6 2 4 9 2 2 16 2 2 2 5 7 4 1 3 BEST TOOLS experimenting with functions 2 2 2 2 ANALAPP CHANGED differences in software functions 2 2 2 2 2 2 2 3 DESCRIBE ANALAPP too much output data 3 3 6 3 6 2 6 4 FULL NAME limited use 6 4 2 4 2 12 4 5 2 2 5 2 2 2 2 2 2 5 HELP NEED identified additional use of SW 9 8 4 4 2 9 2 2 2 6 ISSUES TASKS interesting suggestions to improve training 2 2 2 2 8 4 2 2 12 4 10 4 2 2 7 LINK OEQ linking findings 2 2 7 2 2 5 2 4 2 2 8 MODELTUSEF SW main tool 2 4 4 2 2 4 2 2 4 8 4 9 OTHER ASPECTS unsure about functions 6 2 6 6 15 2 2 10 4 4 4 4 5 10 2 4 4 2 6 14 2 10 QM SIMSTAT OEQ trained someone else to use it 2 2 11 QM TASKSWS OEQ stopped using sw 2 12 QUERY OEQ presentation of findings 2 4 4 13 Questions Coded software changes practice with data 2 10 2 2 8 4 2 2 6 10 2 2 6 10 10 8 2 4 8 SPEC RESOUCRCES difficulties applying coding structure 2 2 2 2 2 2 14 USE specific method/ approach 4 26 26 2 20 26 4 SW CHANGE 15 APPROACH Mutlimedia data 4 4 13 2 2 2 2 6 2 15 6 2 16 SW OEQ back up project 2 2 2 4 2 17 TASK SPEC missing functions 15 10 14 2 3 4 2 10 3 18 2 2 2 2 2 2 18 TASKS MODELS statistical measurements 2 2 2 19 TEC PROB Gaining insight via training 2 10 2 2 2 10 6 12 12 2 2 20 TOOLS SPEC autocoding in mind 2 2 2 2 4 2 2 2 21 TRAIN NEG mixed methods 4 4 2 4 22 TRAIN POS analysing text 2 2 2 2 2 2 2 2 2 2 TRAINING exploring data further 2 2 8 10 10 4 8 2 2 6 7 6 2 3 23 SUGGESTIONS keeping record via output files 2 16 4 12 2 6 2 8 12 24 TYPDAT OeQ codes new meaning 2 6 2 2 6 2 2 2 4 2 2 2 4 4 25 USE OUTPF OEQ saving time by organising data 2 2 4 2 4 4 2 26 USED RESOURCES organizing and preparing data 5 2 4 4 3 4 2 2 2 4 5 3 2 2 2 textual data 4 4 8 2 6 2 4 16 8 4