A Longitudinal Case-Study Project A presentation by Dr. Christine - - PowerPoint PPT Presentation

‘Learning from the Learners’ Drawing on Experience of Researchers A Longitudinal Case-Study Project

A presentation by

• Dr. Christine Rivers

QUIC Node, University of Surrey, UK

Overview  Goals and Background of Project  Method and Analyses Descriptive Analysis using QDA Miner Conceptual Analysis using MAXQDA  Discussion and Conclusion

Goals

Understanding how users learn Tracking users over time Developing online materials Improving software training provision Contributing to software learning research

“…user experience must be studied and analysed to provide optimum solutions to meet pedagogical needs of students and teachers.” (Machado and Tao, 2007)

Background of LP

Recruiting Users Collecting data Analysis 2-Day course 5 CAQDAS packages 23 users 1month Wave1 17 responses 6 months Wave2 16 resp. 12 months Wave3 So far 9 resp. Significant decline in responses 09/2009 07/2011

Method and Analysis

Descriptive Analysis using QDA Miner (Aug 2010 – Feb 2011) Conceptual Analysis using MAXQDA (Feb 2011 –

• ngoing)

Descriptive Analysis using

• Research Questions

How do people use CAQDAS software after having an introductory 2-day training workshop and what can we learn from their experiences?

– How useful are CAQDAS packages and which tools are best? – What analytical challenges and usability issues arise? – What workarounds or suggestions do users give? – How can we improve our training to better address common issues?

• Thematic approach

– Identification of themes & recognizing patterns across the packages and time – Coding Co-occurrences, Coding Frequencies, Coding by Variable

• Analysis using QDA Miner : (open-ended questions)
• Textual analysis tool for qualitative analysis and quantitative output
• Presented at Qualitative Computing Conference in Turkey, Feb

2011

Conceptual Analysis using

• Research Questions

What can we learn from the users’ experiences to develop online materials and improve software training?

– What software related issues to users point out in the OeQ? – What suggestions to users give? – What online materials can be developed based on the findings? – How can we improve our training to better address common issues?

• Thematic approach

– Identification of themes & recognizing patterns across the packages and time – Coding Co-occurrences, Coding Frequencies, Coding by Variable

• Analysis using MAXQDA: (open-ended questions and closed questions)
• Textual analysis tool for qualitative analysis and quantitative output
• Will be published in special issue FQS 2012 together with

descriptive analysis

Descriptive Analysis using

Identifying Themes

DATA ANALYSIS Open-ended questions

Feelings Software Use Software Tools

Learning Experience Aspects of Usability

Identifying Themes: Feelings

Negative Feelings Positive Feelings Feels Frustrated when using it Helps analytical thinking process Feels confused when using it Just want to play around and familiarize Thinks he or she needs refreshing Feels it will be very useful or is already

Identifying Themes: Software Use

Stage of Use Value of Software Use Starting within days after training Increases analytical thinking Starting within weeks after training New insights Frequent use Project management Rare use Literature review Have not started yet Use of field notes

Identifying Themes: Software Tools

Coding Linking & Output Coding Structure Creating Relationships Open Coding Linking different types of data Thematic Coding Models Use of Color Networks

Recognizing Patterns

Feelings, Value of Software Use & Software Tools Wave Software THEMES VARIABLES Combined with e.g. Coding Frequency, Co-occurrences (based on case similarity analysis), Coding by Variable

Recognizing Patterns

• Focus on Research Question:

– What can we learn from the data?

• learning experience (feelings), analytical aspects

(software use), usability (software tools)

• Patterns related to: Feelings, Wave, Software
• 1. Changing of Feelings per Wave
• 2. Feelings towards Software
• 3. Changing of Feelings per Wave and Software

Recognizing Patterns

Pattern 1: Feelings and Wave

increase increase decrease

Recognizing Patterns

Pattern 1: Feelings and Software

Recognizing Patterns

Pattern 1: Feelings, Wave and Software

Recognizing Patterns

Code-Occurrences in Wave3 and Software

Confusion

Recognizing Patterns

Code-Occurrences in Wave3 and Software

Frustration

Recognizing Patterns

Frequency of Code Co-occurrences

Frustration Confusion Linking Tool (3) Tools best (3) Tools best (3) Output (3) Difficulties (2) Technical problems (3) Output (2) Need refresh (3)

Recognizing Patterns

Frequency of Code Co-occurrences

Linking Tool Tools Best Models (7) Coding (6) Helps analytical thinking process (7) Output (4) Linking different types of data (6) Linking Tools (3) Using Memos (4) Frustration (3)

Summary: What can we learn?

• The learning process is ongoing

– Later-used (more sophisticated) tools can be as challenging as getting started

• In general software viewed as useful especially

– helping analytical thinking – project management aspects (linking data)

• But there are certain tools and aspects of use

which cause confusion and frustration

– Linking tools in particular seem to be problematic

• Valued in terms of integrating data
• Confusing in terms of linking concepts

– Output options also criticised

Conceptual Analysis using

Frequency of Conceptual Codes

Code Co-occurrence of Conceptual Codes

Pos Conceptual Codes 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 1 understanding data 2 3 6 3 2 5 2 5 3 2 2 experimenting with functions 2 2 2 2 2 2 3 differences in software functions 4 too much output data 4 3 2 5 limited use 3 4 2 2 2 6 identified additional use of SW 7 interesting suggestions to improve training 2 4 2 4 2 2 8 linking findings 9 SW main tool 2 2 10 unsure about functions 6 2 4 4 4 2 2 4 8 10 6 8 4 11 trained someone else to use it 12 stopped using sw 13 presentation of findings 14 software changes practice with data 3 2 2 2 4 2 2 4 8 15 difficulties applying coding structure 2 2 2 2 2 2 16 specific method/ approach 4 2 2 2 2 2 17 Mutlimedia data 5 2 4 4 2 2 2 2 2 2 6 18 back up project 2 2 2 2 19 missing functions 2 8 2 2 3 3 3 2 20 statistical measurements 21 Gaining insight via training 2 2 22 autocoding in mind 2 23 mixed methods 24 analysing text 25 exploring data further 2 3 10 2 3 7 26 keeping record via output files 5 2 2 6 4 2 2 3 4 2 27 codes new meaning 3 2 2 8 8 2 2 2 3 7 4 28 saving time by organising data 29

• rganizing and preparing data

30 textual data 2 2 2 4 2 2 6 2 2

Conceptual Codes and OeQs

Pos OeQ 1 3 BEST TOOLS 2 ANALAPP CHANGED 3 DESCRIBE ANALAPP 4 FULL NAME 5 HELP NEED 6 ISSUES TASKS 7 LINK OEQ 8 MODELTUSEF 9 OTHER ASPECTS 10 QM SIMSTAT OEQ 11 QM TASKSWS OEQ 12 QUERY OEQ 13 Questions Coded 14 SPEC RESOUCRCES USE 15 SW CHANGE APPROACH 16 SW OEQ 17 TASK SPEC 18 TASKS MODELS 19 TEC PROB 20 TOOLS SPEC 21 TRAIN NEG 22 TRAIN POS 23 TRAINING SUGGESTIONS 24 TYPDAT OeQ 25 USE OUTPF OEQ 26 USED RESOURCES

Conceptual Codes/ OeQs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 understanding data 2 4 18 6 2 4 9 2 2 16 2 2 2 5 7 4 experimenting with functions 2 2 2 differences in software functions 2 2 2 2 2 2 2 too much output data 3 3 6 3 6 2 6 limited use 6 4 2 4 2 12 4 5 2 2 5 2 2 2 2 2 2 identified additional use of SW 9 8 4 4 2 9 2 2 2 interesting suggestions to improve training 2 2 2 2 8 4 2 2 12 4 10 4 2 2 linking findings 2 2 7 2 2 5 2 4 2 2 SW main tool 2 4 4 2 2 4 2 2 4 8 4 unsure about functions 6 2 6 6 15 2 2 10 4 4 4 4 5 10 2 4 4 2 6 14 2 trained someone else to use it 2 2 stopped using sw 2 presentation of findings 2 4 4 software changes practice with data 2 10 2 2 8 4 2 2 6 10 2 2 6 10 10 8 2 4 8 difficulties applying coding structure 2 2 2 2 2 2 specific method/ approach 4 26 26 2 20 26 4 Mutlimedia data 4 4 13 2 2 2 2 6 2 15 6 2 back up project 2 2 2 4 2 missing functions 15 10 14 2 3 4 2 10 3 18 2 2 2 2 2 2 statistical measurements 2 2 2 Gaining insight via training 2 10 2 2 2 10 6 12 12 2 2 autocoding in mind 2 2 2 2 4 2 2 2 mixed methods 4 4 2 4 analysing text 2 2 2 2 2 2 2 2 2 2 exploring data further 2 2 8 10 10 4 8 2 2 6 7 6 2 3 keeping record via output files 2 16 4 12 2 6 2 8 12 codes new meaning 2 6 2 2 6 2 2 2 4 2 2 2 4 4 saving time by organising data 2 2 4 2 4 4 2

• rganizing and preparing data

5 2 4 4 3 4 2 2 2 4 5 3 2 2 2 textual data 4 4 8 2 6 2 4 16 8 4

Conceptual Codes QDA Miner

I was disappointed to find that it is not possible to automatically retrieve and code individual words. The smallest unit that one can code is a sentence. This has caused me some difficulties which I have to work around. The complex query function is limited to two terms or term dictionaries. I found this rather limiting. I haven't probed the possibilities of QDA Miner because my project is very sequential. It is not possible to jump over steps. In particular, I hadn't figured out how to search only among already-coded text for words/phrases based on dictionaries or complex queries.

Conceptual Codes ATLAS.ti

Have also colour coded according to code family - am a bit frustrated that output of the network (e.g. to print or image file) seems to come out black and white. (wave 1) Have used colours to indicate code families and taken this through into network views. would be *incredibly* useful if this could be

• utput when printing / saving network views

as image files (seem to come through in B&W). (wave 2) It would be very useful if the network view could also give some indication of how frequently a code is used - perhaps by size, as in the 'tag cloud' visualizations sometimes used in social networking sites. (w2) Currently a bit unsure about how to generate relationships amongst the codes and what they mean. I have tried a few basic queries but have not gained any huge benefit from these - again I feel it is user error / lack of knowledge. (w2) I feel as if I don't know the half of its usefulness and do not have the confidence that I have set up the initial coding to enable me to make the best use of all the tools. The networking / linking and query tools are still rather mysterious. (w3)

Conceptual Codes NVivo

I think the output issue is the one troubling me but it is probably because I have not done a lot of that

• yet. (w2)

I am not sure it has enabled me to think in an unexpected way but it has made me more 'wordy' and I think my initial coding has been more thorough possibly a bit excessive compared to the way I coded with cards prior to using NVivo.

Conceptual Codes MAXQDA

Maybe a bit too much focus on using colour

Conceptual Codes Transana

I did not dare go into coding when I realised how reliant I would have been on the tool itself and did not feel competent enough to risk it … (w1) Yes- as I am currently using this tool to try and analyse processes and keep the analysis in context of when and where the child was at the time- this tool enables me to do that so this is a new part of practice for

• me. (w1)

However, using a Grounded Theory approach wiht line-by-line coding seems prohibitive for using transana as I found it too onerous to try and use the coding in Transana and went instead back to doing it by hand.....(w2)

Summary of conceptual analysis

• The learning process is ongoing for users

– Understanding what functions users are unsure about and providing support – Collaborating with software developers in regards to ‘missing functions’ – Being aware that the use of software is not everybody’s cup of tea and it requires skills and knowledge one has to acquire which should be considered when teaching software – CAQDAS if used adequately can widen knowledge about data and change research practice

Conclusion

• Use of CAQDAS software is an ongoing learning

process for users and trainers

– Developing online materials that focus on these ‘unsure/ missing functions’ is crucial

• E.g. Online Materials (word and visual tutorials) on Linking

Tools, Output Tools etc.

– Improving software training by changing the way we teach

• E.g. being explicit about functions in courses, offering short

webinars that cover these aspects, emphasize phone and email support in courses