Sensemaking in Dual Artefact Tasks The Case of Business Process - PowerPoint PPT Presentation

Sensemaking in Dual Artefact Tasks – The Case of Business Process Models and Business Rules By Tianwa Chen, Shazia Sadiq and Marta Indulska ER 2020 - 39th International Conference on Conceptual Modeling November 3-6, 2020 in Vienna, Austria 0

Sensemaking in Dual Artefact Tasks – The Case of Business Process Models and Business Rules Research Background Aim of Study Study Design Scanning and attention Task Specific Information Processing Summary 1

Research Background • The widespread problem of information silos in organizations. • Knowledge workers have to navigate multiple information artefacts. • Understanding of a task will be based on both the business process model and any related business rules [1]. References: [1] Wang, W., Indulska, M., Sadiq, S.W.: Cognitive efforts in using integrated models of business processes and rules. In: CAiSE Forum. pp. 33–40 (2016) 2

Research Background Business Process and Rule Integration [2, 3] • Text annotation • Diagrammatic integration • Link integration Fig. 1. Business rules integration approaches References: [2] Knolmayer, G., Endl, R., Pfahrer, M.: Modeling processes and workflows by business rules. In: Business Process Management, pp. 16–29. Springer (2000) [3] Chen, T., Wang, W., Indulska, M., Sadiq, S.: Business process and rule integration approaches-an empirical analysis. In: International 3 Conference on Business Process Management. pp. 37–52. Springer (2018)

Research Background Sensemaking • Defined as “the process of searching for a representation and encoding data in that representation to answer task-specific questions” [4] • Individual cognitive processes: information foraging and task specific information processing [5]. • Foundational sensemaking constructs of attention (search and encoding) and memory (performance on task-specific questions). • Limited knowledge on how knowledge workers make sense of the various representations in the context of business process and business rule integration References: [4] Russell, D.M., Stefik, M.J., Pirolli, P., Card, S.K.: The cost structure of sensemaking. In: Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems. pp. 269–276 (1993) [5] Klein, G., Moon, B., Hoffman, R.R.: Making sense of sensemaking 2: A macrocognitive model. IEEE Intelligent systems 21(5), 88–92 (2006) 4

Aim of Study How knowledge workers make sense of dual artefacts (case of business process models and business rules)? What effect does integration approaches have on the efficacy of accomplishing a task that required dual artefacts, including quality of the task performance, time and effort efficiency? 5

Study Design • Controlled lab experiment • Experiment instruments: a tutorial, the treatments and a questionnaire • Participants: 75 university students with foundational knowledge in conceptual modeling (such as flowcharts, BPMN, UML or ER) • Experiment data: a pre-experiment questionnaire, eye tracking log data and task performance data • Tobii Pro TX300 eye tracker: captures data on fixations, gaze, saccades, etc, with timestamps • No limit on the experiment duration nor a word count limit on participants’ answers 6

Study Design Areas of Interest (AOI) to capture eye movements Fig. 2. Visual experiment design 7

Study Design • Treatment: Informationally equivalent models for three integration approaches with 25 participants per treatment group • Diversity in terms of constructs and coverage Table 1. Comparison of questions Questi Model Constructs Model Coverage on Q1 Sequence, AND gateways Local area Q2 Sequence, AND gateways Local area Q3 Sequence, AND gateways, Global and local XOR gateways areas 8

Study Design Searching and Encoding Task specific information Phase processing phase (Understanding phase) (Answering phase) Time commences when the commences when the participant first fixates on participant starts to type the the experiment screen . answer in the question area for the first time. Capture sensemaking behaviour: fixation durations, frequencies, task performance data, measurements related to AOI specific fixations, and transitions between AOIs. 9

Scanning and attention Fig. 3. Mean fixation duration of each question for all participants 10

Scanning and attention Fig. 4. Heat maps and AOI measures in phase 1 for best performers 11

Scanning and attention When question complexity increases • Measured through mean fixation duration. • Link representation requires less attention. • For all participants this is observed in the initial question (Q1) and again as task complexity increases in the global question (Q3). • For best performers, the lower level of attention required is again noted as task complexity increases, reflected through global question (Q3). • We also note that the transition loops are diverse in the text and link group compared with the diagrammatic group, which has the highest transition frequency between relevant area and question area . 12

Task Specific Information Processing Number of 0 1 2 3 correct answers Text 9 9 6 0 Diagrammatic 9 7 6 2 Link 4 13 7 0 (a) Number of correct answers (b) Understanding accuracy Fig. 5. Task performance 13

Task Specific Information Processing Fig. 6. Sequence of fixations in answering phase for best performers 14

Task Specific Information Processing • Link representation requires the least attention on global question Q3, indicating favorable performance as task complexity increases. • For all groups: reduced transitions (proportion of transition frequency count) in the answering phase as compared to the understanding phase, between relevant and other area, and between question and other area. • Link group: All questions show reduced transitions between rule and other area and reduced transitions between rule and relevant area. • After the understanding phase is complete, participants still engage in deep processing (number of long fixations above 500 milliseconds) of information in the answering phase. 15

Summary By using a sensemaking lens, we investigated how user behavior occurs in dual artefact tasks when the form of integrated representation of the artefacts (namely business process models and business rules) and task complexity changes. Link representation shows better task performance in terms of accuracy as well as efficiency, especially as task complexity increases. Our results provide some evidence that diagrammatic integration has better task performance on local questions in terms of accuracy, but also requires the most effort in the initial information foraging (understanding) phase. Limitation: basic constructs in business process models; the limitation of the eye tracking software limits the granularity of the AOI; We have mostly analyzed and presented the results of performers who answered the questions correctly. Our results contribute to a better understanding of the sense making processes in various settings and inform modeling practice. Our study provides a methodological contribution by offering an approach to visualize the different behaviors inherent in the two phases of sensemaking. 16

Sensemaking in Dual Artefact Tasks – The Case of Business Process Models and Business Rules Thank you! • Tianwa Chen tianwa.chen@uq.edu.au • Shazia Sadiq shazia@itee.uq.edu.au • Marta Indulska m.indulska@business.uq.edu.au ER 2020 - 39th International Conference on Conceptual Modeling November 3-6, 2020 in Vienna, Austria