Presentation Mobile: experiments on Android and iOS by Elisabeth - - PDF document

Technische Ondersteuning Psychologie __________________________________________________________________________________ 1

Presentation Mobile: experiments on Android and iOS

by Elisabeth Nieuwburg Last updated: June 2017

TABLE OF CONTENTS

• 1. INTRODUCTION ……………………………………………………………………… 2
• 2. TOUCH ……………………………………………………………………………….… 2
• 3. TEST MODE ………………………………………………………………………….... 5
• 4. PACKAGING EXPERIMENTS ……………………………………………………….. 5
• 5. HOSTING EXPERIMENTS ………………………………………………………….... 6
• 7. DOWNLOADING EXPERIMENTS IN THE APP………………………………….…. 6
• 8. OUTPUT ……………………………………………………………………………… .. 6
• 9. RESTRICTIONS FOR CROSS-PLATFORM EXPERIMENTS ……………………… 7
• 10. ADAPTING A WINDOWS TASK FOR MOBILE USE: TIPS AND TRICKS …...… 7

APPENDIX I: EXAMPLE SCENARIO FILE …………………………………………..... 9 APPENDIX II: EXAMPLE SCENARIO FILE ……………………………………..…..... 11

Technische Ondersteuning Psychologie __________________________________________________________________________________ 2

• 1. INTRODUCTION

In 2016, the stimulus delivery and experiment control program Presentation introduced Presentation Mobile: the possibility to run experiments on Android and iOS devices. In addition to Presentation’s documentation on mobile experiments1,2, this document provides a quick introduction to Presentation Mobile for the novice user. After programming an experiment in Presentation for Windows, it can be run on the freely available Presentation Mobile app3. Please note that in order to design an experiment with touch responses, Windows 8 or higher is needed. Also make sure the latest version of Presentation is installed. There are a few important things to keep in mind while designing an experiment for mobile use. Firstly, the experiment settings and scenario file need to be adapted to touch screen properties. Intermediate versions of the experiment can be tested without a license, but once the experiment is finished it should be packaged for hosting on the NBS website, in order to remove the test mode watermark. The experiment needs to be downloaded in the mobile app and output can be transferred to a computer. Importantly, mobile experiments have restrictions as compared to Windows experiments. All these issues, as well as some tips and tricks to adapt a task for mobile use will be discussed below.

• 2. TOUCH

Experiment settings A task that requires a mouse device when run on a Windows device does not automatically work on a touch-based mobile device. To use touch in an experiment, the pointing device should be selected in the response settings of the experiment. This is done by selecting Settings tab > Response Panel > Pointing Device > Use Mouse. Also, at least one active button needs to be specified (most likely the Pointer Down button and optionally other Pointing Device buttons as well).

1 https://www.neurobs.com/menu_presentation/menu_features/mobile_docs 2 https://www.neurobs.com/pres_docs/html/03_presentation/15_creating_mobile_experiments.htm 3 Available for Android (https://play.google.com/store/apps/details?id=com.neurobs.presentation) and

iOS (available from the App Store)

Technische Ondersteuning Psychologie __________________________________________________________________________________ 3

In the properties of the pointing device (see Figure 1) ‘touch buttons’ can be specified, that can be used as active response buttons in your scenario. Each touch button (e.g. pointer down) has a from and a to specification, with the states ‘up’ (not touching the screen), ‘hover’ (almost touching the screen, used for a stylus), and ‘down’ (touching the screen). These from and to settings describe one transition from one state to another. Also note the two drop-down boxes, that can be used to select all states ‘higher than’ or ‘lower or equal to’ a certain state. As an example, to generate one button response when the screen is touched, ‘up’ and ‘hover’ can be ticked for from, and ‘down’ for to, meaning that one response is generated by moving from a state where the screen is not touched, to a state where the screen is touched. Using the drop-down boxes, the same result can be achieved by selecting ‘states higher than down’ for from and selecting ‘states lower or equal to down’ for to. Edges can be defined to restrict the area to which each touch button is responsive. It is recommended to specify this in ‘custom units’ rather than pixels, because of the variability in size of mobile devices. The settings of the touch buttons can be tested using the ‘test button’ function. A very useful feature is the mouse test mode, which enables you to test the touch functions on a non- touch screen using the computer mouse. This setting is not saved in the experiment file and needs to be ticked every time the experiment is opened.

Figure 1. Properties of the Pointing Device.

Technische Ondersteuning Psychologie __________________________________________________________________________________ 4

Scenario file Just like any other button presses, touch button presses result in response_data objects in the response_manager. For example, the amount of touch responses can be accessed by entering:

response_manager.total_response_count(1)

The touch_response conversion function can be used to obtain a touch_response data object (a child type of the response_data object maintained by the response_manager) and can be treated like other response_data objects.

touch_response touch = touch_response(response_manager.last_response_data() );

The touch_response object can for example be used to obtain the location of the touch down event by applying the functions x( ) and y( ).

term.print(touch.x()); term.print(touch.y());

Instead of obtaining touch_responses from the response_manager, the touch_device conversion function can be used to convert the mouse object associated with the response_manager to a touch_device (a child type of the mouse object).

touch_device tdevice = touch_device( response_manager.get_mouse(1) );

The touch_device provides access to touch_interaction objects, the basic units of a touch event, it represents a single complete swipe, from touch-down to touch-up, including the complete trajectory of the finger/stylus while it was down, with information on location and associated timestamps. Touch_interactions can be accessed by applying the get_interaction function to the touch_device. This function requires the index of the touch_interaction as an argument.

touch_interaction ti = tdevice.get_interaction(i);

At the lowest level, touch_interactions are build up from touch_samples, with their own time and x and y values. To access a touch_sample, the get_sample function can be applied to a touch interaction. This function also requires the touch_sample index as an argument.

touch_sample ts = ti.get_sample(i);

Also, the sample_count function can be used to obtain the number of touch_samples in an interaction.

term.print(ti.sample_count());

In the case of multitouch, multiple touch_interactions are active at the same time. Once an interaction has ended (i.e. the last sample has the ‘up’ state), the active property of the touch_interaction becomes false:

if (ti.active()) then term.print(“active”);

Technische Ondersteuning Psychologie __________________________________________________________________________________ 5

else term.print(“inactive”); end;

More detailed information on settings for touch screens can be found in the sections ‘Touch devices’4 and ‘Touch Buttons’5 in the Presentation Documentation. TEST MODE To test intermediate versions of an experiment in the mobile app, no license is needed. The experiment can be copied to the device by USB transfer and tested immediately. For experiments in test mode, only the experiment filename is shown in the Presentation Mobile app (with ‘test mode’ appended). A watermark with ‘test mode’ will appear throughout the

• experiment. Note that if you do happen to already have access to a license it is easier to host

intermediate versions using the license and then replace that version with a final version once

• ready. The license only cares about the number of active experiments. The following applies to

a situation where you do not have access to a license: Android The experiment and all other relevant files can be copied to an Android-device by USB transfer to the following folder: Device \ Internal storage \ Android \ data \ com.neurobs.presentation \ files \ experiments. When connecting the device to the computer with a USB-cable, it might be set to ‘USB for charging’ mode. Make sure the device is set to ‘USB transfer mode’ instead. Depending on the device and settings, this might require a specific driver, or changes in the USB settings in the developer mode on the Android device. iOS USB file sharing on an iOS device requires iTunes6. Once connected, the iOS device should appear in iTunes. Under ‘File Sharing’, the experiment and all other relevant files should be added to the Presentation app. Please note that the folder structure of the experiment copied to iOS should be exactly the same as in Windows. PACKAGING EXPERIMENTS To host an experiment on the NBS website in order to run it in non-test-mode on a mobile device, it needs to be packaged using the ‘create secure package’ option in the Presentation ‘experiment’menu. Before packaging, the following criteria need to be met:

4

https://www.neurobs.com/pres_docs/html/03_presentation/05_response_devices/02_axis_devices/04_t

• uch_devices/index.html

5

https://www.neurobs.com/pres_docs/html/03_presentation/05_response_devices/01_button_devices/0 5_touch_buttons.htm

6 See https://support.apple.com/en-us/HT201301 for an overview of file sharing in iTunes

Technische Ondersteuning Psychologie __________________________________________________________________________________ 6

• The experiment needs to have a name (Main tab)
• The experiment needs to have an author (Main tab)
• The experiment needs to have a version (Main tab)
• The logfile naming type must be "Automatic" (Settings tab > Logfiles)
• Output data file encryption must be used (Settings tab > Logfiles). An encryption key

should be provided using hexadecimal notation (i.e. using the numbers 0-9 and the letters A-F). The encryption key should have at least 4 bytes and all bytes after 56 are

• ignored. Each byte contains two characters (example: AAAA1111)
• Logfile metadata support must be selected (Settings tab > Logfiles)
• The display device adapter must be set to "Primary Display Driver", and the display

mode to "current setting" (Settings tab > Video)

• The Sound Device sound card must be set to "Primary Sound Driver" (Settings tab >

Audio) To package an experiment, click on Experiment > Create Secure Package. After confirming that the experiment will be hosted online, presentation provides a list of files that are used in the experiment. Please note that text input files are not listed automatically, and need to be added manually. Once all files are selected, an encryption key should be selected for encryption

• f all files used in the experiment. When selecting ‘use default key’, the same key as for output

file encryption is used. The licensing behaviour can be changed in this window as well. After selecting ‘next’, the package is saved under a name provided by the user. The .ppk file that is created can be uploaded to the NBS website for hosting. HOSTING EXPERIMENTS To host experiments on the NBS website, an account and a license are needed. A step-by-step instruction on how to do this is provided on the NBS website7 and a short summary is provided here. Go to www.neurobs.com > My Account > Licenses and log in. Click on ‘file manager’ under ‘Web Hosting – See the file manager for Hosted Experiment information’ and re-enter your user information. Upload your packaged experiment (.ppk extension) by selecting ‘choose file’ and ‘Upload’ at the bottom of the page. Then, assign the experiment to your license by clicking the drop-down menu next to ‘Click to refresh selections’ and select ‘Assign’. A few settings can be changed in in this menu, e.g. making the experiment searchable in the app, providing keywords, and setting a password that is required to download the experiment. DOWNLOADING EXPERIMENTS IN THE APP Once these settings are saved, the experiment can be downloaded in the app by clicking ‘Download’ in the top right. An experiment can be searched for by author, name, keyword or package ID. After the experiment appears, it can be downloaded (after entering a password, if required). A downloaded experiment will appear in the list of experiments and can be run by clicking on it.

7 https://www.neurobs.com/menu_licensing/menu_instructions/mobile_activate

Technische Ondersteuning Psychologie __________________________________________________________________________________ 7

OUTPUT All log files and output files can be exported in the app (see Figure 2). Output can also be deleted immediately. After choosing ‘Export all’ or ‘Export new’, a menu appears in which you can choose an app to export the data, for example via e-mail. Log files and output files will be zipped and attached. Both output files and log files can also be opened on your Windows computer via USB transfer. They appear in ‘data’ folder under Device \ Internal storage \ Android \ data \ com.neurobs.presentation \ files. RESTRICTIONS FOR CROSS-PLATFORM EXPERIMENTS Presentation provides a complete list of methods and types that are not available for iOS or Android.8 The most important functions that are not supported on non-windows devices are listed below.

• 3D graphics
• Video
• Audio recording
• Keyboard responses
• Eye tracking
• Kinect
• Input- and output ports
• Force feedback
• FTP features

8 https://www.neurobs.com/pres_docs/html/04_reference/03_pcl_reference/01_os_compatibility.htm

Figure 2. Output in the Presentation Mobile app.

Technische Ondersteuning Psychologie __________________________________________________________________________________ 8

Most notable are the lack of video and 3D support and the inability to access the smartphone or tablet virtual keyboard during the experiment (although the latter could be emulated with some effort). ADAPTING A WINDOWS TASK FOR MOBILE USE: TIPS AND TRICKS Generalizability of variables Make sure you define all variables as generic as possible (e.g. using display_device.width() instead of the number of pixels of your screen), so the experiment will still work properly on a different device. Case sensitivity Because iOS (and Android in some cases) is case sensitive, make sure you refer to refer to the right filenames in your scenario file (e.g. picture.jpg is not the same as Picture.jpg). A reference to an incorrect filename might only cause an error on a non-windows platform (possibly not on a windows computer). Editing the scenario on a mobile device The mobile app provides the ability to make changes to the scenario file on your mobile device. Make sure you also make these changes in the original scenario file on your Windows computer, because editing the script in the app only affects the copy on your mobile device. Target platforms Under Settings > General > Secure Package Target Platforms you can define on which platforms (i.e. Windows, Android, iOS) your experiment will be run. This allows you to test whether your experiment contains functions that are not supported in your target platform, without having to test this on the platform. Once packaged, the experiment can only be run on the selected platforms. Image formats It is possible that some images are not loaded on a non-Windows platform. In that case, the images need to be reformatted to a standard format (e.g. jpg or png). Example experiment A simple example experiment called ‘Elmo’ by author Elisabeth Nieuwburg with Experiment ID 33FDC21E-3F32-4B11-A93B-021E897D5643 can be searched for in the mobile app. Enter elmo1 as password to download and run the experiment. An adapted version of the scenario file can be found under Appendix I.

Figure 3. Target Platforms for the Secure Package

Technische Ondersteuning Psychologie __________________________________________________________________________________ 9

APPENDIX I: EXAMPLE SCENARIO FILE

## Requirements for this scenario: ## # Any picture with the file name "elmo.png" needs to be present # in the stimulus folder # The Pointing Device needs to be listed under Active Axis / Force Devices # The Pointer Down button needs to be listed under Active Buttons # # This example was created by Elisabeth Nieuwburg, TOP, UvA, 2017 ## Headers ## default_font_size = 30; default_font = "arial"; active_buttons = 1; default_background_color = 255,255,255; default_text_color = 0,0,0; ## SDL ## begin; picture { bitmap { filename = "elmo.png";} elmo_pic; x = 0; y = 0; } main_pic; picture { text { caption = "TEST EXPERIMENT PRESENTATION MOBILE\n\n\nTry to click on Elmo as fast as possible!\n\nPress to start.";} instruct_txt; x = 0; y = 0;}instruct_pic; trial { trial_type = first_response; trial_duration = forever; picture instruct_pic; }instruct_trial; ## PCL ## begin_pcl; #Output file

• utput_file out = new output_file;

string subject_name = logfile.subject(); if subject_name.count() > 0 then

• ut.open(subject_name + "_click_on_elmo.txt");

else

• ut.open("no_ID_click_on_elmo.txt");

end;

• ut.print("trial_no\telmo_x\telmo_y\taccuracy\tRT\n");

#Display width/height int max_x = display_device.width() / 2; int min_x = -max_x; int max_y = display_device.height() / 2; int min_y = -max_y; #Picture width/height int elmo_height = int(elmo_pic.height()); int elmo_width = int(elmo_pic.width()); #Instruction instruct_trial.present();

Technische Ondersteuning Psychologie __________________________________________________________________________________ 10

#Trial loop loop int t = 1 until t>5 begin; #Determine random location of elmo int elmo_x = random((min_x+(elmo_width/2)), (max_x-(elmo_width/2))); int elmo_y = random((min_y+(elmo_height/2)), (max_y-(elmo_height/2))); main_pic.set_part_x(1,elmo_x); main_pic.set_part_y(1,elmo_y); #Present picture until response int loop_in = clock.time(); loop int count = response_manager.total_response_count( 1 ) until response_manager.total_response_count( 1 ) > count begin main_pic.present(); end; int loop_out = clock.time(); touch_response touch = touch_response( response_manager.last_response_data() ); #Feedback string accuracy; text feedback_txt = new text(); feedback_txt.set_transparent_color(255,255,255); if (touch.x() >= elmo_x-(elmo_width/2)) && (touch.x() <= elmo_x + (elmo_width/2)) && (touch.y() >= elmo_y-(elmo_height/2)) && (touch.y() <= elmo_y + (elmo_height/2)) then feedback_txt.set_font_color(0,200,0); feedback_txt.set_caption("Woohoo!", true); main_pic.add_part(feedback_txt,0,0); main_pic.present(); wait_interval(500); accuracy = "correct"; else feedback_txt.set_font_color(0,0,200); feedback_txt.set_caption("Too bad!", true); main_pic.add_part(feedback_txt,0,0); main_pic.present(); wait_interval(500); accuracy = "incorrect"; end; main_pic.remove_part(2); #Calculation of RT int RT = loop_out-loop_in; #Printing to outputfile

• ut.print(t); out.print("\t");
• ut.print(elmo_x); out.print("\t");
• ut.print(elmo_y); out.print("\t");
• ut.print(accuracy); out.print("\t");
• ut.print(RT); out.print("\n");

t = t + 1; end;

Technische Ondersteuning Psychologie __________________________________________________________________________________ 11 APPENDIX II: EXAMPLE SCENARIO FILE

active_buttons=1; begin; annulus_graphic{inner_width=0;inner_height=0;outer_width=12;outer_height=12 ;color=0,0,255;}pointer; annulus_graphic{inner_width=0;inner_height=0;outer_width=12;outer_height=12 ;color=255,255,0;}click1; annulus_graphic{inner_width=0;inner_height=0;outer_width=12;outer_height=12 ;color=255,0,0;}click2; begin_pcl; #basic module for a random dot task #created by J.G.Wijnen, UvA, 2015 #adapted for touchscreen May 2017 #task specs that may vary between blocks of trials can best be put here as globals and adjusted as needed sub present_RD_trial (int maxdur, int dominant_dot_direction, double perc_dominant) begin #task specs that vary between individual trials can best be included as input arguments #task specs that stay the same throughout the experiment can be put in the sub body rgb_color dotcolor = rgb_color (255,255,255,255); rgb_color domdotcolor = rgb_color (100,100,255,255); double dotsize = 2.0; int nr_of_dots = 800; #lower this value if exp seems sluggish int windowX = 600; int windowY = 600; double movspeed = 4.0; #derived specs int leftEdgeX = -windowX/2; int rightEdgeX = windowX/2; int topEdgeY = windowY/2; int bottomEdgeY = -windowY/2; int nr_of_dominant_dots = int(double(nr_of_dots) * perc_dominant); #determine direction for each dot array <int> movdir_deg [nr_of_dots]; array <double> movdirX [nr_of_dots]; array <double> movdirY [nr_of_dots]; loop int x=1 until x>nr_of_dots begin if x<=nr_of_dominant_dots then movdir_deg[x] = dominant_dot_direction; else

Technische Ondersteuning Psychologie __________________________________________________________________________________ 12

movdir_deg[x] = random(0,360); end; movdirX[x]=sin(double(movdir_deg[x])/180.0 * pi_value)*movspeed; movdirY[x]=cos(double(movdir_deg[x])/180.0 * pi_value)*movspeed; x=x+1; end; #determine dot start position array <double> dotX [nr_of_dots]; array <double> dotY [nr_of_dots]; loop int x=1 until x>nr_of_dots begin dotX[x]=double(random(leftEdgeX,rightEdgeX)); dotY[x]=double(random(topEdgeY,bottomEdgeY)); x=x+1; end; #create picture and dots box domdot = new box(dotsize, dotsize, dotcolor); box dot = new box(dotsize, dotsize, dotcolor); picture RD_pic = new picture(); loop int x=1 until x>nr_of_dots begin if x<=nr_of_dominant_dots then RD_pic.add_part(domdot,dotX[x],dotY[x]); else RD_pic.add_part(dot,dotX[x],dotY[x]); end; x=x+1; end; text infotxt1 = new text(); infotxt1.set_font_size(20); infotxt1.set_font("calibri"); infotxt1.set_caption("x",true); RD_pic.add_part(infotxt1,0,-display_device.height()/2 + 100); int resp1 = response_manager.total_response_count(1); double x1;double x2; double y1;double y2; double angle; int starttime = clock.time(); bool touched = false; int answertime; bool answered = false; loop until false begin if !answered then if resp1 != response_manager.total_response_count(1) then resp1 = response_manager.total_response_count(1); response_data respdat = response_manager.last_response_data(); touch_response touchresp = touch_response(respdat); if !is_null(touchresp) then x1=touchresp.x();

Technische Ondersteuning Psychologie __________________________________________________________________________________ 13

y1=touchresp.y(); RD_pic.add_part(click1,x1,y1); touched = true; end; end; if touched then response_data respdat = response_manager.last_response_data(); touch_response touchresp = touch_response(respdat); touch_interaction t_interact = touchresp.interaction(); if !is_null(t_interact) then if !t_interact.active() then x2 = t_interact.x(); y2 = t_interact.y(); angle = arctan2d( x2 - x1,y2 - y1 )/pi_value * 180.0; infotxt1.set_caption("your answer: " + string(round(angle,1)) + " degrees\ncorrect answer: " + string(dominant_dot_direction) + " degrees",true); RD_pic.add_part(click2,x2,y2); domdot.set_color(domdotcolor); answered = true; answertime = clock.time(); end; end; end; end; loop int x=1 until x>nr_of_dots begin dotX[x]=dotX[x]+movdirX[x]; dotY[x]=dotY[x]+movdirY[x]; if dotX[x]<leftEdgeX then dotX[x] = rightEdgeX;end; if dotX[x]>rightEdgeX then dotX[x] = leftEdgeX;end; if dotY[x]<bottomEdgeY then dotY[x] = topEdgeY;end; if dotY[x]>topEdgeY then dotY[x] = bottomEdgeY;end; RD_pic.set_part_x(x,dotX[x]); RD_pic.set_part_y(x,dotY[x]); x=x+1; end; ######################################### RD_pic.present(); ######################################### if clock.time() - starttime > maxdur then break;end; if answered && clock.time() - answertime > 4000 then break; end; end;

Technische Ondersteuning Psychologie __________________________________________________________________________________ 14

end; #end of sub int nr_of_trials=20; loop int t=1 until t>nr_of_trials begin present_RD_trial(30000,random(0,360),0.5); picture empty_pic = new picture(); empty_pic.present();wait_interval(500); t=t+1; end;