22 22nd Annual SIO SIOPSA Conference Leadin ing through in - - PowerPoint PPT Presentation

22 22nd Annual SIO SIOPSA Conference –

Leadin ing through in innovatio ion: Navig igatin ing th the new world rld of f work rk

4 – 6 August 2020 The Canvas, Riversands, Fourways

This years themes

• Teams and culture
• Neuroscience of innovation
• Medico-legal innovation
• Wellness and health
• Employee experience and engagement
• Talent attraction, development and mobility
• Innovative leadership

Keynote speakers

People Assessments in Industry (PAI)

Technological Advances in Testing & Associated Practitioner Implications 2020 ACSG Conference

Discussion Points

• About People Assessments in Industry (PAI)
• Current SA legislation and its implications
• ASSA
• Evolution of the World of Work
• Implications on Assessments
• Big Data – LinkedIn and Facebook
• Virtual Reality and Assessments
• The use of Artificial Intelligence in Assessments
• Gamification
• Robotics
• Implication for Assessment Practitioners
• Questions

About People Assessments in Industry (PAI)

PAI background

• Established in 1998 to proactively engage various stakeholders interested

in psychological assessment in South African industry.

• Functions as an interest group of the Society for Industrial and

Organisational Psychology in South Africa (SIOPSA) since 2005.

• Purpose of PAI is to ensure testing exists in the South African context as a

value adding, ethical and fair practice.

PAI acts as an independent body but seeks to build relationships and to collaborate with such bodies as the:

• Health Professions Council of South Africa (HPCSA),
• Association of Test Publishers (ATP),
• Psychological Society of South Africa (PsySSA)
• Assessment Centre Study Group of South Africa (ACSG),
• The International Test Commission (ITC),
• Business SA, test users and test takers who have similar interests with

regards to assessments in the workplace.

PAI Background

• David Bischof – (Organisational Management Technology - Evalex)
• Jaco de Jager (Top Talent Solutions)
• Nadene Venter (Independent Consultant)
• Dr Paul Vorster (Researcher)
• Kevin Distiller (Organisational Management Technology - Odyssey)
• Ntswaki Raboroko (Head Talent Acquisition - Nedbank)

PAI Committee

Current SA Legislation and its implications

According to the Health Professions Act (no. 56 of 1974), instruments that measure psychological constructs must be used, interpreted and controlled by psychologists. Furthermore, only individuals registered with the Professional Board of Psychology (HPCSA) may use psychological tests. Psychological Tests - Chapter 5, point 55 in particular states that - A psychologist shall ensure that tests used have been classified by the board and that the provisions of any applicable legislation, such as the Employment Equity Act, 1998 (Act No. 55 of 1998), have been complied with.

What does the Legislation say

Employers and/or psychologists and psychometrists need to comply with the provisions of the EEA act section 8 (a), (b) and (c) which state that the use of Psychological testing and other similar assessments of an employee are prohibited unless the test or assessment being used-  Has been scientifically shown to be valid and reliable;  Can be applied fairly to all employees  Is not biased against any employee or group; The above prohibition holds true for all testing of employees irrespective if it is a Psychological test, or not, being used. All tests measuring a Psychological construct must be registered with the HPCSA and can only be used by a person registered with the HPCSA.

What does the Legislation say

• To classify any device, instrument, questionnaire, apparatus, method, technique or test aimed at the

evaluation of emotional, behavioural and cognitive processes or adjustment of personality of individuals or groups of persons, or for the determination of intellectual abilities, psychopathology, personality make-up, personality functioning, aptitude or interests by the usage and interpretation of questionnaires, tests projections or other techniques or any apparatus, whether of SA origin or imported, and to report thereon to the Professional Board.

• The Annual publication of a list of psychological tests/psychometric instruments classified by the

Professional Board

• Develop training guidelines/standards related to psychometrics and psychological assessment that can

inform and be used in the accreditation of qualifications, universities and internship programmes, when setting the national Board examinations, and for continuing professional development purposes

• Develop guidelines for ethical practice related to test use and psychological assessment and how to assess

whether a psychological test meets the required standards

• Develop minimum requirements/standards for psychological tests
• Classification will entail verifying whether a test was psychological or not. To this end, practitioners and

publishers should submit the full test manual that states the construct(s) tapped by the test, evidence of psychometric properties, an indication as to whether the item content was culturally appropriate. No costs would be attached to test classification by the Professional Board.

HPCSA – Revised Mandate of the Psychometrics Committee

• f the Professional Board for Psychology and its Implications

• Message was that the Psychometrics committee mandate was to classify tests that then go
• nto list;
• There is no legislative framework for HPCSA to issue certification; no mandate to certify
• The reason for training guidelines is to quality assure so that practitioners are trained well

enough to evaluate if can or cannot use a test

• The registration of a Psychological test with the HPCSA as published and Gazetted by the

HPCSA is not a certification of the compliance of the EEA or any other evaluative “certification”. The HPCSA is indicating that the test measures a psychological construct and as such is classified as a Psychological test that must be controlled and used by a HPCSA registered person

• Online and Computer - Based testing - SA guidelines on computerised testing would no longer

be reviewed; HPCSA to develop a position statement instead of guidelines; position statement provide practitioners with board stance concerning

• nline

assessments; collaboration with all stakeholders on this position statement; position statement to guide profession; maybe adopt international guidelines

HPCSA presentation by Dr Thandeka Moloi and Justin August at PsySSA 2019 conference

HPCSA presentation by Dr Thandeka Moloi and Justin August at PsySSA 2019 conference

Test classification process entails:  test developer submits test to the board  You have to be registered Psychologist/Psychometrist to submit test  Test manual examined to determine psych constructs  classified as psychological or not psychological  test publishers no longer obliged to inform board of tests under development or being adapted  Once test classified, test material shipped back to publishers due to no space available to store tests  no cost for classification  Classification indicates test is limited to use by psychologists - guideline for the public and practitioners  Test classification list to be published twice year on HPCSA website

• An independent non-regulatory external assessments evaluation ‘body’

working collaboratively with the support of the Department of Labour and the HPCSA to assist in implementing a robust, best practice and technology enabled process that could be used to review people assessment instruments and tests.

• It is envisaged that this body will look at the broad spectrum of instruments

that are used in South Africa and will not be limited to psychological tests only. In cases where an instrument meets the criteria for test classification as a psychological test, the applicants will be informed accordingly about submitting the test for classification to the HPCSA as per the statutory requirements.

ASSA Formation

The core purpose of Assessment Standards South Africa (ASSA) would be to review the quality of tests that are available for use in South Africa. Based on the experience in other countries the voluntary submission of assessment instruments for objective evaluation and reviews will raise the general standard and awareness of using quality tests.

Vision and Mission of ASSA

• The ASSA would provide a minimum standards focus in South Africa for all activities in

relation to tests and testing.

• Maintain a website and online platform to provide information for best practice, results of

test reviews and access to information about tests and testing

• Management and dissemination of information on standards relating to tests and testing.

This includes information on qualifications in test use, tests reviews, test registration and the publishing and dissemination of advisory statements, in addition to guidelines and

• ther literature on standards for the construction, use and availability of tests.
• Quick evolution of testing – (AI, Gamification, Big Data) – field is evolving faster than

legislation can keep pace with; ASSA can assist in staying on top of this and collaborating locally and internationally to look at ways to maintain quality and adherence to EEA.

ASSA Activities

Evolution of the World of Work

1970s 1980s 2000s 1990s Today 2010s

Technology Individuals Businesses Public policy

Rate of change

The gap between Technology and Public Policy - 2017 Deloitte Global Human Capital Trends

The top 10 skills that will be in demand by all employers by 2020 - World Economic Forum - 2016 report

Position Competence Questionnaires Ability / Cognitive Simulations 10 Cognitive flexibility No Yes Somewhat 9 Negotiation skills Propensity to No Yes 8 Service orientation Yes No Yes 7 Judgment No Yes Yes 6 Emotional intelligence Yes No Somewhat 5 Coordinating Propensity to No Yes 4 People management Propensity to No Yes 3 Creativity Propensity to Somewhat Yes 2 Critical thinking No Yes Yes 1 Complex problem solving No Somewhat Yes

World Economic Forum – Future of Jobs Report 2018

World Economic Forum – Jobs of Tomorrow Mapping Opportunity in the New Economy – January 2020

• Successful professions in future will increasingly require both “digital” and “human” competence and emerging

professions reflect the continuing importance of human interaction in the new economy.

• There are seven emerging professional clusters and 96 jobs of tomorrow within them that vary in their

individual rate of growth and in the scale of job opportunities they offer in the aggregate. These clusters are:  Data and AI  Engineering and Cloud Computing  People and Culture  Product Development  Sales, Marketing and Content  Care Economy  Green Economy

• The highest-demand skills required in these emerging professional clusters span both technical and cross-

functional skills. These in-demand skills can be divided into five distinct skills clusters:  Business Skills  Specialized Industry Skills  General and Soft Skills  Tech Baseline Skills  Tech Disruptive Skills

Implications on Assessments

2017 - Robert McHenry - Psychometric Testing: Critical Perspectives – ‘The Future of Psychometric Testing’

• Prediction #1: Smartphones will replace computers for employee assessment
• Prediction #2: High-quality psychometric testing services will be sold direct to

consumers

• Prediction #3: Advances in neuroscience of personality will reveal which are the

most valid individual differences to measure and how best to measure them

• Prediction #4: The digital movement, coupled with the use of big data and new

forms of digital CV will render many of the current applications for high-stakes psychometric testing redundant

• Prediction #5: The basis for employee development will be derived from the

data yielded by wearable devices and not from psychometric tests

• The testing industry - has been in ideational stagnation, change is slow and with minor

cosmetic changes, only the numbers in the names of tests substantially change

• Next

Generation Assessments are different in design, look, feel, scoring and administration

• Innovation in technology allow for delivery online through mobile phone and tablets
• Completely autonomous scoring and reporting generation is the new kid on the block
• Next generation tests – use of artificial intelligence, machine learning, behavioural and

predictive modelling (Computational Modelling and Psychology) gamification, bio screening, virtual reality, robotics, auto transcribing, language modelling and natural language processing, personality assessment via behaviour-linguistic analysis and social network data (Facebook; LinkedIn and Twitter).

• Video interviews where AI software is used to analyse the candidate’s facial expression,

tone and language to ascertain job fit

• Resume filters scan for keywords and phrases on resumes
• Chatbots schedule interviews and algorithms help predict a job candidate’s future success

The next generation of Assessments

The rise of Digital Psychometrics

Big Data - Facebook

Big Data - Facebook

• LinkedIn used as ‘screening tool’ by hiring managers
• Provides for Skills Assessments: short, multiple-choice tests that users can take to verify

their knowledge in areas like computer languages, software packages and other work- related skills.

• Research conducted by Julia Levashina, Ph.D., associate professor of management at Kent

State University, along with her colleague Nicolas Roulin, associate professor

• f

industrial/organizational psychology at Saint Mary's University in Canada

• LinkedIn profiles could be used to make inferences about job applicants’ visible skills (such

as leadership, communication and planning), personality traits (such as extraversion) and cognitive ability. However, hiring managers are not recommended to use LinkedIn profiles to assess less visible skills (such as conflict management or adaptability) and personality traits (such as conscientiousness and emotional stability).

• The researchers also demonstrate that hiring recommendations based on LinkedIn profiles

are influenced by the job applicant’s LinkedIn profile length, photo and number of

• connections. It was found that other features of LinkedIn profiles (such as listing more skills,

collecting endorsements, joining groups or describing volunteering involvements) had a negligible impact on hiring recommendations.

Big Data - LinkedIn

• Virtual and augmented reality (VR and AR) use biometric tracking data - micro-movements of

head, torso, hands, and eyes - on the job training

• Can diagnose or predict anxiety, depression, schizophrenia, addiction, ADHD, autism spectrum

disorder and more about a person’s cognitive and physical function.

• What if this medically-relevant data is fed into users’ psychometric profiles?
• Profiles may start out as relatively harmless, merely predicting when someone might be getting

ready to buy a new car.

• However, sprawling psychographic profiles with medical inputs could leave people vulnerable
• Anonymizing VR and AR tracking data is nearly impossible because individuals have unique

patterns of movement.

• Using gaze, head direction, hand position, height, and other behavioural and biological

characteristics collected in VR headsets, researchers have personally identified users with 8 to 12 times better accuracy than chance.

• Dirk Geldenhuys – Use VR to develop resilience in the VUCA world?
• Need for stronger regulation

Virtual Reality and Assessments

• Intelligent software has predominantly been used to automate screening
• Chatbots used as a part of the screening process
• Robotics - Call center in the UK needed to hire representatives to respond to

customers who speak seven different languages; AI-enabled test where candidates were “interviewed” by a robot. The software evaluated their speech for syntax and grammar, and the company made offers based on the results

• Vera - a lifelike Russian robot with impressive conversational skills used to

phone and video interview candidates for positions

• Socially capable robots can provide many advantages to diagnostic practice:

engage people, guarantee standardized administration and assessor neutrality, perform automatic recording of subject behaviours for further analysis by practitioners.

Robotics and Assessments

Robotics and Assessments

Artificial intelligence is about mimicking the human cognitive and behavioural processes to solve more complex problems faster Artificial Intelligence

• Robotic process automation: This is achieved by gathering and transferring expert knowledge and

then programming the system with an 'if/then' rule-based approach. Chatbots are a great example. However, this rule-based system is not capable of learning and improving without being given explicit instructions. In talent assessment, computer-generated interpretative reports make use of this technology.

• Machine learning: Even though a computer cannot think for itself (not yet anyway), statistical tools

can enable a system to model predictions from any given data – and to add to the model to improve its predictions over time. This is used in data analysis to create predictive people analytics as a means to help employers make better talent decisions.

• Pattern matching: This AI technique uses a computer to check the sequence of responses to

determine if there is a pattern. It can be used to carry out some 'human' tasks, such as recognizing faces or identifying emotions.

• Natural language processing: Makes use of text and speech analytics to extract the underlying
• meaning. This can be applied to analyse speech in interview question responses

Artificial Intelligence and Assessments

Artificial Intelligence and Assessments

• Filip Lievens, (Professor from the Department of Personnel Management, Work and Organizational

Psychology at Ghent University) - EASE tool (Electronic Assessment Simulation Exercise) -all participants start with one e-mail. They can then obtain further information by organizing meetings, requesting archive documents or sending e-mails to colleagues. The participants need to ask the right people the right questions in the right way. The computer then generates the requested information, and it also functions as an assessor that evaluates and records all actions objectively.

• http://www.fmaconferences.org/Vietnam/Papers/Hrazdil%20et%20al.%20FMA%202019.pdf (2018)

study on utilising IBM Watson Personality Insights service to measure executive personalities (big 5) based on CEO’s responses to questions raised by analysts during conference calls.

• Video interviewing – Audio transcribed and analysed for clarity of speech and language proficiency.

AI used to analyse the visual elements through emotion tracking software and facial recognition

• Assessment systems designed to integrate constructs from across psychometric instruments to

assist with the development of psychometric competency frameworks, the identification of inhibitors and job position fit

Benefits:

• Improvement of assessment process flows
• Capturing of real-time data as it happens
• Equality - a system cannot identify race, gender or age.
• Engagement: the candidate finds these types of assessments modern, engaging and novel.
• Depth of Information: a tapestry of information collection; not possible by even an

experienced assessor.

• If systems are stable then large volumes can be handled
• Software algorithms produce accurate and error-free calculations
• An AI neurological network can produce a result without any human intervention
• AI can analyse vast amounts of data accurately and error-free. This assists with faster decision

making

• Elimination of human conscious and unconscious bias BUT an algorithm is only as good as the

data that’s fed into it.

Artificial Intelligence and Assessments

Concerns:

• Companies exploit public confusion slap the “AI” label on whatever they are selling ‘ (How to

recognise AI snake oil - Arvind Narayanan https://www.cs.princeton.edu/~arvindn/talks/MIT- STS-AI-snakeoil.pdf)

• Ethical concerns regarding accuracy of areas such as content identification, facial recognition,

medical diagnosis from scans, speech to text and ‘deepfakes’.

• Defensibility. How do you defend the outcomes of standardised 'plug-and-play' AI systems which

are not aligned with specific job requirements and lead to custom decisions?

• Ethics. There is an ethical question around how much support to take from an AI system for

decision making. i.e. allow AI to reject outright or flag to check?

• Do and can organisations provide proof that AI predicts performance?
• Data handling. AI excels at analysing massive amounts of data, but the results can be

misinterpreted or even deliberately abused.

• Video interviewing – applicability – can you really identify emotions from facial recognition?

Generalisability of facial expressions across racial and cultural groups; If you look anxious are you really anxious?

Artificial Intelligence and Assessments

Gamification

• Instead of developing a full game, gaming elements are layered over an existing program
• r context in order to amplify users’ motivation to engage within that context (Kapp,

2014)

• Designed to motivate users by providing feedback, recognition, status and the potential

for competition among users (Muntean, 2011).

Gamification

Advantages of Gamification

Big Data Rich data analytics, in a concise meaningful manner relevant to business needs and digital transformation. Psychological Constructs Founded on strong, well established tried and tested psychometric principals of existing assessments. Generation Z Strongly appeals to the new generation of job seekers, millennials / generation z. Higher Engagement Provides a unique and positive recruitment or development experience for potential candidates. Mobile Apps Immersive, rich and fun interactivity for the candidate, yet still providing reliable and usable data

Challenges of Gamification

• The time involved in truly customising and validating a game,
• Ongoing investment as the life cycle of the games may be short,
• Some people just don’t enjoy games,
• Games are only fun in low-stake environments,
• Candidates may feel belittled / not taken seriously,
• If poorly designed and irrelevant to the construct being assessed, mechanics introduce error and

noise into assessments,

• The game might lack face validity and proving psychometric properties difficult.
• If a game is developed in a Eurocentric environment, gaming mechanics, skin, storyline and

challenges may not be culturally relevant to SA candidates.

• Beware high graphics and data requirements in developing environments.

Implications for Practitioners

Implications for Practitioners

• In South Africa psychologists and psychometrists still need to comply with the provisions of the

EEA act section 8 (a), (b) and (c)

• The above prohibition holds true for all testing of employees irrespective if it is a Psychological

test or not being used.

• All tests measuring a Psychological construct must be ‘classified’ with the HPCSA and can only be

used by a registered person with the HPCSA

• ASSA are looking at a Test Criteria Evaluation process and partnership with the HPCSA in this

regard

• Local (and global) Legislation and regulations, research and classification/certification is lagging

in terms of the next generation of assessments which is already here

• Innovations in technology are allowing for easier delivery of assessments as well autonomous

scoring and reporting to improve assessment efficiency and accuracy

• The use of Big Data, Machine Learning, Artificial Intelligence, VR, Robotics and Gamification and

are leading the way in the assessment and talent space

• Although proving to be more effective, efficient and accurate in nature, we need to be careful of how

these assessments are constructed, where they can be effectively used and what the implications are for the South African environment.

• Research into validity and reliability on digital assessments should be a combination of theory- and

data-driven models and more research is required.

• AI’s role should be restricted to providing additional information and enhancing efficiency. AI can

deliver useful information, at various stages of the selection process, but should be used to support, and not make the final decision.

• We need to be careful as Algorithms don’t have an ethical compass. They may be based on undesired

societal biases. Rubbish in, rubbish out!

• Data-driven models and final decision-making based on assessment results should remain under

control of humans - leverage technology and assessments learnings to enable better decisions

• Role of practitioners will probably change – more about job analysis and algorithm design and

ensuring ethical and scientific use of technological enabled assessments.

Implications for Practitioners

https://www.psychologytoday.com/za/blog/the-situation-lab/201910/beware-these- marketing-trends-in-psychological-assessment

Ryne A. Sherman, Ph.D., - Chief Science Officer at Hogan Assessment Systems –

‘Many of the new assessment firms use flashy technology and claim new insights into workplace performance. Hiring managers and HR professionals need to be wary of companies using these common marketing trends. Only two things matter in psychological assessment: fairness and predicting

• performance. Companies that emphasize neuroscience, big data, and

gamification may be trying to distract you from the fact that their assessments don’t predict workplace performance’. Understand what is inside the black box!

Implications for Practitioners

David Bischof Chairperson: People Assessments in Industry (PAI) Director – Research Psychologist & Psychometrist OMT | Evalex Email: David@evalex.com Office: 073 254 5119

Thank You!