Autonomous Robots: Towards Founded Assessment of Robustness Gerald - - PowerPoint PPT Presentation

▶
autonomous robots towards founded assessment of robustness
SMART_READER_LITE
LIVE PREVIEW

Autonomous Robots: Towards Founded Assessment of Robustness Gerald - - PowerPoint PPT Presentation

Jan 21, 2024 507 likes •724 views

Autonomous Robots: Towards Founded Assessment of Robustness Gerald Steinbauer (1), Lisa-Christina Winter (2), Alexander Nussbaumer (2), and Dietrich Albert (2,3) (1) Institute for Software Technology, Graz University of Technology, Austria (2)

slide-1
SLIDE 1

Autonomous Robots: Towards Founded Assessment of Robustness

Gerald Steinbauer (1), Lisa-Christina Winter (2), Alexander Nussbaumer (2), and Dietrich Albert (2,3)

(1) Institute for Software Technology, Graz University of Technology, Austria (2) Knowledge Technologies Institute, Graz University of Technology, Austria (3) Department of Psychology, University of Graz, Austria

Presented by Alexander Nussbaumer

SDIR Workshop @ ICRA Conference Karlsruhe, Germany 6 Mai 2013

slide-2
SLIDE 2

Problem Statement

  • How can we guarantee that

– a robot is dependable in the sense that it is able to successfully perform a task – in a non-deterministic dynamic environment – and facing internal and external difficulties

2013-05-06 SDIR Workship @ ICRA Conference 2

Environment Decision Making Acting Perception

Execution Failures Sensing Failures Exogenous Event Incomplete or Wrong Knowledge Hardware Failures Software Failures

slide-3
SLIDE 3

Example: task to bring the letter to room B and the folder to C

2013-05-06 SDIR Workship @ ICRA Conference

Problem Statement

3

Execution Fault Sensing Fault Exogenous Fault

slide-4
SLIDE 4

Challenges

  • develop algorithms that make a robot dependable when

facing complex tasks, nondeterministic environments, and failures?

– is done in the robotics group

  • evaluate, quantify and compare the dependability of such

systems in the context of a very high number of varying environment instances, tasks, difficulties and faults?

– research task described in this paper

2013-05-06 SDIR Workship @ ICRA Conference

Problem Statement

4

slide-5
SLIDE 5

These days …

  • only a few formal approaches for evaluating performance

and correctness for tasks available

– NIST standard test methods for USAR robots (hardware centred) – workshop on conditions for replicable experiments and performance comparison in robotics research

  • no systematic, reliable, and valid assessment methods

available for robots (?)

  • psychological sciences have well elaborated models and

procedures to design and interpret capability tests for humans.

– Item Response Theory (IRT) – Knowledge Space Theory (KST)

2013-05-06 SDIR Workship @ ICRA Conference

Problem Statement

5

slide-6
SLIDE 6

2013-05-06 SDIR Workship @ ICRA Conference

Objective and Idea

6

techniques, algorithms and systems from the AI and robotics communities long tradition of psychological sciences in assessing capabilities and performances of individuals

test framework for robots based on psychological methods for testing abilities and robustness of robots combine build build

slide-7
SLIDE 7
  • most prominent example: PISA studies
  • probabilistic test theory for assessing abilities of people
  • test consists of items

– question or problem to be answered; questionnaire

  • assessing latent abilities, e.g. certain mathematical skills

– unidimensionality: only one ability is tested at the same time – e.g. in math word problems, only mathematical ability is tested (not reading ability)

  • assumption: stochastic relationship between the latent ability

and the responses to items

  • different IRT models

– e.g. Rasch Model: most common and empirically best evidenced

  • IRT has specific advantages

– unidimensionality, specific objectivity, adaptive testing

2013-05-06 SDIR Workship @ ICRA Conference

Item Response Theory

7

slide-8
SLIDE 8
  • constructing a test

– definition of abilities to be tested – definition of items for the test – two types of parameter

  • person parameter ("ability")
  • item parameter ("difficulty")

– method for parameter estimation

  • large sample of individuals required

– checking model fit

  • conducting a test

– IRT test is easy to use and analyse – can be used in different contexts (schools, countries, etc.) – can be used adaptively

2013-05-06 SDIR Workship @ ICRA Conference

Item Response Theory

8

logistic function for one item

slide-9
SLIDE 9
  • problem := task + environment + failure

– problems are used as IRT items – difficulty is defined by environmental setting and failures – tests needed if problems are solved or not

  • examples of problems

– navigation to a target indoor on smooth surface – navigation to a target outdoor – navigation to a target outdoor with perception failure – navigation to a target in disaster area – manipulation of rigid object – manipulation of glass with liquid

  • ability

– abstract concept related to problems

  • robustness

– defined on abilities, difficulties, and mastered failures

2013-05-06 SDIR Workship @ ICRA Conference

IRT for Robotics

9

Task

Failures

Environment

slide-10
SLIDE 10
  • framework for testing robots on different levels

2013-05-06 SDIR Workship @ ICRA Conference

IRT for Robotics

10

Performance Abilities Robustness Manipulation Object Recognition Navigation Problems

slide-11
SLIDE 11
  • robot: PR2

– standard robot with clear set of abilities – common software architecture ROS – good simulation possibilities – different control architectures for performing tasks – variants within a control architecture (e.g. modification in perception) – other robots in future

  • environment: simulation

– variable environments can easily be created in simulation – environment is controllable (difficulties, failures, …)

2013-05-06 SDIR Workship @ ICRA Conference

Robots and Environments

11

slide-12
SLIDE 12
  • definition abilities and problems

– based on functions of robots – analysis of tasks based on experiences and literature review – different environments where tasks are performed

  • creation of individuals

– control architectures and different variants

  • designing the empirical study

– including failures

  • preparation of technical environment

– simulation, tasks, individual robots, …

  • conducting empirical test
  • analysis

– analysis of model fit – creation of assessment framework

2013-05-06 SDIR Workship @ ICRA Conference

Procedure for Test Construction

12

slide-13
SLIDE 13
  • behavioristic mathematical-psychological theory

– for representing a knowledge domain and the knowledge of a learner – problems are structured in prerequisite relations – directed acyclic graph – validation of structure in large samples – knowledge state: problems that can be soloved

  • allows for efficient assessment

– adaptive assessment – not all problems have to be posed

  • KST in robotics

– problems as defined before – e.g. if a robot can solve problem A that the robot can also solve problem B

2013-05-06 SDIR Workship @ ICRA Conference

Knowledge Space Theory

13

slide-14
SLIDE 14
  • competence-based extension to KST

– focus on underlying cognitive constructs in order to explain problem solving – assignment of competences to problems – structuring competences through prerequisite relations – validation of structure in large samples – competence state: competences that a person has

  • CbKST in robotics

– abilities can be treated as competences – e.g. object recognition is a prerequisite for navigation

2013-05-06 SDIR Workship @ ICRA Conference

Competence-based Knowledge Space T.

14

slide-15
SLIDE 15
  • approach for creation of an assessment framework that

allows fro testing abilities an robustness of robots

– by combining methods from robotics and founded test theory from psychology

  • answering questions regarding

– abilities of a specific robot – robustness of a specific robot – comparing robots

  • long-term vision is to create certification framework for

robots

2013-05-06 SDIR Workship @ ICRA Conference

Conclusion

15

slide-16
SLIDE 16
  • application for an Austrian national research project (FWF)
  • detailed elaboration of

– conceptual and technical approach – ontology of problems, abilities, tasks, …

  • 3 phases: IRT - KST - CbKST

– investigation how well they are applicable

2013-05-06 SDIR Workship @ ICRA Conference

Next Steps

16

slide-17
SLIDE 17

2013-05-06 SDIR Workship @ ICRA Conference

Further Information:

  • Institute for Software Technology

http://www.ist.tugraz.at/

  • Cognitive Science Section @ KTI

http://kti.tugraz.at/css/

Any Questions?

17

slide-18
SLIDE 18
  • Q1 : what are the problems of software development in robotics that can

be considered "solved"? – none, in our opinion

  • Q2 : what are the solutions to these problems that can be considered

best practice and why? – Robot systems are heavily related to integration of various hardware and software sub-systems. Modern frameworks and proved approaches ease the problem a little, for example the ROS hype. It eases the integration but has still a number of drawbacks

  • Q3 : to what extent the solutions to these problems are robotic-specific?

– To a large extent integration is robotics-centered

2013-05-06 SDIR Workship @ ICRA Conference

Open Discussion

18

slide-19
SLIDE 19
  • Q4 : What problems of software development in robotics remain to be

solved? – Currently, there is no solution for a sound assessment strategy to test robots according their robustness in changing environment and facing internal and external difficulties

  • Q5 : Why state-of-the art software technology is not adequate to solve

these problems? – Testing of robots is partially done in the context of the performance if individual capabilities like navigation or mapping. However, it is mainly not focused on complex integrated systems acting in a real world.

  • Q6 : What are the promising research directions to solve the open

problems? – Developing a new and sound framework for assessing robustness and abilities of robots based on validated methods from psychology which have a long tradition in assessing capabilites of individuals.

2013-05-06 SDIR Workship @ ICRA Conference

Open Discussion

19

slide-20
SLIDE 20

2013-05-06 SDIR Workship @ ICRA Conference 20

ability probability of success

0.0 1.0