Hayato Kobayashi , Tsugutoyo Osaki, Tetsuro Okuyama, Akira Ishino, - - PowerPoint PPT Presentation

▶

Oct 25, 2023 231 likes •514 views

Hayato Kobayashi , Tsugutoyo Osaki, Tetsuro Okuyama, Akira Ishino, and Ayumi Shinohara Tohoku University, Japan (Team Jolly Pochie) 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 1 Robot position Ball position We need to check the screen

SLIDE 1

Hayato Kobayashi, Tsugutoyo Osaki, Tetsuro Okuyama, Akira Ishino, and Ayumi Shinohara Tohoku University, Japan (Team Jolly Pochie)

1 2008/7/17 RoboCup Symposium 2008 in Suzhou, China

SLIDE 2

/ 25 2 2008/7/17 RoboCup Symposium 2008 in Suzhou, China

Ball position Robot position We need to check the screen and field at the same time https://youtu.be/mB5MuDy9GFw

SLIDE 3

/ 25 3 2008/7/17 RoboCup Symposium 2008 in Suzhou, China

Camera Augmented environment https://youtu.be/yGzA6hC9YY8 AR can alleviate the difficulty

SLIDE 4

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 4

Simulated environment in “Haribote” developed by team ARAIBO Real environment Augmented environment Real robot Virtual ball Virtual robot Intermediate role Perceptible Touchable

SLIDE 5

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 5

Camera Projector

Recognition Program Virtual Application Augmented Soccer Field System

SLIDE 6

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 6

Other Object Robot

Extraction of contours by a background subtraction method Identification of robots’ orientation by a template matching method

SLIDE 7

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 7

Real environment Virtual application Positions of real objects e.g., real robots Positions of virtual objects e.g., virtual ball and robots

SLIDE 8

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 8

Robots can interact the virtual ball

SLIDE 9

/ 25

 Essential for robot soccer

 No lost point, no lost game

 Learning has been difficult so far

2008/7/17 RoboCup Symposium 2008 in Suzhou, China 9

SLIDE 10

/ 25

 Human intervention  Time consuming  Motor failure

2008/7/17 RoboCup Symposium 2008 in Suzhou, China 10

Stroke its head for successful saving Spank its head for failed saving

Ex. Learning of goal saving skills in the real environment

https://youtu.be/9oHA-GH9JT8 https://youtu.be/3Pluuk20xqs

SLIDE 11

/ 25

 Gap from real environments

 Serious, especially for legged movements

2008/7/17 RoboCup Symposium 2008 in Suzhou, China 11

Real environment with human intervention, time consuming process, and motor failure fear Simple simulator without any difficulties Gap

SLIDE 12

/ 25

 To bridge the gap

 Using the movements of real robots

 To allow autonomous learning

 Using the convenience of virtual balls

2008/7/17 RoboCup Symposium 2008 in Suzhou, China 12

Simple simulator without any difficulties Augmented environment without human intervention Real environment with many difficulties

Hard mode Normal mode Easy mode

SLIDE 13

/ 25

 Acquire the map from states to actions

maximizing the sum of rewards

 Sarsa(λ) [Rummery and Niranjan 1994; Sutton 1996]

 Tile-coding (aka CMACs [Albus 1975])

2008/7/17 RoboCup Symposium 2008 in Suzhou, China 13

Agent (AIBO) Environment Action at State st Reward rt+1

SLIDE 14

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 14

Vh Vv

r

φ

X Y State representation (r, φ, Vv, Vh, X, Y)

(r, φ): Ball position
(Vv, Vh): Ball velocity
(X, Y): Robot position

(We removed the orientation

f the robot by considering the

PK situation only.)

SLIDE 15

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 15

Actions (10 kinds)

8 directional walks
stay; prepare enemy’s kicks
save; interrupt enemy’s goals

8 directional walk actions stay action save action

SLIDE 16

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 16

Rewards (punishments)

save_rewared: 0.5

when save is successful

save_punishment: -0.02

when save is failed

lost_punishment: -10

when a goal is scored

dist_reward: 1-|ydist|/112.5

when the game is over

passive_punishment: -0.0000001

when save is not selected 1 episode (game) is over, when the ball is out, when a goal is scored, or when save is successful ydist For getting near to the ball For accelerating the initial phase Aim: To stop the ball using save action as safely as possible

SLIDE 17

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 17

Initial strategy Learned strategy Success: Blue screen Failure: Red screen https://youtu.be/xCoSGsQHkRY https://youtu.be/C2YTw6d7xPw

SLIDE 18

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 18

save action stay action 8 walk actions × Robot’s actions

(r, φ): Ball position
(Vv, Vh): Ball velocity

SLIDE 19

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 19

save action stay action 8 walk actions × Robot’s actions

(r, φ): Ball position
(Vv, Vh): Ball velocity
(X, Y): Robot position

SLIDE 20

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 20

save action stay action 8 walk actions × Robot’s actions

(r, φ): Ball position
(Vv, Vh): Ball velocity
(X, Y): Robot position

SLIDE 21

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 21 500 1000 1500 2000 Number of Episodes 20 40 60 80 100 Success Rate

Number of Episodes Success Rate 1200 episodes 95 % success Average success rate in past 100 episodes

SLIDE 22

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 22

The action of the real robot The true positions of the virtual ball and robot https://youtu.be/HVx6TlHkPgw

SLIDE 23

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 23

https://youtu.be/F3-3o2oCP14

SLIDE 24

/ 25 50 100 150 200 Number of Episodes 20 40 60 80 100 Success Rate

Success Rate Number of Episodes

2008/7/17 RoboCup Symposium 2008 in Suzhou, China 24

45% Success 75% Success Starting from the result (95% success) in the simulator

SLIDE 25

/ 25

50 100 150 200 Number of Episodes 20 40 60 80 100 Success Rate

500 1000 1500 2000 Number of Episodes 20 40 60 80 100 Success Rate

Number of Episodes Success Rate

2008/7/17 RoboCup Symposium 2008 in Suzhou, China 25

Gap of legged movements 2000 Episodes 200 Episodes

SLIDE 26

/ 25

 Augmented soccer field system

 Intermediate role between simulated

environments and real environments

 Autonomous learning of goalie strategies

 Movements of real robots  Convenience of virtual balls

2008/7/17 RoboCup Symposium 2008 in Suzhou, China 26

SLIDE 27

/ 25 2008/7/17 RoboCup Symposium 2008 in Suzhou, China 27

Hayato Kobayashi, Tsugutoyo Osaki, Tetsuro Okuyama, Akira Ishino, and Ayumi Shinohara Tohoku University, Japan (Team Jolly Pochie)

Ball position Robot position We need to check the screen and field at the same time https://youtu.be/mB5MuDy9GFw

Camera Augmented environment https://youtu.be/yGzA6hC9YY8 AR can alleviate the difficulty

Simulated environment in “Haribote” developed by team ARAIBO Real environment Augmented environment Real robot Virtual ball Virtual robot Intermediate role Perceptible Touchable

Camera Projector

Extraction of contours by a background subtraction method Identification of robots’ orientation by a template matching method

Real environment Virtual application Positions of real objects e.g., real robots Positions of virtual objects e.g., virtual ball and robots

Robots can interact the virtual ball

 Essential for robot soccer

 Learning has been difficult so far

 Human intervention  Time consuming  Motor failure

Stroke its head for successful saving Spank its head for failed saving

https://youtu.be/9oHA-GH9JT8 https://youtu.be/3Pluuk20xqs

 Gap from real environments

Real environment with human intervention, time consuming process, and motor failure fear Simple simulator without any difficulties Gap

 To bridge the gap

 To allow autonomous learning

Simple simulator without any difficulties Augmented environment without human intervention Real environment with many difficulties

Hard mode Normal mode Easy mode

 Acquire the map from states to actions

maximizing the sum of rewards

 Sarsa(λ) [Rummery and Niranjan 1994; Sutton 1996]

Agent (AIBO) Environment Action at State st Reward rt+1

Vh Vv

r

φ

X Y State representation (r, φ, Vv, Vh, X, Y)

(We removed the orientation

PK situation only.)

Actions (10 kinds)

8 directional walk actions stay action save action

Rewards (punishments)

when save is successful

when save is failed

when a goal is scored

when the game is over

when save is not selected 1 episode (game) is over, when the ball is out, when a goal is scored, or when save is successful ydist For getting near to the ball For accelerating the initial phase Aim: To stop the ball using save action as safely as possible

Initial strategy Learned strategy Success: Blue screen Failure: Red screen https://youtu.be/xCoSGsQHkRY https://youtu.be/C2YTw6d7xPw

save action stay action 8 walk actions × Robot’s actions

save action stay action 8 walk actions × Robot’s actions

save action stay action 8 walk actions × Robot’s actions

Number of Episodes Success Rate 1200 episodes 95 % success Average success rate in past 100 episodes

The action of the real robot The true positions of the virtual ball and robot https://youtu.be/HVx6TlHkPgw

https://youtu.be/F3-3o2oCP14

Success Rate Number of Episodes

45% Success 75% Success Starting from the result (95% success) in the simulator

Number of Episodes Success Rate

Gap of legged movements 2000 Episodes 200 Episodes

 Augmented soccer field system

environments and real environments

 Autonomous learning of goalie strategies

Air hockey game using our system