Shimon An Intelligent Music-Playing Robot Capable of Improvising - - PowerPoint PPT Presentation

▶
shimon
SMART_READER_LITE
LIVE PREVIEW

Shimon An Intelligent Music-Playing Robot Capable of Improvising - - PowerPoint PPT Presentation

Sep 12, 2022 306 likes •812 views

13.01.2020 Shimon An Intelligent Music-Playing Robot Capable of Improvising with Humans Vincent Rolfs Shimon: An intelligent music-playing robot capable of 1 improvising with humans Once upon a time... Vincent Rolfs Shimon: An

slide-1
SLIDE 1

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Shimon

An Intelligent Music-Playing Robot Capable of Improvising with Humans

1 13.01.2020

slide-2
SLIDE 2

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Once upon a time...

2

slide-3
SLIDE 3

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

A “player piano”

3

[1]

slide-4
SLIDE 4

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Topics for today

  • Why musical robots?
  • Introduction to Shimon
  • Physical architecture
  • Algorithms for Musicianship
  • Related work and outlook

4

slide-5
SLIDE 5

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Why musical robots?

5

slide-6
SLIDE 6

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Motivation: Why musical robots?

6

Interdisciplinary education Play in band settings but with full musical control Overcome human shortcomings in music

slide-7
SLIDE 7

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Introduction to Shimon

7

slide-8
SLIDE 8

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

What is Shimon?

An intelligent music-playing robot Plays full range of marimba using four arms Developed by Guy Hoffmann and Gil Weinberg

8

[2, 3, 4, 5]

slide-9
SLIDE 9

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

What can Shimon do?

9

Play set melodies alone

slide-10
SLIDE 10

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Shimon playing a simple melody

10

[6]

slide-11
SLIDE 11

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

What can Shimon do?

Play set melodies alone

11

Play set melodies with others in a call-and-response fashion

slide-12
SLIDE 12

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Shimon playing call-and response (slow)

12

[7]

slide-13
SLIDE 13

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Shimon playing call-and response (fast)

13

[7]

slide-14
SLIDE 14

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

What can Shimon do?

Play set melodies alone Play set melodies with others in a call-and-response fashion

14

Play improvisation together with other actors

slide-15
SLIDE 15

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Shimon improvising

15

[8]

slide-16
SLIDE 16

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Physical Architecture

16

slide-17
SLIDE 17

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Physical architecture: Goals

17

Large movements for visibility Fast movements for virtuosity Wide range of note combinations

slide-18
SLIDE 18

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Physical architecture: High-level view

18

  • Four arms, running

along a shared rail using linear actuators

  • Each arm can

reach an acceleration of up to 3g (105 km/h per second)

  • Each arm can

cover a full octave in 0.25 seconds [5]

slide-19
SLIDE 19

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Physical architecture: Arms in detail

19

  • Each arm contains two

mallets, one for “black”,

  • ne for “white” keys
  • Both mallets are

controlled by an ON/OFF rotary solenoid

  • These actuators are

positioned at the crosshairs in the figure [5]

slide-20
SLIDE 20

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Problem: Collision avoidance

20

[5]

slide-21
SLIDE 21

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Controlling of the arm motors

21

Collisions of arms must be avoided Movement of arms should look natural Solution: Slider Manager Compute position update

  • f each arm

Avoid collisions using simulated springs Animation System gives natural movements

slide-22
SLIDE 22

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Algorithms for Musicianship

22

slide-23
SLIDE 23

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

The three types of interaction modules

23

Call-and-response Opportunistic overlay Rhythmic phrase-matching improvisation

slide-24
SLIDE 24

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

The three types of interaction modules

24

Call-and-response Opportunistic overlay Rhythmic phrase-matching improvisation

  • Plays a set melody in response to

a musical sequence played at an arbitrary tempo

  • Beat-matched and synchronized,

starting on time, without delay

slide-25
SLIDE 25

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

The three types of interaction modules

25

Call-and-response Opportunistic overlay Rhythmic phrase-matching improvisation

  • Focuses on choreographic

movement

  • Plays a sparse improvisation that

is beat-matched, synchronized and chord-adaptive

slide-26
SLIDE 26

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Shimon using the opportunistic overlay module

26

[7]

slide-27
SLIDE 27

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

The three types of interaction modules

27

Call-and-response Opportunistic overlay Rhythmic phrase-matching improvisation

  • Beat-matched and

chord-synchronized improvisation

  • Tries to match style and density of

human player

slide-28
SLIDE 28

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Shimon using the improvisation module

28

[8]

slide-29
SLIDE 29

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

The improvisation algorithm: Finding probabilities

29

1 Quantize the bar into 16 beats and declare probabilities pi, k for arm i to strike at beat k. 2 Listen to one bar of human play and quantize the playing into 16 beats. 3 For each of the 16 beats, cluster the notes played in that beat into 4 bins (one of for each arm) by pitch.

slide-30
SLIDE 30

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

The improvisation algorithm: Finding probabilities

30

slide-31
SLIDE 31

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

The improvisation algorithm: Finding probabilities

31

1 Quantize the bar into 16 beats and declare probabilities pi, k for arm i to strike at beat k. 2 Listen to one bar of human play and quantize the playing into 16 beats. 3 For each of the 16 beats, cluster the notes played in that beat into 4 bins (one of for each arm) by pitch. 4 Declare hi, k as 1 if the human played a note in bin i at beat k, and 0 otherwise. Then apply

pi, k ← λhi, k + (1 - λ) pi, k.

slide-32
SLIDE 32

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

The improvisation algorithm: Playing

32

1 Detect the current chord played by comparing human play to all chords in the piece. 2 Position the arms according to the detected chord. Each arm now targets one key. 3 Strike arm i at beat k with probability pi, k. 4 …Profit!

slide-33
SLIDE 33

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Related work and

  • utlook

33

slide-34
SLIDE 34

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Compressorhead, a band consisting only of robots

34

slide-35
SLIDE 35

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Compressorhead, a band consisting only of robots

35

[9]

slide-36
SLIDE 36

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Haile, a robotic percussionist by Weinberg et al.

36

slide-37
SLIDE 37

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Haile, a robotic percussionist by Weinberg et al.

37

[10]

slide-38
SLIDE 38

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

A “hyperinstrument” by Kapur et al.

38

slide-39
SLIDE 39

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

A “hyperinstrument” by Kapur et al.

39

[11]

slide-40
SLIDE 40

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Strengths and weaknesses of Shimon

40

Algorithms depend strongly

  • n chord classification

Can improvise in a variety of genres Tempo detection is very simplistic Is able to play in sync with humans in a natural way Improvisation does not improve over time Can improvise on the fly and invent new melodies Many new advances are not published Shimon is continually improved

slide-41
SLIDE 41

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

The future of Shimon

41

Shimon as a movie score composer DeepShimon: Using neural networks Shimon’s head: Further research [12]

slide-42
SLIDE 42

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Thank you for your attention!

42

slide-43
SLIDE 43

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

References

43

slide-44
SLIDE 44

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

References

[1] L. Mickey. RagTime Scott Joplin Pianola Autoleon Triumph Auto London 1924. URL: https://www.youtube.com/watch?v=4iymj0gZpj4 (visited on 01/12/2020). [2] Shimon Robot. Gallery. URL: https://www.shimonrobot.com/gallery (visited on 01/12/2020). [3] Georgia Tech Center for Music Technology. People. URL: https://gtcmt.gatech.edu/people (visited on 01/12/2020). [4] G. Hoffmann. Bio. URL: http://guyhoffman.com/bio/ (visited on 01/12/2020). [5] G. Hoffman and G. Weinberg. “Interactive improvisation with a robotic marimba player”. In: Autonomous Robots 31.2 (Oct. 2011), pp. 133–153. issn: 1573-7527. doi: 10.1007/s10514-011-9237-0. URL: https : / /doi.org/10.1007/s10514-011-9237-0. [6] gerbilsproductions. Shimon Robot plays True Romance Theme. URL: https://www.youtube.com/watch?v=PjWSiaMQn08 (visited on 01/12/2020). [7] gerbilsproductions. Human-Robot Jazz Improvisation (Full Performance). URL: https://www.youtube.com/watch?v=qy02lwvGv3U (visited on 01/12/2020).

44

slide-45
SLIDE 45

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

References

[8] vpro vrije geluiden. Robot Shimon and friends in Concert (VG Live). URL: https://www.youtube.com/watch?v=Rx-QSphvr20 (visited on 01/12/2020). [9] S. Kanatyev. Compressorhead - Smells Like Teen Spirit (Nirvana Cover) (live in Moscow, Russia). URL: https://www.youtube.com/watch?v=TaY7vohLXK4 (visited on 01/12/2020). [10] leech. Haile, the interactive robot drummer. URL: https://www.youtube.com/watch?v=veQS6tsogAA (visited on 01/12/2020). [11] TEDx Talks. Composing, programming, and performing musical robots: Ajay Kapur at TEDxCalArts. URL: https://www.youtube.com/watch?v=GsqiJqMFcvM (visited on 01/12/2020). [12] Purdue University. URL: https://www.purdue.edu/convocations/wp-content/uploads/2016/06/Shimon_1920x552.jpg (visited on 01/12/2020). Icons designed by Freepic from Flaticon.

45

slide-46
SLIDE 46

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Appendix

46

slide-47
SLIDE 47

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Appendix

47

[5]

slide-48
SLIDE 48

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Controlling of the arm motors

48

[5]

slide-49
SLIDE 49

Vincent Rolfs Shimon: An intelligent music-playing robot capable of improvising with humans

Controlling of the mallet motors

49

[5]