Automatic Singing System of Japanese Songs in Chinese 6th Nov. 2008 - PowerPoint PPT Presentation

Automatic Singing System of Japanese Songs in Chinese 6th Nov. 2008 CJNLP-2008 EHARA Terumasa Tokyo University of Science, Suwa CJNLP-2008 1

Motivation • To enhance the cultural exchange between Japan and China • To make Chinese automatic singing system • To improve the quality of singing software to it of the human singing CJNLP-2008 2

Procedure of automatic singing Japanese song Musical score Lyrics Machine-translation Tone / duration / post-editing Speaking in Chinese Converting to singing voice Singing in Chinese CJNLP-2008 3

Converting to singing voice Speaking voice Musical score STRAIGHT* (analyzing part) F0 contour Spectrum envelope Duration and F0 control STRAIGHT (synthesizing part) Singing voice * Kawahara, H. et al., “ Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds ” , Speech Communication, Vol. 27, pp. CJNLP-2008 4 187–207, 1999.

Duration control • piecewise linear function inp - out 000-000 360-200 540-400 630-650 830-750 CJNLP-2008 5

F0 control • Model 0 F0(t) = Y0 • Model 1 F0(t) = Y0 + (a1 ＊ t + a0) • Model 2 F0(t) = Y0 + (a2 ＊ t 2 + a1 ＊ t + a0) CJNLP-2008 6

Controlled F0 (model 0) Blue line ： original F0 Red line ： controlled F0 CJNLP-2008 7

Preliminary study Model 0 Model 1 Model 2 妈 麻 马 骂 Original CJNLP-2008 8

9 Experiment CJNLP-2008

Original Japanese song (Spring has come) CJNLP-2008 10

Machine-translation / post-editing 春が来た 春天来临了 春天已来了 春が来た 春天来临了 春天已来了 どこに来た 来了哪里 来到了哪里 山に来た 来了到山 来到了山里 里に来た 来了村落 来到了村落 野にも来た 也来了野地 也来到野地 Machine-translation Post-editing CJNLP-2008 11

Convert to a singing voice 来 到 了 哪 里 来 到 了 哪 里 ～～ CJNLP-2008 12 Speaking Singing (F0 model 0)

13 Duration control 哪 里 ～～～～ 了 到 来 里 哪 了 到 来 CJNLP-2008

14 Speaking voice CJNLP-2008

15 Singing voice CJNLP-2008

Conclusion • To develop an automatic singing system of Japanese songs in Chinese. • To translate Japanese lyrics into Chinese using machine-translation and post-editing. • To convert a speaking voice to a singing voice in Chinese. CJNLP-2008 16

Future works • To control Chinese syllabic tones appropriately. • To use more sophisticated F0 and duration control technique* • To extend our method to more complicated Japanese songs other than children’s songs * Takeshi Saitou et. al : Vocal conversion from speaking voice to singing voice using STRAIGHT, Interspeech 2007, Aug. 2007. CJNLP-2008 17

Acknowledgement • Providing STRAIGHT in research use Prof. KAWAHARA Hideki (Wakayama univ.) • Collaborators Assistant prof. OTA Kenko Former graduate student KUBO Yasushi Under graduate student KOBAYASHI Minoru, YAMAKOSHI Yuuta Research student Qigeqi CJNLP-2008 18

19 Thank you for your attention CJNLP-2008