Cyber Performance for Network Performing Arts Production Workshop - - PowerPoint PPT Presentation

Cyber Performance for Network Performing Arts Production Workshop London, May 2015

Locations

• Prague: Music and Dance Faculty of the Academy of Performing Arts
• London: Royal College of Music
• Distance: ~1000 km / 650 miles
• Performance: real-time playing music together over distance

using 4K Gateway hardware HAMU RCM

Matouš Pěruška – violin Alison Rhind - piano

Music

1) J. Brahms: Violin concerto in D major Op. 77, 1st movement 2) M. Ravel: La Cigane

Prague London

Prague

Network

Frankfurt Amsterdam London

Sven Ubik CESNET, Prague, Czech Republic

Low-latency transmissions for playing together and cyber performances

Royal College of Music London, 6 May 2015

• Czech Academic Network, 100 Gbps backbone, connected to Géant
• Now operating “e-infrastructure” (network, cloud computing & storage,

remote collaboration services) for universities and research institutions

• National and EU research projects

CESNET

Motivation

• Low-latency audio and video transmissions can enable real-time

collaboration in multiple fields, including performing arts

• Driven by practical uses cases as well as curiosity
• We cannot beat the propagation delay, but we can minimize


additional processing delay, by building a real-time system

• Movie industry offered 4K experience several years ago, why not

to do 4K distributed concerts

• 4K Gateway device, up to 2x 4K video, up to 96 audio channels
• FPGA-based, HW processing connector-to-connector
• approx. 3 ms latency in uncompressed mode for audio and 4K video
• in JPEG 2000 mode 5 frames / subframe latency
• multi-point connections are possible (8 locations)
• spinoff company created to sell a commercial version
• in 2015 lightweight version (fewer video channels, low-latency

compression)

Hardware development

4K Gateway 10GE to Géant HDMI 4K Blackmagic Design camera 2x XLR balanced audio output

Prague / London side equipment

studio mixer audio input 4xHD-SDI 6G-SDI to 4xHD-SDI 4xHD-SDI to HDMI audio dis-embedder 4K TV video from remote side 6G-SDI audio embedder

Applications

§ Playing together over distance for teaching, rehearsals, etc.


• minimum latency required

§ Distributed performances for cultural exchange


• high-quality video required

§ Medical, industrial, post-processing, …

eMusic project

• 18-months GN3plus OpenCall project
• Partners:
• 3 academic networks (CESNET, GARR, JANET)
• 3 musical schools (HAMU in Prague, Conservatorio Tartini in

Trieste, University of Edinburgh)

• Goal: to verify if real-time collaboration in music over Géant can be

improved by the Bandwidth on Demand (BoD) service

• Experiments in playing together, remote sound mixing, medical

voice inspection https://intranet.geant.net/JRA0/eMusic/SitePages/Home.aspx

Prague Bratislava Vienna Athens

Playing together for the Geant Symposium, February 2015

Athens, Megaron Conference Center Prague, SAGElab laboratory

• Approx. 1000 km

away

• Marry keys quartet in Prague

(piano, drums, saxophone, vocal)

• Sebastiano Frattini in Athens

(violin)

• Several jazz-style pieces

Music

SAGElab

• Network Visualization Lab of CESNET + Czech Tech. Univ.
• LCD wall >8K resolution (9600 x 4320 pixels), bezels <5 mm

between tiles

• 10 Gbps connection to Géant

Dance with delay

• Experimental evaluation of delay effects on dance performance
• Dance was especially selected to be sensitive to delay, by

exchaging a “virtual ball”

• Performed in the SAGElab Visualization Laboratory in Prague

Dance with delay

3D live music between continents

Live music harp and guitar in the Academy of Performing Arts in Prague Transmission using 4K Gateway, January 2013 University

• f Hawaii

Cyber Performances

• „Dancing in space“, for APAN 38 meeting in Taiwan, August 2014
• Live performance between Prague, Barcelona, Miami and Taiwan

To be filled

Chicago APAN38 NanTou

Amsterdam Barcelona

Miami

Prague

NanTou Taipei

10G 20G 20G 20G 10G 10G 10G

Artists on Prague site

Ivan Vokac is a graduate of Prague Conservatory of Music. For the time being Ivan is studying with

• Prof. Miroslav Petras at the Academy of Performing Arts in Prague.

Stage 2 | Dancing in Space

Ivan Vokac

• V. Cello

Jan Mracek Violin Lukas Klansky Piano

Artists

Musical trio in Prague: Choreography and dance in Barcelona: Live video interaction Choreography Dance

Alain Baumann Rosa Sánchez Victoria Macarte

Music in Miami: Andrew Larson

New World Symphony

Dance in Taiwan: Yi Tzu Lee Winnie

TW Dancer BC Dancer Composed Picture CZ Trio US Cellist Normal Background

Black Background

Video capture and projection

Prague Barcelona Miami Taiwan

Main stage in Nantou, Taiwan

Musical trio in Prague Music in Miami: Dance in Barcelona Dance in Taiwan

Main stage in Nantou, Taiwan

Cyber performance for APAN meeting in Daejeon, Korea

• Live music at the Academy of Performing Arts in Prague
• Dancers in Barcelona, Salvador, Daejeon
• Real-time performance between 4 countries, 3 continents, approx.

100 people involved, main audience in Daejeon at APAN 36 meeting

• 4K video from Prague transmitted by 4K Gateway

Cyber performance for APAN meeting in Daejeon, Korea

Delay components

• Network propagation delay
• Transmission equipment (codec + processing)
• Camera, TV / projector
• Video convertors

Video converters

• Between HDMI, SDI, analog, etc.
• Converters without frame buffer (cannot change frame rate)

good for playing together, several µs latency

• Converters with frame buffer (can also change frame rate)

bad for playing together, tens of millisecond latency

Audio delay measurement

Signal generator, 1 kHz sin wave in bursts Two channel

• scilloscope

audio chain

Audio delay measurement

embedder + disembedder (2x AJA HD10AMA) audio delay 2.2 ms embedding + disembeding as a function in video converters (BMD HDMI to SDI + AJA Hi5-3G) audio delay 800 microseconds

Video delay measurement

video generator, SDI output video chain Two channel

• scilloscope

2x digital to analog

Video delay measurement

Adding video interface convertors HDMI to SDI + SDI to HDMI (BMD + AJA Hi5-3G) video delay 8 microseconds

Audio + video delay measurement

Loop Prague – Athens – Prague (a pair of L3 tunnels) 2x 4K Gateway, embedder, disembedder total audio and video delay 48 ms (network RTT 36 ms, equipment 12 ms) =>

• ne way delay will be 30 ms

Camera delay measurements

Flash light Two channel

• scilloscope

SDI to analog Photodetector

Camera delay measurement

Canon Legria M56 (consumer HDMI camera): video delay 100 ms GoPro Hero4 Black Edition: video delay 60 ms Sony XC-HR70 (analog): video delay 25 ms Blackmagic Design 4K: video delay 60 ms

TV / projector delay measurements

Two channel

• scilloscope

SDI to analog Photodetector video generator, SDI output SDI to HDMI / DVI

Placement of photodetector

Be aware, that monitor changes a frame from top to bottom, Placement of photodetector matters, here captured by GoPro in 720p240 mode

Projector Optoma LH501 (Full HD) HDMI / DVI / analog input video delay 60 ms

TV / projector delay measurements

ACER Full HD LCD monitor HDMI input DVI input video delay 16 / 36 ms

• Technology that combines low latency transmissions for

playing together with high-quality video is available

• Network requirements are high, but it was shown that it can

be done even over long distances

• In addition to network propagation delay, additional delay is

now mostly in camera and TV / projector (transmission equipment is negligible)

Technology summary

Observations

• It seems that the order of importance for good feeling in

playing together is: 1) good sound, 2) feeling other musicians and 3) good vision of other musicians

• Good sound means not only small delay, but also the quality of

sound capture and reproduction, volume, dynamics, space and echo

• For what it was considered useful? Mostly for rehearsals, also

for occasional live performances, but probably not for recording

Thank you for your attention!

Sven Ubik ubik@cesnet.cz