Panoramic video content distribution in the xTV project Peter Quax, - - PowerPoint PPT Presentation

Panoramic video content distribution in the xTV project

Peter Quax, Panagiotis Issaris, Wouter Vanmontfort, Wim Lamotte Hasselt University, Belgium

Panoramic/omni-directional Video Concept

§ Comparison : Google streetview with video instead of still images

Panoramic/omni-directional Video Concept

§ Mobile setups

Panoramic/omni-directional Video Concept

§ Panoramic super high-definition (22 HD cameras)

Panoramic/Omni-directional video workflow

Processing workflow

§ Pre-processing delivers a stitched and pre- warped image @ at least 4xFull-HD resolution (or higher if desired/depending on camera)

Recordings made at Main Square Rock festival @ Arras (France)

Streaming panoramic video to the end-user

Streaming panoramic video to the end-user

What if we could save 80% on bandwidth utilization and still deliver video in its

• riginal resolution ?

High-level goals

Focus on resolution and quality Ensure low interactivity delays Lower bandwidth and processing requirements Compatibility with existing network technologies and devices

Possible solutions (1) : rescaling

4 x Full-HD resolution Full-HD resolution Compress aggressively Decode entire frame and crop required viewport

Possible solutions (1) : rescaling

4 x Full-HD resolution Compress aggressively Decode entire frame and crop required viewport Full-HD resolution

Possible solutions (2) : transcoding

4 x Full-HD resolution Very fast compression Low quality Decode viewport and display Crop required viewport server-side

Possible solutions (2) : transcoding

4 x Full-HD resolution Crop required viewport server-side Very fast compression Low quality Decode viewport and display

Proposed solution : segmentation

Segmentation – Preparation

Segmentation – Preparation

Segmentation - Encoding

Segmentation - Encoding

Segmentation - Delivery

HTTP REQUESTS SEGMENT DELIVERY

Segmentation - Visualization

Segmentation - Visualization

Segmentation - Visualization

Evaluation

Evaluation - compression

Parameter Value H.264 Profile Main Preset Medium Tune Fastdecode GOP 16 Segment size 256x216 (unless indicated otherwise) Number of frames 200 Total sequence resolution 3840 x 2160

Evaluation - compression

§ Storage/compression overhead (on disk)

§ Versus non-segmented 200 frame sequence @ full resolution § Remark for segmentation approach : this is not what is sent to the client ! (only viewport, baseline has to send everything)

Evaluation - compression

§ Encoding speed increase

§ Based on single host (with multiple cores) § Baseline1 uses libx264, multi-threaded § Baseline2 uses libx264 with slice support (real-time optimized)

Evaluation - compression

§ Encoding scalability

§ Over multiple hosts § Scaling over hosts not trivial in case of standard codecs (real- time / motion vector limitation)

Evaluation - quality

§ Remarks on quality comparison

§ Individual segments need to be encoded using the same quality parameter (VBR, not CBR) to avoid patchwork-effect in tiles § Not all segments compress equally well, total bandwidth required depends on the scene that is being watched § Some examples and terminology in following slides

Evaluation - quality

§ Least amount of bandwidth (MIN)

§ Consider case where user is looking at the sky § Minimal motion, best compression

Evaluation - quality

§ Most bandwidth consumption (MAX)

§ Consider case where user is looking at the audience § Lots of motion, worst compression

Evaluation - quality

§ Typical bandwidth usage (AVG)

§ Consider case where user is looking at main character on stage § Medium motion, medium compression

Evaluation - quality

Parameter Value Segment size 256x216 Horizontal segments in viewport 4 Vertical segments in viewport 3 Number of frames 3454 Total sequence resolution 3840 x 2160

Evaluation - quality

§ Quality comparison (PSNR)

§ Instruct the baseline codec to use as much bandwidth as the selected viewport (segmentation approach) requires under min/ max/avg conditions (baseline can do CBR !) § Note : the baseline needs to compress/transmit the entire frame (not only the viewport as in segmentation approach)

Evaluation - quality

§ Quality comparison (SSIM)

§ Equal conditions as before § Given equal bandwidth allowance, segmentation approach delivers much higher quality

Evaluation – streaming characteristics

§ Bandwidth utilization under different streaming conditions/container formats § HTTP Live streaming

§ Uses 10 second fragments

Evaluation – streaming characteristics

§ MKV over HTTP

§ Buffers are flushed completely as soon as they are filled

Evaluation – streaming characteristics

§ MP4 over HTTP

§ Buffers are gradually flushed and re-filled as required

• > TCP flow control

Evaluation – seeking comparison

§ Rapid seeking is required for segmentation approach

§ Free/open source codec implementations support seeking to nearest I-frame only (and are often broken in their support) § We need frame-precise seeking !

§ Adapt the decoding speed to quickly find the required frame

§ Support also patched in for HLS (now part of FFMPEG)

§ Container choice has an impact on seeking performance

Conclusions

§ Goals achieved ? Yes !

§ Bandwidth reduction from 45mbps for full resolution video to 4Mbps § Back-end scalability is ensured through parallelization § Speed of interactivity optimized through pre-caching and enhanced seeking methods § Solution is fully compliant with existing distribution methods and current-generation tablet hardware

§ Only ideas/theory ? No !

§ We have a fully working prototype based on a second screen setup (STB and tablet) § User experience is under investigation, early feedback is very positive

Questions ?

peter.quax@uhasselt.be