DECLARATIVE AR AND IMAGE PROCESSING ON THE WEB WITH XFLOW Felix - - PowerPoint PPT Presentation

▶
declarative ar and image processing on the web
SMART_READER_LITE
LIVE PREVIEW

DECLARATIVE AR AND IMAGE PROCESSING ON THE WEB WITH XFLOW Felix - - PowerPoint PPT Presentation

Jan 19, 2024 371 likes •823 views

DECLARATIVE AR AND IMAGE PROCESSING ON THE WEB WITH XFLOW Felix Klein, Dmitri Rubinstein, Kristian Sons, Farshad Einabadi, Stephan Herhut, Philipp Slusallek 1 MOTIVATION THE WEB IS READY FOR AR Fast JavaScript WebGL, WebCL upcoming

slide-1
SLIDE 1

DECLARATIVE AR AND IMAGE PROCESSING ON THE WEB

WITH XFLOW

Felix Klein, Dmitri Rubinstein, Kristian Sons, Farshad Einabadi, Stephan Herhut, Philipp Slusallek

1

slide-2
SLIDE 2
slide-3
SLIDE 3

MOTIVATION

slide-4
SLIDE 4

THE WEB IS READY FOR AR

Fast JavaScript WebGL, WebCL upcoming getUserMedia, WebRTC Geolocation, Orientation, Motion Problem: Usability

>

slide-5
SLIDE 5

HTML

Declarative, High-Level Pretty straight forward!

WEBGL

Imperative, Low-Level Pretty difficult...

NEW WEB IS POWERFUL AND COMPLICATED

<div> <p> This is some declarative HTML </p> <img src="blub.png" alt="with an image too" > <a href="someWhereElse.html" > ...and a famous link. </a> </div> var canvas = document.getElementById("cvs"); initGL(canvas); initShaders(); initBuffers(); gl.clearColor(0.0, 0.0, 0.0, 1.0); gl.enable(gl.DEPTH_TEST); drawScene();

Plz make 3D simpler for Web developers kthxbye.

>

slide-6
SLIDE 6

DECLARATIVE 3D FOR THE WEB

W3C Community Group Goals

Extension to HTML5 for 3D Content Integrated with DOM, CSS etc. Accessible to Web developers

Evaluation Platform (polyfills)

X3DOM - X3D inside the DOM XML3D & Xflow

slide-7
SLIDE 7

XML3D

An extension to HTML5 for 3D graphics. Presented at Web3D 2010 Key features

Declarative 3D inside of Web document Minimal extension to HTML Generic core - matching modern GPUs

<html xmlns="http://www.w3.org/1999/xhtml"> <!-- ... --> <body> <xml3d xmlns="http://www.xml3d.org/2009/xml3d"> <!-- ... --> <group shader="shaders.xml#xml3dTex" > <mesh src="cube.json" /> </group> </xml3d> </body> </html>

slide-8
SLIDE 8
slide-9
SLIDE 9

XFLOW

Declarative Data Processing based on Dataflows. Presented at Web3D 2012 Key features

Declare Dataflows inside the Web document Dataflows execution parallelized / mapped on GPU Generic Design - reusable operators for processing

<data id="wave" compute="(position, normal) = xflow.mywave(pos, norm, str, len, phase)" > <float name="str">0.01</float> <float name="len">40.0</float> <float name="phase">0.0</float> <data compute="(pos, norm, texcoord, index) = xflow.mygrid(size)"> <int name="size">50</int> </data> </data>

slide-10
SLIDE 10
slide-11
SLIDE 11

GOAL

Extend XML3D & Xflow

Support for images processing Support for augmented reality

Go for a minimal extension

slide-12
SLIDE 12

IMAGE PROCESSING

What do we need for image processing with Xflow?

slide-13
SLIDE 13

XFLOW FOR REGULAR MESHES

Use compute attribute

access typed arrays (e.g. with float data)

  • utput typed arrays

<data id="morphedPos" compute="pos = xflow.morph(pos, pAdd, weight)" > <float3 name="pos" >1.0 0.04 -0.5 ...</float3> <float3 name="pAdd" >0.0 1.0 2.0 ...</float3> <float name="weight" >0.45</float> </data>

slide-14
SLIDE 14

XFLOW WITH IMAGES

Use texture element to pass images or videos to Xflow

  • perators

Xflow operator generates new image Output images can have arbitrary size Default: output image same size as input image

<!-- Process image --> <data compute="grayImage = xflip.grayscale(image)"> <texture name="image" > <img sr c="someImage.png" /> </texture> </data> <!-- Process videos --> <data compute="grayImage = xflip.grayscale(image)"> <texture name="image" > <video src="someVideo.avi" autoplay /> </texture> </data>

slide-15
SLIDE 15

CONNECT PROCESSED IMAGES TO 3D

Used processed image as surface texture

Link processed images to shader Link shader to group

<shader id="ipShader" script="urn:xml3d:shader:phong" > <data compute="diffuseTexture=xflip.grayscale(image)"> <texture name="image" > <img src="someImage.png" /> </texture> </data> </shader> <group shader="#ipShader" > <mesh src="squareMesh.xml" /> </group>

slide-16
SLIDE 16

USE PROCESSED IMAGE WITHOUT 3D: XFLIP

Runs indepentent of XML3D Display processed images with xflip element

Exactly like regular images

<!-- Process image --> <xflip id="ipData" compute="output = xflip.grayscale(input)" > <texture name="input" > <img src="someImage.png" /> </texture> </xflip> <!-- Display process images in HTML --> <h3>Input:</h3> <xflimg src="#ipData" srcName="input" />

slide-17
SLIDE 17

EXAMPLE: IMAGE PROCESSING

slide-18
SLIDE 18

AUGMENTED REALITY

slide-19
SLIDE 19

THE ACTUAL XFLOW CODE

Based on JSARToolKit Output

transforms: world-space transformation for each marker visibilities: visibility flag for each marker perspective: matrix for intrinsic camera transformation

<data id="arBase" compute="transforms, visibilities, perspective = xflar.detect(arvideo, markers, threshold)"> <texture name="arvideo"> <video autoplay="true" src="ar_marker.ogg"></video> </texture> <int name="markers">22 64</int> <int name="threshold" >110</int> </data>

slide-20
SLIDE 20

CONNECTION WITH DOCUMENT

slide-21
SLIDE 21

UPDATE INTRINSIC CAMERA PARAMETERS

Refer xflow element via perspective attribute Expected to contain perspective value of type float4x4

<!--AR Data (as before) --> <data id="arBase" compute="transforms,visibilities,perspective= ..."> ... </data> <!-- Connect to view --> <view id="mainView" perspective="#arBase" />

slide-22
SLIDE 22

UPDATE POSITION OF GEOMETRY

<!--AR Data (as before) --> <data id="arBase" compute="transforms,visibilities,perspective= ..."> ... </data> <!--Extract transform for specific marker --> <data id="obj1Xfm" compute="transform = xflow.select(index, transforms)"> <int name="index">0</int> <data src="#arBase"/> </data> <!--Apply transform to a group node containing geometry--> <group transform="#obj1Xfm" > <!-- Geometry placed relative to Marker 0 --> </group>

slide-23
SLIDE 23

UPDATE VISIBILITY OF MESH

<!--AR Data (as before) --> <data id="arBase" compute="transforms,visibilities,perspective= ..."> ... </data> <!--Extract visibility for specific marker --> <data id="obj1Vis" compute="visibility = xflow.select(index, visibilities)"> <int name="index">0</int> <data src="#arBase"/> </data> <!--Assign visibility flag to shader--> <shader id="obj1Shader" script="urn:xml3d:shader:phongvs" > <float3 name="diffuseColor">1.0 0.4 0.2</float3> <float name="ambientIntensity">0.2</float> <data src="#obj1Vis" /> </shader>

slide-24
SLIDE 24

Basic AR application

Fiducial Marker Detection Based on JSARToolkit

Fully Declarative

No additional JavaScript

FIRST RESULTS FOR AR

slide-25
SLIDE 25

AR IMPROVEMENTS

slide-26
SLIDE 26

We simply overlap a video with an xml3d element

Two videos: one in 2D layout,

  • ne in 3D scene

3D content delayed by marker algorithm + rendering

PROBLEM#1: VIDEO AND 3D NOT SYNCHRONIZED

slide-27
SLIDE 27

Use canvas instead of video Draw video frame currently processed by Xflow Draw it right before rendering

Drawing in canvas works only via JavaScript So far, we have only declarative content

WORKAROUND: DRAW VIDEO INTO CANVAS

Need JavaScript integrated with Xflow and Rendering

>

slide-28
SLIDE 28

INTRODUCTION: DATA OBSERVERS

Designed after DOM MutationObservers Efficiently integrated

Responses between dataflow computation and rendering

Great way to efficiently integrate scripts for more flexibility

var observer = new XML3DDataObserver(function (records, observer) { var arvideo = records[0].result.getValue("arvideo"); if (arvideo) drawCanvas(canvasElement, arvideo); });

  • bserver.observe(dataElement, { names: ["arvideo"] });
slide-29
SLIDE 29

JSARToolKit expects threshold

Used to pre-process image into black/white version

Fixed threshold only works for certain lighting conditions

Add a way to automatically adapt threshold to lighting conditions

PROBLEM#2: FIXED THRESHOLD

slide-30
SLIDE 30

POWER OF GENERIC DESIGN!

Reuse generic Xflow image processing operators

Grayscale image, compute histogram

Use new xfloar.getOtsuThreshold

compute threshold from histogram

<data id="arBase" compute="transforms, visibilities, perspective = xflar.detect(arvideo, markers, threshold)"> <int name="markers">22 64</int> <data compute="threshold = xflar.getOtsuThreshold(hgram)"> <data compute="hgram = xflip.createHistogram(grvideo,channel)"> <int name="channel">0</int> <data compute="grvideo = xflip.grayscale(arvideo)"> <texture name="arvideo"> <video autoplay="true" src="ar_marker.ogg"></video> </texture> </data> </data> </data> </data>

slide-31
SLIDE 31

RESULT: USING OTSU THRESHOLD COMPUTATION

slide-32
SLIDE 32

PARALLELIZATION

slide-33
SLIDE 33

PARALLELIZATION WITH RIVERTRAIL

Parallel JavaScript API by Intel Labs (RiverTrail project)

Data-parallelism with regular JavaScript functions Available as Firefox Plug-in Also first native support in Firefox Beta

Effectively for image processing

Process each pixel in parallel

slide-34
SLIDE 34

PERFORMANCE RESULTS

Xflow Operator Time in ms Speedup Factor Sequential Parallel Convolution 3x3 236 52 4.5 Convolution 5x5 502 89 5.6 Convolution 7x7 880 141 6.2 Convolution 9x9 1420 218 6.5 Image size: 896x512 pixel; Processor: Intel Core i7-2670QM

slide-35
SLIDE 35

EXAMPLES

slide-36
SLIDE 36

JUMPING TEAPOTS!

slide-37
SLIDE 37
slide-38
SLIDE 38

I HEARD YOU LIKE WEBCAM STREAMS...

slide-39
SLIDE 39
slide-40
SLIDE 40

CONCLUSION

slide-41
SLIDE 41

CONTRIBUTION

Support for Image Processing and AR

For Declarative 3D for the Web

Minimal extension to existing standard

0 new nodes For AR: 2 Xflow operators (+3 otsu) Several new connections for Xflow (reusable)

Generic Design is awesome!

>

slide-42
SLIDE 42

FUTURE WORK

Further decompose xflar.detect into smaller operators

Option to replace individual parts of the detection algorithm

Implement Post-Processing

A better way to combine video with rendered content

Further optimize Xflow

Merge image-processing operators Execution with parallel JavaScript

slide-43
SLIDE 43

THANKS TO OUR SUPPORTERS

Intel Visual Computing Institute Verve EU Project FI-CONTENT EU Project

Part of Future Internet PPP program

slide-44
SLIDE 44

THANKS!

QUESTIONS?