Movies as Programs Leif Andersen Accessibility (prominent code) - - PowerPoint PPT Presentation

▶
movies as programs
SMART_READER_LITE
LIVE PREVIEW

Movies as Programs Leif Andersen Accessibility (prominent code) - - PowerPoint PPT Presentation

Dec 01, 2022 474 likes •1.96k views

Movies as Programs Leif Andersen Accessibility (prominent code) (some code) One down One down 19 more to go We Need Automation e Landscape Tool Example Experience Plugin-Ins Blender Script, AE Script UI Automation Apple Script

slide-1
SLIDE 1

Movies as Programs

Leif Andersen

slide-2
SLIDE 2

Accessibility

(prominent code)

(some code)

slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11

One down

slide-12
SLIDE 12

One down 19 more to go…

slide-13
SLIDE 13
slide-14
SLIDE 14
slide-15
SLIDE 15

We Need Automation

slide-16
SLIDE 16

e Landscape

Tool Example Experience

Plugin-Ins

Blender Script, AE Script

UI Automation

(Macros) Apple Script

Shell Scripts

FFmpeg, AVISynth

slide-17
SLIDE 17

e Landscape

Tool Example Experience

Plugin-Ins

Blender Script, AE Script

UI Automation

(Macros) Apple Script

Shell Scripts

FFmpeg, AVISynth

slide-18
SLIDE 18

e Landscape

Tool Example Experience

Plugin-Ins

Blender Script, AE Script

UI Automation

(Macros) Apple Script

Shell Scripts

FFmpeg, AVISynth

slide-19
SLIDE 19

e Landscape

Tool Example Experience

Plugin-Ins

Blender Script, AE Script

UI Automation

(Macros) Apple Script

Shell Scripts

FFmpeg, AVISynth

slide-20
SLIDE 20

We have a problem…

slide-21
SLIDE 21

We have a problem… We want to solve it in the problem domain's own language…

slide-22
SLIDE 22

We have a problem… We want to solve it in the problem domain's own language…

DSLs are the

"Ultimate Abstraction"

Paul Hudak

slide-23
SLIDE 23

Make a DSL!

slide-24
SLIDE 24

Library

slide-25
SLIDE 25

Language

Library

slide-26
SLIDE 26

Library

slide-27
SLIDE 27

Producers Filters Playlists Multitracks

slide-28
SLIDE 28

Producers

slide-29
SLIDE 29

Producers

slide-30
SLIDE 30

Producers

render : Producer →

slide-31
SLIDE 31

Producers

render : Producer → clip : String → Producer

slide-32
SLIDE 32

Producers

render : Producer → clip : String → Producer (render (clip "demo.mp4")) ⇒

slide-33
SLIDE 33

Filters

slide-34
SLIDE 34

Producer Producer

Filter

slide-35
SLIDE 35

(attach-filter bunny-clip (sepia-filter))

slide-36
SLIDE 36

(attach-filter bunny-clip (sepia-filter))

slide-37
SLIDE 37

Playlists

slide-38
SLIDE 38

Producer Producer Producer Producer

Time

slide-39
SLIDE 39

(playlist (clip "jumping.mp4") (clip "flying.mp4"))

slide-40
SLIDE 40

Producer Producer Producer Producer

Time

slide-41
SLIDE 41

Producer Producer Producer Producer Transition

Time

slide-42
SLIDE 42

(playlist (clip "jumping.mp4") (fade-transition 1) (clip "flying.mp4"))

slide-43
SLIDE 43

Multitracks

slide-44
SLIDE 44

Producer Producer Producer Producer

Time Layers

slide-45
SLIDE 45

Producer Producer Producer Producer

Merge

Time Layers

slide-46
SLIDE 46

(define WIDTH 1920) (define HEIGHT 1080) (multitrack (color "black") (overlay-merge 0 0 (/ WIDTH 2) HEIGHT) (clip "running.mp4") (overlay-merge (/ WIDTH 2) 0 (/ WIDTH 2) HEIGHT) (clip "flying.mp4"))

slide-47
SLIDE 47

Library

Producers Filters Playlists Multitracks

slide-48
SLIDE 48

Language

Library

Producers Filters Playlists Multitracks

slide-49
SLIDE 49
slide-50
SLIDE 50

Primitives

slide-51
SLIDE 51

List Comprehensions

slide-52
SLIDE 52

Modules

slide-53
SLIDE 53
slide-54
SLIDE 54

Functions

slide-55
SLIDE 55
slide-56
SLIDE 56

#lang video ;; Create a mosaic of four videos (for/vertical ([i (in-range 2)]) (for/horizontal ([j (in-range 2)]) (external-video "branded.vid" (clip "logo.png") (clip (format "~aX~a.mp4" i j)))))

slide-57
SLIDE 57

#lang video (clip "dragon.mp4") ;; Create a mosaic of four videos (for/vertical ([i (in-range 2)]) (for/horizontal ([j (in-range 2)]) (external-video "branded.vid" (clip "logo.png") (clip (format "~aX~a.mp4" i j)))))

slide-58
SLIDE 58

Implementing Video + Editing Manual Editing

slide-59
SLIDE 59

From Libraries to

Languages

slide-60
SLIDE 60

We make DSLs using

Linguistic Inheritance

slide-61
SLIDE 61

We make DSLs using

Linguistic Inheritance

slide-62
SLIDE 62

We make DSLs using

Linguistic Inheritance

Movie Script

Video Implementation

Racket

slide-63
SLIDE 63

We make DSLs using

Linguistic Inheritance

Movie Script

Video Implementation

Racket

Re-export construct

slide-64
SLIDE 64

We make DSLs using

Linguistic Inheritance

Movie Script

Video Implementation

Racket

Re-export construct Remove construct

slide-65
SLIDE 65

We make DSLs using

Linguistic Inheritance

Movie Script

Video Implementation

Racket

Re-export construct Remove construct New construct

slide-66
SLIDE 66

We make DSLs using

Linguistic Inheritance

Movie Script

Video Implementation

Racket

Re-export construct Remove construct New construct Change construct

slide-67
SLIDE 67

(for/playlist ([scene (in-list scene-list)]) (multitrack scene (overlay-merge 10 10 300 300) (clip "logo.mp4")))

slide-68
SLIDE 68

(define (for/playlist seq body) (apply playlist (for/list ([i (in-list seq)]) (body i))))

slide-69
SLIDE 69

(define (for/playlist seq body) (apply playlist (for/list ([i (in-list seq)]) (body i)))) > (for/playlist (list (clip "a.mp4") (clip "b.mp4")) (λ (scene) (multitrack scene (overlay-merge 10 10 300 300) (clip "logo.mp4"))))

slide-70
SLIDE 70

(define-macro (for/playlist seq . body) `(apply playlist (for/list ,seq ,@body)))

slide-71
SLIDE 71

(for/playlist ([s (list (clip "a.mp4"))]) (multitrack ...))

⇒ elaborates

(apply playlist (for/list ([s (list (clip "a.mp4"))]) (multitrack ....)))

slide-72
SLIDE 72

(for/playlist ([s (list (clip "a.mp4"))]) (multitrack ...))

⇒ elaborates

(apply playlist (for/list ([s (list (clip "a.mp4"))]) (multitrack ....)))

⇒ evaluates

#<playlist>

slide-73
SLIDE 73

(let ([playlist 42]) (for/playlist ....))

slide-74
SLIDE 74

(let ([playlist 42]) (for/playlist ....))

⇒ elaborates

(let ([playlist 42]) (apply playlist ....))

slide-75
SLIDE 75

(let ([playlist 42]) (for/playlist ....))

⇒ elaborates

(let ([playlist 42]) (apply playlist ....))

⇒ evaluates

slide-76
SLIDE 76

(define-macro (for/playlist seq . body) `(apply playlist (for/list ,seq ,@body))) > (let ([playlist 42]) (for/playlist ([s (list (clip "a.mp4"))]) (multitrack s (overlay-merge 10 10 300 300) (clip "logo.mp4"))))

slide-77
SLIDE 77

(define-syntax-rule (for/playlist seq body ...) (apply playlist (for/list seq body ...)))

slide-78
SLIDE 78

(define-syntax-rule (for/playlist seq body ...) (apply playlist (for/list seq body ...))) > (let ([playlist 42]) (for/playlist ([s (list (clip "a.mp4"))]) (multitrack s (overlay-merge 10 10 300 300) (clip "logo.mp4"))))

slide-79
SLIDE 79
slide-80
SLIDE 80
slide-81
SLIDE 81
slide-82
SLIDE 82

Non-Local Language Features

slide-83
SLIDE 83

#lang video (define talk ...) (define logo ...) talk logo logo

slide-84
SLIDE 84

#lang video (define talk ...) (define logo ...) (provide vid) talk logo logo (define vid (playlist ))

slide-85
SLIDE 85

Interposition Points

#%app #%module-begin

slide-86
SLIDE 86

(+ 1 2)

⇒elaborates

(#%app + 1 2)

slide-87
SLIDE 87

#lang video logo talk ;; Where (define logo ...) (define talk ...)

(module anon video ( #%module-begin logo talk (define logo ...) (define talk ...)))

parses

slide-88
SLIDE 88

(module anon video ( #%module-begin logo talk (define logo ...) (define talk ...))) (module anon racket ( #%module-begin (require vidlib) (define logo ...) (define talk ...) (vid-begin vid logo talk)))

elaborates

slide-89
SLIDE 89

#lang racket

slide-90
SLIDE 90

(require syntax/wrapping-modbeg) (define-syntax video-module-begin (make-wrapping-module-begin ...))

slide-91
SLIDE 91

(require syntax/wrapping-modbeg) (define-syntax video-module-begin (make-wrapping-module-begin ...))

slide-92
SLIDE 92

#lang racket/base ... run time code ... (define-syntax macro-name ... compile time code ...) ... run time code ...))

slide-93
SLIDE 93

(define-syntax id expr)

id : run time binding expr : compile time expression

slide-94
SLIDE 94

Movies as Programs: A Tower of Languages

slide-95
SLIDE 95
slide-96
SLIDE 96
slide-97
SLIDE 97

FFI

slide-98
SLIDE 98

Source File Video Data Structure FFmpeg Filter Graph

101010 010101

Runtime Output

slide-99
SLIDE 99

We have a problem…

FFI

V

slide-100
SLIDE 100

We have a problem… We want to solve it in the problem domain's own language…

FFI

V

slide-101
SLIDE 101

We have a problem… We want to solve it in the problem domain's own language…

Make a DSL!

FFI

V

slide-102
SLIDE 102

An FFI DSL

int av_frame_get_buffer(AVFrame *frame, int align);

(Scheme Workshop, 2004)

slide-103
SLIDE 103

An FFI DSL

int av_frame_get_buffer(AVFrame *frame, int align);

(define-ffmpeg av-frame-get-buffer (_fun [frame : _av-frame] [align : _int]

  • > [ret : _int]
  • > (maybe-error? ret)))

(Scheme Workshop, 2004)

slide-104
SLIDE 104

An Object DSL

(define-ffmpeg av-frame-alloc ...) (define-ffmpeg av-frame-free ...) (define-constructor clip video ... av-frame-alloc ... av-frame-free ...)

slide-105
SLIDE 105

OpenGL FFmpeg

slide-106
SLIDE 106

Documentation

slide-107
SLIDE 107

We have a problem…

Documentation

V

slide-108
SLIDE 108

We have a problem… We want to solve it in the problem domain's own language…

Documentation

V

slide-109
SLIDE 109

We have a problem… We want to solve it in the problem domain's own language…

Make a DSL!

Documentation

V

slide-110
SLIDE 110

A Documentation DSL

(ICFP, 2009)

slide-111
SLIDE 111

A Documentation DSL

#lang video/documentation @title{Video: The Language} @(defmodulelang video) Video Language (or VidLang, sometimes referred to as just Video) is a DSL for editing...videos. It aims to merge the capabilities of a traditional

(ICFP, 2009)

slide-112
SLIDE 112
slide-113
SLIDE 113

OpenGL FFmpeg

slide-114
SLIDE 114

Types

slide-115
SLIDE 115

(clip "clip.mp4" #:start 0 #:end 50)

slide-116
SLIDE 116

(cut-producer #:start 0 #:end 100) (clip "clip.mp4" #:start 0 #:end 50)

slide-117
SLIDE 117

(cut-producer #:start 0 #:end 100) (clip "clip.mp4" #:start 0 #:end 50)

slide-118
SLIDE 118

A Typed DSL

slide-119
SLIDE 119

We have a problem…

Types

V

slide-120
SLIDE 120

We have a problem… We want to solve it in the problem domain's own language…

Types

V

slide-121
SLIDE 121

We have a problem… We want to solve it in the problem domain's own language…

Make a DSL!

Types

V

slide-122
SLIDE 122

A Typed DSL

(POPL, 2017)

slide-123
SLIDE 123

A Type Implementation DSL

(define-typed-syntax (clip f) ≫ [⊢ f ≫ _ ⇐ File] #:where n (length f)

  • [⊢ (untyped:clip f) ⇒ (Producer n)])

(POPL, 2017)

slide-124
SLIDE 124

OpenGL FFmpeg

slide-125
SLIDE 125

We have a problem…

DSL

V

slide-126
SLIDE 126

We have a problem… We want to solve it in the problem domain's own language…

DSL

V

slide-127
SLIDE 127

We have a problem… We want to solve it in the problem domain's own language…

syntax-parse

A DSL for making DSLs

DSL

V

(ICFP, 2010)

slide-128
SLIDE 128

(define-syntax-rule (define/playlist (name args ...) body ...) (define name (λ (args ...) (playlist body ...))))

slide-129
SLIDE 129

(define-syntax-rule (define/playlist (name args ...) body ...) (define name (λ (args ...) (playlist body ...)))) > (define/playlist (double A) A A)

slide-130
SLIDE 130

(define-syntax-rule (define/playlist (name args ...) body ...) (define name (λ (args ...) (playlist body ...)))) > (define/playlist (double (A B C)) A)

slide-131
SLIDE 131

(define-simple-macro (define/playlist header:function-header body ...) (define header.name (λ header.args (playlist body ...))))

slide-132
SLIDE 132

(define-simple-macro (define/playlist header:function-header body ...) (define header.name (λ header.args (playlist body ...)))) > (define/playlist (double A) A A)

slide-133
SLIDE 133

(define-simple-macro (define/playlist header:function-header body ...) (define header.name (λ header.args (playlist body ...)))) > (define/playlist (double (A B C)) A)

slide-134
SLIDE 134

OpenGL FFmpeg

slide-135
SLIDE 135
slide-136
SLIDE 136
slide-137
SLIDE 137

Editor-Oriented

Programming

slide-138
SLIDE 138

e Future…

slide-139
SLIDE 139

begin-for-syntax define-syntax

slide-140
SLIDE 140

begin-for-editor define-editor

slide-141
SLIDE 141

#lang editor (define-editor video-editor ...) ... (play )

slide-142
SLIDE 142

OpenGL FFmpeg

slide-143
SLIDE 143
slide-144
SLIDE 144

https://lang.video

slide-145
SLIDE 145

https://lang.video

slide-146
SLIDE 146
slide-147
SLIDE 147
slide-148
SLIDE 148

anks For Watching

http://lang.video @videolang

We make DSLs using

Linguistic Inheritance

OpenGL FFmpeg