Super Resolution Video Group - - PowerPoint PPT Presentation

▶
super resolution
SMART_READER_LITE
LIVE PREVIEW

Super Resolution Video Group - - PowerPoint PPT Presentation

Mar 30, 2023 697 likes •1.23k views

Super Resolution Video Group CS MSU Graphics & Media Lab Only for Maxus Example-based SR SRME HMRF SR 2 CS MSU

slide-1
SLIDE 1

Super Resolution для сжатого видео

Моисейцев Алексей Video Group CS MSU Graphics & Media Lab

slide-2
SLIDE 2

CS MSU Graphics & Media Lab (Video Group) Only for Maxus  2

Содержание

 Введение  Example-based SR  SRME  HMRF SR

slide-3
SLIDE 3

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Введение

3

slide-4
SLIDE 4

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Введение

4

source Bi-cubic Super-res Lossless 2Mbps 1Mbps 256Kbps

Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 2004

slide-5
SLIDE 5

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Введение

Алгоритмы SR:

 Iterative Backprojection (IBP)  Projection Onto Convex Sets (POCS)  Probabilistic Methods

 Maximum a posteriori (MAP)

 Model-based approach (MBSR)

 Example-based

6

slide-6
SLIDE 6

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Введение

IBP

7

C2 C C1 C3

g P1g P3g P2g f=Pg

POCS

slide-7
SLIDE 7

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Введение

8

Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience and Remote Sensing Symposium, 2005

slide-8
SLIDE 8

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Введение

Типичная скорость работы: 0.15-0.4 fps

9

slide-9
SLIDE 9

CS MSU Graphics & Media Lab (Video Group) Only for Maxus  10 10 10

Содержание

 Введение  Example-based SR  SRME  HMRF SR

slide-10
SLIDE 10

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Example-based SR

 В SR только по LR-кадрам

существует теоретический предел качества

 Иногда есть доступ и к

отдельным кадрам в высоком разрешении

11

Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006

slide-11
SLIDE 11

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Example-based SR

Registration Для компенсации движения используется алгоритм Lucas-Kanade:

12

Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006

slide-12
SLIDE 12

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Example-based SR

Registration

13

В 99% случаев ошибка не превышает 0.15 LR-пикселя и 0.45 HR-пикселя

Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006

slide-13
SLIDE 13

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Example-based SR

Построение пар HR-LR

14

 Используются LR-блоки

8x8

 Рассматриваются

повороты блоков

 Ключ —

10 AC-коэффициентов LR- блока и 68 граничных пикселей HR-блока

Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006

slide-14
SLIDE 14

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Example-based SR

Восстановление

15

Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006

slide-15
SLIDE 15

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Example-based SR

Восстановление

16

Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006

slide-16
SLIDE 16

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Example-based SR

Результат

MJPEG, Q=50

Обучение на первом кадре

Восстановление двадцатого кадра

17

source, frame 0 compressed frame 20 SR, frame 20

Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006

slide-17
SLIDE 17

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Example-based SR

Результат

18

compressed frame 20 SR, frame 20 Spatial example-based SR, frame 20

Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006

slide-18
SLIDE 18

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Key-frame based SR

19

Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008

Идея: использовать HR ключевые кадры

slide-19
SLIDE 19

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Key-frame based SR

Восстановление

20

Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008

slide-20
SLIDE 20

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Key-frame based SR

Результаты

21

Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008

slide-21
SLIDE 21

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Key-frame based SR

Результаты

22

Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008

slide-22
SLIDE 22

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Key-frame based SR

Результаты

23

Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008

slide-23
SLIDE 23

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Example-based SR

 Слабая обоснованность метода  Требуется наличие специфичного

видеопотока

 Зависимость от обучающей выборки

24

slide-24
SLIDE 24

CS MSU Graphics & Media Lab (Video Group) Only for Maxus  25 25 25

Содержание

 Введение  Example-based SR  SRME  HMRF SR

slide-25
SLIDE 25

CS MSU Graphics & Media Lab (Video Group) Only for Maxus  26

SRME

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

slide-26
SLIDE 26

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

27

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

slide-27
SLIDE 27

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

28

— интересующее решение

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

— апостериорная вероятность

slide-28
SLIDE 28

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

Задание вероятностей (1)

29

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

slide-29
SLIDE 29

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

Задание вероятностей (2)

30

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

slide-30
SLIDE 30

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

Задание вероятностей (3)

31

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

slide-31
SLIDE 31

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

Решение (1)

32

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

slide-32
SLIDE 32

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

Решение (2)

Две итерации:

 Компенсация движения  Построение улучшенного кадра 33

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

slide-33
SLIDE 33

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

34

Y-PSNR = 20.44dB

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

source

slide-34
SLIDE 34

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

35

Y-PSNR = 21.98dB

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

bilinear

slide-35
SLIDE 35

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

SRME

36

Y-PSNR = 25.64dB

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

proposed SR

slide-36
SLIDE 36

CS MSU Graphics & Media Lab (Video Group) Only for Maxus  37 37 37

Содержание

 Введение  Example-based SR  SRME  HMRF SR

slide-37
SLIDE 37

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

38

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

Standard decompression Smoothed image Sharpened image

slide-38
SLIDE 38

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

39

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-39
SLIDE 39

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Шум квантования (1)

40

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-40
SLIDE 40

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Шум квантования (2)

41

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-41
SLIDE 41

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Шум компенсации (1)

42

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-42
SLIDE 42

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Шум компенсации (2)

43

— общее выражение — независимость шума

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-43
SLIDE 43

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Условие гладкости

44

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-44
SLIDE 44

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

45

— гладкость — квантование — компенсация

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-45
SLIDE 45

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Решение

46

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

Градиент:

slide-46
SLIDE 46

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Решение

Упрощение: Градиентный спуск:

47

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-47
SLIDE 47

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Результаты

48

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-48
SLIDE 48

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Результаты

49

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-49
SLIDE 49

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

HMRF SR

Результаты

50

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

slide-50
SLIDE 50

CS MSU Graphics & Media Lab (Video Group) Only for Maxus 

Заключение

 Рассмотрены Example-based и MAP методы  Многие методы используют информацию из

потока или требуют дополнительные данные для работы

 Прямые реализации алгоритмов сложны, но

тем не менее многие из них возможно распараллелить (например, с использованием CUDA)

51

slide-51
SLIDE 51

CS MSU Graphics & Media Lab (Video Group) Only for Maxus  52

Литература

Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000

Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001

DCT Quantization Noise in Compressed Images, Mark A. Robertson and Robert L. Stevenson, 2004

Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 2004

  • Z. Lin and H-Y. Shum. Fundamental limits of reconstruction-based

superresolution algorithms under local translation. PAMI, 2004

Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006

T.Q. Pham, M. Bezuijen, L.J. van Vliet, K. Schutte, and C.L. Luengo Hendriks. Performance of optimal registration estimators, SPIE 2005

Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience and Remote Sensing Symposium, 2005

Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008

slide-52
SLIDE 52

CS MSU Graphics & Media Lab (Video Group) Only for Maxus  53 53

Вопросы

?

53