Automatic 3D Mapping for Infrared Image Analysis i i r r f f m - - PowerPoint PPT Presentation

▶
automatic 3d mapping for infrared image analysis
SMART_READER_LITE
LIVE PREVIEW

Automatic 3D Mapping for Infrared Image Analysis i i r r f f m - - PowerPoint PPT Presentation

Feb 15, 2023 155 likes •493 views

Automatic 3D Mapping for Infrared Image Analysis i i r r f f m m c c a a d d a a r r a a c c h h e e V. Martin, V. Gervaise, V. Moncada, M.H. Aumeunier, M. Firdaouss, J.M. Travere (CEA) S. Devaux (IPP), G. Arnoux (CCFE)

slide-1
SLIDE 1
  • V. Martin et al. 1 (19) WFDPVA, ENEA Frascati 28/03/12

Automatic 3D Mapping for Infrared Image Analysis

  • V. Martin, V. Gervaise, V. Moncada, M.H. Aumeunier, M. Firdaouss, J.M. Travere (CEA)
  • S. Devaux (IPP), G. Arnoux (CCFE) and JET-EFDA contributors

Workshop on Fusion Data Processing Validation and Analysis, ENEA Frascati, 26-28 March 2012

i

a c a

r m f

c a r e d h

i

a c a

r m f

c a r e d h

slide-2
SLIDE 2
  • V. Martin et al. 2 (19) WFDPVA, ENEA Frascati 28/03/12

3D IR Scene Calibration

JET #81313 KL7

(images in DL) Bulk W Bulk Be Be coated linconel W coated CFC Bulk Be Bulk Be W coated CFC

slide-3
SLIDE 3
  • V. Martin et al. 3 (19) WFDPVA, ENEA Frascati 28/03/12
  • Issue: a complex thermal scene
  • 1. Wide angle views with high geometrical effects:

depth of field and curvature

  • 2. Many metallic materials (Be, W) with different

and changing optical (reflectance) and thermal (emissivity) properties

3D IR Scene Calibration

JET #81313 KL7

(images in DL) Bulk W Bulk Be Be coated linconel W coated CFC Bulk Be Bulk Be W coated CFC

slide-4
SLIDE 4
  • V. Martin et al. 4 (19) WFDPVA, ENEA Frascati 28/03/12
  • Issue: a complex thermal scene
  • 1. Wide angle views with high geometrical effects:

depth of field and curvature

  • 2. Many metallic materials (Be, W) with different

and changing optical (reflectance) and thermal (emissivity) properties

  • Objective: Match each pixel with the 3D

scene model of in-vessel components for:

  • 1. getting the real geometry of the viewed objects
  • 2. reliable linking between viewed objects and their

related properties

3D IR Scene Calibration

JET #81313 KL7

(images in DL) Bulk W Bulk Be Be coated linconel W coated CFC Bulk Be Bulk Be W coated CFC

slide-5
SLIDE 5
  • V. Martin et al. 5 (19) WFDPVA, ENEA Frascati 28/03/12
  • Issue: a complex thermal scene
  • 1. Wide angle views with high geometrical effects:

depth of field and curvature

  • 2. Many metallic materials (Be, W) with different

and changing optical (reflectance) and thermal (emissivity) properties

  • Objective: Match each pixel with the 3D

scene model of in-vessel components for:

  • 1. getting the real geometry of the viewed objects
  • 2. reliable linking between viewed objects and their

related properties

  • Applications
  • 1. Image processing (event characterization)
  • 2. IR data calibration: T

surf = f(material emissivity)

3D IR Scene Calibration

JET #81313 KL7

(images in DL) Bulk W Bulk Be Be coated linconel W coated CFC Bulk Be Bulk Be W coated CFC

slide-6
SLIDE 6
  • V. Martin et al. 6 (19) WFDPVA, ENEA Frascati 28/03/12

Methodology

Image Stabilization 2D/3D Scene Model Mapping Image Processing Image Correction

Camera NUC Dead pixel Map Reference image 2D/3D scene models Knowledge base

  • f the thermal

scene

  • Calibration chain

Registered & Calibrated Image

slide-7
SLIDE 7
  • V. Martin et al. 7 (19) WFDPVA, ENEA Frascati 28/03/12

Illustration of Motion in Images

  • Camera vibrations lead to misalignments of ROIs (PFC RT

protection) = false alarms or worth missed alarms

  • Image stabilization is a mandatory step for heat flux deposit

analysis based on Tsurf(t)-Tsurf(t-1) estimations

slide-8
SLIDE 8
  • V. Martin et al. 8 (19) WFDPVA, ENEA Frascati 28/03/12
  • Important factors for method selection
  • Deformation type: planar (homothety), non-planar
  • Target application: real-time processing, off-line analysis
  • Data quality and variability: noise level, pixel intensity changes, image entropy
  • Required precision level: pixel, sub-pixel
  • Applications in tokamaks (non-exhaustive list)

Image Stabilization

Motion amplitude Target application Precision required Difficulty JET KL7 wide-angle 5-10 pixels (camera vibrations) Hot spot detection PFC protection pixel low image entropy JET KL7 windowed up to 15 pixels (disruptions) Physics analysis (e.g. heat load during disruptions…) pixel pixel intensity changes JET KL9 divertor tiles <1 pixel (sensor affected by magnetic fields) Physics analysis (power deposit influx) sub-pixel low resolution, slow motion, aliasing

slide-9
SLIDE 9
  • V. Martin et al. 9 (19) WFDPVA, ENEA Frascati 28/03/12

Image Stabilization

See Zitova’s survey, Image and Vision Computing, vol. 21(2003), pp. 977-1000

slide-10
SLIDE 10
  • V. Martin et al. 10 (19) WFDPVA, ENEA Frascati 28/03/12
  • Classical Methodology

Image Stabilization

See Zitova’s survey, Image and Vision Computing, vol. 21(2003), pp. 977-1000

slide-11
SLIDE 11
  • V. Martin et al. 11 (19) WFDPVA, ENEA Frascati 28/03/12
  • Classical Methodology
  • 1. Feature Detection
  • Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu

talk), SIFT, SURF, FAST…

  • Global descriptors: Tsallis entropy (see Murari talk), edge detectors…
  • Fourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping…

Image Stabilization

See Zitova’s survey, Image and Vision Computing, vol. 21(2003), pp. 977-1000

slide-12
SLIDE 12
  • V. Martin et al. 12 (19) WFDPVA, ENEA Frascati 28/03/12
  • Classical Methodology
  • 1. Feature Detection
  • Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu

talk), SIFT, SURF, FAST…

  • Global descriptors: Tsallis entropy (see Murari talk), edge detectors…
  • Fourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping…
  • 2. Feature Matching
  • Spatial cross-correlation techniques: normalized cross-correlation, Hausdorff distance…
  • Fourier domain: normalized cross-spectrum and its extensions

Image Stabilization

See Zitova’s survey, Image and Vision Computing, vol. 21(2003), pp. 977-1000

slide-13
SLIDE 13
  • V. Martin et al. 13 (19) WFDPVA, ENEA Frascati 28/03/12
  • Classical Methodology
  • 1. Feature Detection
  • Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu

talk), SIFT, SURF, FAST…

  • Global descriptors: Tsallis entropy (see Murari talk), edge detectors…
  • Fourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping…
  • 2. Feature Matching
  • Spatial cross-correlation techniques: normalized cross-correlation, Hausdorff distance…
  • Fourier domain: normalized cross-spectrum and its extensions
  • 3. Transform Model Estimation
  • Shape preserving mapping (rotation, translation and scaling only)
  • Elastic mapping: warping techniques…

Image Stabilization

See Zitova’s survey, Image and Vision Computing, vol. 21(2003), pp. 977-1000

slide-14
SLIDE 14
  • V. Martin et al. 14 (19) WFDPVA, ENEA Frascati 28/03/12
  • Classical Methodology
  • 1. Feature Detection
  • Local descriptors: Harris corners, MSER, codebooks, Gabor wavelets (see Craciunescu

talk), SIFT, SURF, FAST…

  • Global descriptors: Tsallis entropy (see Murari talk), edge detectors…
  • Fourier analysis: spectral magnitude & phase, pixel gradients, log-polar mapping…
  • 2. Feature Matching
  • Spatial cross-correlation techniques: normalized cross-correlation, Hausdorff distance…
  • Fourier domain: normalized cross-spectrum and its extensions
  • 3. Transform Model Estimation
  • Shape preserving mapping (rotation, translation and scaling only)
  • Elastic mapping: warping techniques…
  • 4. Image transformation
  • 2D Interpolation: nearest neighboor, bilinear, bicubic…

Image Stabilization

See Zitova’s survey, Image and Vision Computing, vol. 21(2003), pp. 977-1000

slide-15
SLIDE 15
  • V. Martin et al. 15 (19) WFDPVA, ENEA Frascati 28/03/12

Proposed Algorithm

  • 1. Masked FFT-based image registration [1]

 Deterministic computing time  Accelerating hardware compatible algorithm (e.g. FFT on GPU) → real time applications  Local analysis with dynamic intensity-based pixel masking (e.g. mask the divertor bright region)

  • 2. with sub-pixel precision [2]

 Slow drift compensation

  • 3. and dynamic update of the reference image

 Robust to image intensity changes (context awareness)  Evaluation of registration quality over time [1] D. Padfield, IEEE CVPR’10, pp. 2918-2925, 2010 [2] M. Guizar-Sicairos et al., Opt. Lett., vol. 33, no. 2, pp. 156-158, 2008

slide-16
SLIDE 16
  • V. Martin et al. 16 (19) WFDPVA, ENEA Frascati 28/03/12

Principle of Fourier-based Correlation

slide-17
SLIDE 17
  • V. Martin et al. 17 (19) WFDPVA, ENEA Frascati 28/03/12

Principle of Fourier-based Correlation

  • Let Iref a reference image, It an image at time t and DFT the Discrete 2D Fourier transform

such as It ( x , y ) = Iref ( x-x0 , y-y0 ) ) ( (.,.) (.,.) (.,.) (.,.) (.,.) ) ( ) (

1

  • 2

1 2 1 2 1 NCC NCC t ref

F DFT NCC F F F F F I DFT F I DFT F Iref It

slide-18
SLIDE 18
  • V. Martin et al. 18 (19) WFDPVA, ENEA Frascati 28/03/12

Principle of Fourier-based Correlation

  • Let Iref a reference image, It an image at time t and DFT the Discrete 2D Fourier transform

such as It ( x , y ) = Iref ( x-x0 , y-y0 )

  • NCC is the Normalized Cross Correlation figure (image)

and the position of the peak gives the coordinates of the translation ( x0 , y0 ) ) ( (.,.) (.,.) (.,.) (.,.) (.,.) ) ( ) (

1

  • 2

1 2 1 2 1 NCC NCC t ref

F DFT NCC F F F F F I DFT F I DFT F Iref It

NCC y x

y x,

max arg ,

max (NCC(Iref, It)) NCC(Iref, It)

slide-19
SLIDE 19
  • V. Martin et al. 19 (19) WFDPVA, ENEA Frascati 28/03/12

Sub-pixel Precision

) ( aliasing)

  • (anti

) s frequencie high ( ) s frequencie low ( ) (

  • therwise

integer an is if ) , ( ) , (

1

  • .

UP NCC UP UP NCC UP NCC UP NCC UP UP NCC k k v k u UP

F DFT NCC F F F NCC DFT F NCC v u NCC

  • Up-sample k times the DFT of NCC (trigonometric interpolation):
slide-20
SLIDE 20
  • V. Martin et al. 20 (19) WFDPVA, ENEA Frascati 28/03/12

Sub-pixel Precision

) ( aliasing)

  • (anti

) s frequencie high ( ) s frequencie low ( ) (

  • therwise

integer an is if ) , ( ) , (

1

  • .

UP NCC UP UP NCC UP NCC UP NCC UP UP NCC k k v k u UP

F DFT NCC F F F NCC DFT F NCC v u NCC

  • Up-sample k times the DFT of NCC (trigonometric interpolation):
  • The peak coordinates ( x0 , y0 ) give F the translation with 1/k pixel of precision:

UP y x

NCC k y x

,

max arg 1 ,

slide-21
SLIDE 21
  • V. Martin et al. 21 (19) WFDPVA, ENEA Frascati 28/03/12

Reference Image Updating

  • Goal: maintaining a good reliability of the motion estimator (NCC

peak value) while image appearance changes during the pulse.

slide-22
SLIDE 22
  • V. Martin et al. 22 (19) WFDPVA, ENEA Frascati 28/03/12

Reference Image Updating

  • Solution: use the NCC peak value

to trigger the update of Iref such as:

t ref

I I then T t NCC T if

max min

)) ( max(

NCC peak too low, no Iref update

update Iref update Iref update Iref update Iref

slide-23
SLIDE 23
  • V. Martin et al. 23 (19) WFDPVA, ENEA Frascati 28/03/12

Results

  • JET #81313 (MARFE, disruption), KL7, 480x512 pixels, 50 Hz,

251 frames

k=1/4 pixel

slide-24
SLIDE 24
  • V. Martin et al. 24 (19) WFDPVA, ENEA Frascati 28/03/12

Results

k=1/2 pixel

  • JET #80827 (disruption), KL7, 128x256 pixels, 540 Hz,

13425 frames

slide-25
SLIDE 25
  • V. Martin et al. 25 (19) WFDPVA, ENEA Frascati 28/03/12

Results

  • JET #82278, KL9B (slow drift), 32x96 pixels, 6 kHz, 4828 frames

32 pixels 96 pixels

10 , 25

unstab stab

T T

slide-26
SLIDE 26
  • V. Martin et al. 26 (19) WFDPVA, ENEA Frascati 28/03/12

Computational Performance

  • High frame rate performance using GPU

256x256, k=1/4

→ 700 fps

slide-27
SLIDE 27
  • V. Martin et al. 27 (19) WFDPVA, ENEA Frascati 28/03/12

From 2D to 3D

  • Challenge

– transform pixel coordinates into machine coordinates: (x, y) (r, θ, φ)

  • Method

– Ray-tracing method from 3D/simplified CAD files

slide-28
SLIDE 28
  • V. Martin et al. 28 (19) WFDPVA, ENEA Frascati 28/03/12

3D Scene Model for Image Processing

  • S. Palazzo, A. Murari et al., RSI 81, 083505, 2010
  • V. Martin et al.

Blobs 1 & 2 must not be merged!

1 2 1 2

mm

Z Map (depth)

2 1

2m 7m

slide-29
SLIDE 29
  • V. Martin et al. 29 (19) WFDPVA, ENEA Frascati 28/03/12

Integrated Framework

  • An integrated software for IR data stabilization & analysis

Image Stabilization 2D/3D Scene Model Mapping Image Processing Image Correction

Camera NUC Dead pixel Map Reference image 2D/3D scene models Knowledge base

  • f the thermal

scene Registered & Calibrated Image

slide-30
SLIDE 30
  • V. Martin et al. 30 (19) WFDPVA, ENEA Frascati 28/03/12

Plasma ImagiNg data Understanding Platform (PINUP)

Integrated Framework

  • An integrated software for IR data stabilization & analysis

Image Stabilization 2D/3D Scene Model Mapping Image Processing Image Correction

Camera NUC Dead pixel Map Reference image 2D/3D scene models Knowledge base

  • f the thermal

scene Registered & Calibrated Image

slide-31
SLIDE 31
  • V. Martin et al. 31 (19) WFDPVA, ENEA Frascati 28/03/12

Integrated Framework

  • An integrated software for IR data stabilization & analysis

Set sub-pixel precision factor Set mask Load/save translations

slide-32
SLIDE 32
  • V. Martin et al. 32 (19) WFDPVA, ENEA Frascati 28/03/12

Integrated Framework

  • An integrated software for IR data stabilization & analysis

Used for PFC protection Used for temperature evaluation Used for event triggering

slide-33
SLIDE 33
  • V. Martin et al. 33 (19) WFDPVA, ENEA Frascati 28/03/12

Conclusion

  • Summary

– Complex IR scenes require a new approach for reliable data analysis including image stabilization and 3D mapping. – A robust and fast image stabilization algorithm with sub-pixel precision has been proposed. – A first demonstration of 3D model for IR data analysis has been successfully carried out at JET on the wide-angle ITER-like viewing system (KL7). – An integrated software (PINUP) implementing these features is available for users upon request.

  • Outlook

– Test of the stabilization algorithm on visible imaging data (JET KL8) with rotation compensation – Full integration of 3D scene models into PINUP – Improvement of image processing algorithms (e.g. hot spot detection) with 3D information