Progress on BF Kernel Correction at UC Davis Craig Lage August 14, - - PowerPoint PPT Presentation

Progress on BF Kernel Correction at UC Davis

Craig Lage August 14, 2018 Acknowledgements:Andrew Bradshaw

1 / 15

Sum of Covariances Overestimates Variance - ??

Sum from -1 to 1

Flux(ADU)

Variance(ADU^2) Variance Only Gain = 4.3968 Intercept = 28.3363 Noise = 23.4 e- C00 (@80K e-) = -0.062318 Red Crosses - Linear part of Variance Variance + Covariance: 1 Pixels Away Gain = 4.3141 Intercept = -6.3875 Quad Term = -2e-07

Photon Transfer Curve and 'Lost Variance' SEGMENT13

Sum from -2 to 2

Flux(ADU)

Variance(ADU^2) Variance Only Gain = 4.3968 Intercept = 28.3363 Noise = 23.4 e- C00 (@80K e-) = -0.062318 Red Crosses - Linear part of Variance Variance + Covariance: 2 Pixels Away Gain = 4.3089 Intercept = -5.8782 Quad Term = 1e-08

Photon Transfer Curve and 'Lost Variance' SEGMENT13

Sum from -4 to 4

Flux(ADU)

Variance(ADU^2) Variance Only Gain = 4.3968 Intercept = 28.3363 Noise = 23.4 e- C00 (@80K e-) = -0.062318 Red Crosses - Linear part of Variance Variance + Covariance: 4 Pixels Away Gain = 4.3717 Intercept = -18.2966 Quad Term = 3e-07

Photon Transfer Curve and 'Lost Variance' SEGMENT13

Sum from -8 to 8

Flux(ADU)

Variance(ADU^2) Variance Only Gain = 4.3968 Intercept = 28.3363 Noise = 23.4 e- C00 (@80K e-) = -0.062318 Red Crosses - Linear part of Variance Variance + Covariance: 8 Pixels Away Gain = 4.3900 Intercept = 2.7966 Quad Term = 5e-07

Photon Transfer Curve and 'Lost Variance' SEGMENT13

.

As we sum to larger and larger distances, the summed covariance

• verestimates the variance. The reason is not understood.

2 / 15

BF Kernel Correction - As Measured Covariances

20 10 20 Covariances(*1E7) 6 4 2 20 7.5 5.0 2.5 0.0 Covariances(*1E7) - X-Slice 20 5 Covariances(*1E7) - Y-Slice 20 10 20 Kernel(*1E7) 1.5 1.0 0.5 0.0 0.5 20 1 Kernel(*1E7) - X-Slice 20 1 Kernel(*1E7) - Y-Slice

B-F Kernel extracted from 3800 Flats

100 101 102 i 2 + j 2 10-5 10-4 10-3 10-2 10-1 Covariance

C00; Data1=-0.0623; Data2=-0.0623 C10; Data=0.0040; Data2=0.0040 C01; Data=0.0131; Data2=0.0131

Covariance vs Distance As_Measured As_Measured As_Measured_Neg As_Measured_Neg

Variance-1

50000 100000 Central Peak(electrons) 1.00 1.02 1.04 1.06 1.08 1.10 Sigma (Pixels) X Slope = 0.80 % per 50K e-, Intercept = 1.027 Y Slope = 0.90 % per 50K e-, Intercept = 1.029 Corrected X Slope = 0.48 % per 50K e-, Intercept = 1.027 Corrected Y Slope = 0.49 % per 50K e-, Intercept = 1.029

Brighter-Fatter - 30 micron Spots - SEGMENT13

Sigma-x Sigma-y Corrected Sigma-x Corrected Sigma-y

3 / 15

Reduce Variance to force Zero Sum

20 10 20 Covariances(*1E7) 10 8 6 4 2 20 10 5 Covariances(*1E7) - X-Slice 20 10 5 Covariances(*1E7) - Y-Slice 20 10 20 Kernel(*1E7) 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 20 4 2 Kernel(*1E7) - X-Slice 20 4 2 Kernel(*1E7) - Y-Slice

B-F Kernel extracted from 3800 Flats

100 101 102 i 2 + j 2 10-5 10-4 10-3 10-2 10-1 Covariance

C00; Data1=-0.0623; Data2=-0.0953 C10; Data=0.0040; Data2=0.0040 C01; Data=0.0131; Data2=0.0131

Covariance vs Distance As_Measured Zero_Sum As_Measured_Neg Zero_Sum_Neg

Variance-1

50000 100000 Central Peak(electrons) 1.00 1.02 1.04 1.06 1.08 1.10 Sigma (Pixels) X Slope = 0.80 % per 50K e-, Intercept = 1.027 Y Slope = 0.90 % per 50K e-, Intercept = 1.029 Corrected X Slope = 0.02 % per 50K e-, Intercept = 1.027 Corrected Y Slope = 0.02 % per 50K e-, Intercept = 1.029

Brighter-Fatter - 30 micron Spots - SEGMENT13

Sigma-x Sigma-y Corrected Sigma-x Corrected Sigma-y

4 / 15

Summary

These slides are intended to summarize the progress we have made at UC Davis in implementing the Coulton,et.al. for correction of the Brighter-Fatter Effect. The work is far from done, but I wanted to capture where we stand. In particular, there are the following issues: Slide 2 shows that when we add up the covariances at larger and larger distances, we begin to over-compensate the lost variance from the center. Why is this?? Using the measured correlations directly in the Coulton,et.al. algorithm significantly undercorrects for the BF effect. In order to fully correct, I need to increase the magnitude of the central lost covariance by a large amount - about 50%. At this point, I’m not sure if these issues are a measurement issue, a code bug, or a deficiency in the method. On the positive side, it is clear that the method can correct well with the right

• kernel. See Slide 4.

Next Steps: Get the algorithms used by the DM team on HSC images and compare the

• code. This may identify a code issue.

Andrew plans to try the “best” kernel on more complex images from the star/galaxy mask.

5 / 15

BF Kernel applied to Optical Simulator galaxy images

6 / 15

Back-up slides and

• ther attempts to improve correction

7 / 15

Measured Covariances - 3800 Flats Total

100 101 102 i2 + j2 10

5

10

4

10

3

10

2

10

1

100 Correlation Coefficient Based on 3800 total flats Slope = -1.124 C10 = 0.003958 C01 = 0.013147 C00 = -0.062278

Correlation Coefficient Matrix

Model Positive Negative

Variance-1

100 101 102 i 2 + j 2 10-5 10-4 10-3 10-2 10-1 100 δ Area / Area Slope = -1.047 C00; Data=-0.0623; Sim=-0.0708 C10; Data=0.0131; Sim=0.0057 C01; Data=0.0040; Sim=0.0122 χ2 = 1071.10

Covariance vs Distance: 80000 e- Sim Sim-Neg Data Data-Neg

Variance-1 .

Measured correlations agree reasonably well with simulated area distortions(right) - we can’t be too far off. Two “close in” negative correlations are C20 and C30 - these are believed to be impacted by serial deferred charge issue. C10 is probably also impacted and this will need to be compensated for.

8 / 15

We estimate covariance by keeping only the part that grows quadratically with flux.

5000 10000150002000025000300003500040000

Flux(ADU)

50 50 100 150 200 250 300 350 400

Correlations(ADU^2)

Intercept = -2.319531 Slope = 0.000763 Quad Term = 1.64157e-07 Cov_Value = 0.013133

Covariance of Pixel (0,1)

5000 10000150002000025000300003500040000

Flux(ADU)

50 50 100 150 200 250 300 350 400

Correlations(ADU^2)

Intercept = 1.956354 Slope = 0.000754 Quad Term = 4.34528e-08 Cov_Value = 0.003476

Covariance of Pixel (1,0)

5000 10000150002000025000300003500040000

Flux(ADU)

50 50 100 150 200 250 300 350 400

Correlations(ADU^2)

Intercept = -1.149540 Slope = 0.000319 Quad Term = 3.74959e-08 Cov_Value = 0.003000

Covariance of Pixel (1,1)

5000 10000150002000025000300003500040000

Flux(ADU)

50 50 100 150 200 250 300 350 400

Correlations(ADU^2)

Intercept = 3.310491 Slope = 0.000432 Quad Term = 7.7489e-10 Cov_Value = 0.000062

Covariance of Pixel (2,0)

5000 10000150002000025000300003500040000

Flux(ADU)

50 50 100 150 200 250 300 350 400

Correlations(ADU^2)

Intercept = -0.116212 Slope = -0.000145 Quad Term = 2.88966e-08 Cov_Value = 0.002312

Covariance of Pixel (0,2)

5000 10000150002000025000300003500040000

Flux(ADU)

50 50 100 150 200 250 300 350 400

Correlations(ADU^2)

Intercept = -0.158036 Slope = -0.000129 Quad Term = 1.10857e-08 Cov_Value = 0.000887

Covariance of Pixel (2,2)

.

The intent here is to remove correlations not due to the BF effect.

9 / 15

BF Kernel Correction - Best Results

20 10 20 Covariances(*1E7) 10 8 6 4 2 20 10 5 Covariances(*1E7) - X-Slice 20 10 5 Covariances(*1E7) - Y-Slice 20 10 20 Kernel(*1E7) 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 20 4 2 Kernel(*1E7) - X-Slice 20 4 2 Kernel(*1E7) - Y-Slice

B-F Kernel extracted from 3800 Flats

50000 100000 Central Peak(electrons) 1.00 1.02 1.04 1.06 1.08 1.10 Sigma (Pixels) X Slope = 0.80 % per 50K e-, Intercept = 1.031 Y Slope = 0.91 % per 50K e-, Intercept = 1.033 Corrected X Slope = 0.01 % per 50K e-, Intercept = 1.032 Corrected Y Slope = 0.05 % per 50K e-, Intercept = 1.034

Brighter-Fatter - 30 micron Spots - SEGMENT13

Sigma-x Sigma-y Corrected Sigma-x Corrected Sigma-y

.

Measured covariances are used for nearest neighbors. For R=2 and beyond, use modeled covariances (the green crosses in the left hand plot of Figure 5). Make C00 more negative to force zero sum. Assumes some effect is causing the magnitude of the lost central variance to be reduced.

10 / 15

Use Model for X,Y > 2

20 10 20 Covariances(*1E7) 6 4 2 20 7.5 5.0 2.5 0.0 Covariances(*1E7) - X-Slice 20 5 Covariances(*1E7) - Y-Slice 20 10 20 Kernel(*1E7) 1.5 1.0 0.5 0.0 0.5 20 1 Kernel(*1E7) - X-Slice 20 1 Kernel(*1E7) - Y-Slice

B-F Kernel extracted from 3800 Flats

100 101 102 i 2 + j 2 10-5 10-4 10-3 10-2 10-1 Covariance

C00; Data1=-0.0623; Data2=-0.0623 C10; Data=0.0040; Data2=0.0040 C01; Data=0.0131; Data2=0.0131

Covariance vs Distance As_Measured Model_GT_2 As_Measured_Neg Model_GT_2_Neg 50000 100000 Central Peak(electrons) 1.00 1.02 1.04 1.06 1.08 1.10 Sigma (Pixels) X Slope = 0.80 % per 50K e-, Intercept = 1.027 Y Slope = 0.90 % per 50K e-, Intercept = 1.029 Corrected X Slope = 0.49 % per 50K e-, Intercept = 1.027 Corrected Y Slope = 0.49 % per 50K e-, Intercept = 1.029 Brighter-Fatter - 30 micron Spots - SEGMENT13

Sigma-x Sigma-y Corrected Sigma-x Corrected Sigma-y

11 / 15

Use Model for X,Y > 2 and reduce C00 to force Zero Sum

20 10 20 Covariances(*1E7) 10 8 6 4 2 20 10 5 Covariances(*1E7) - X-Slice 20 10 5 Covariances(*1E7) - Y-Slice 20 10 20 Kernel(*1E7) 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 20 4 2 Kernel(*1E7) - X-Slice 20 4 2 Kernel(*1E7) - Y-Slice

B-F Kernel extracted from 3800 Flats

100 101 102 i 2 + j 2 10-5 10-4 10-3 10-2 10-1 Covariance

C00; Data1=-0.0623; Data2=-0.0955 C10; Data=0.0040; Data2=0.0040 C01; Data=0.0131; Data2=0.0131

Covariance vs Distance As_Measured Model_Zero_Sum As_Measured_Neg Model_Zero_Sum_Neg 50000 100000 Central Peak(electrons) 1.00 1.02 1.04 1.06 1.08 1.10 Sigma (Pixels) X Slope = 0.80 % per 50K e-, Intercept = 1.027 Y Slope = 0.90 % per 50K e-, Intercept = 1.029 Corrected X Slope = 0.02 % per 50K e-, Intercept = 1.027 Corrected Y Slope = 0.02 % per 50K e-, Intercept = 1.029 Brighter-Fatter - 30 micron Spots - SEGMENT13

Sigma-x Sigma-y Corrected Sigma-x Corrected Sigma-y

12 / 15

Impact of Central Pixel Covariance

0.06 0.08 0.10 0.12 0.14

• Central Pixel Covariance

0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Slope(%/50K e-)

Impact of Central Pixel Covariance on BF Correction Success

XSlope_meas YSlope_meas XSlope_model_gt1 YSlope_model_gt1 XSlope_model_gt2 YSlope_model_gt2 .

The main impact is the value of the C00 covariance. Other impacts are small.

13 / 15

BF Kernel Correction - UnModified Covariances, 2nd Order Derivatives

.

Use of 2nd order instead of 4th order derivatives makes little difference.

14 / 15

BF Kernel Correction - Cut off beyond R=2 to force zero sum

.

Forcing zero sum changes the shape of the kernel significantly, but hardly effects the correction results.

15 / 15