International Earth Science Constellation Mission Operations Working - - PowerPoint PPT Presentation

Mission Operations Working Group

June 13-15, 2017

1

International Earth Science Constellation Mission Operations Working Group

June 13-15, 2017

Earth Observing System Covariance Realism Updates

Juan Ojeda Romero, a.i. solutions, Inc. / Code 595 Fred Miguel, a.i. solutions, Inc. / Code 595 EOS FDS, esmo-eos-fds@lists.nasa.gov, +1.301.614.5050

https://ntrs.nasa.gov/search.jsp?R=20170005601 2018-04-29T17:09:18+00:00Z

Mission Operations Working Group

June 13-15, 2017

• Overview of Aqua/Aura Covariance Operations

– Earth Observing System (EOS) Flight Dynamics System (FDS) Covariance Realism QA (Quality Assurance) and Tuning Flowchart – Covariance QA Automation – Aqua and Aura Covariance Tuning – Automation Results to Date – Covariance Propagation through Maneuvers

• Future Analysis/Work

– Covariance Propagation Implementation through Maneuvers – Covariance Propagation using Polynomial Chaos Expansion

• Conclusion

Agenda

2

Mission Operations Working Group

June 13-15, 2017

• Aqua and Aura Owner/Operator (O/O) covariances are being used in operations

to compute the probability of collision (PC).

• This only includes daily operations and Drag Make-Up (DMU) maneuver

planning.

• Software has been delivering tuned covariance since June 14, 2016.
• Software ensures covariances are tuned for periods devoid of persistently high

and extreme solar activity as well as post maneuver propagation errors.

• Aqua’s last tuning date was on November 7, 2016.
• Aura’s last tuning date was on November 9, 2016.

Overview of Aqua/Aura Covariance Operations

3

Mission Operations Working Group

June 13-15, 2017

Step 1: Input Radial, In-Track, Cross-Track (RIC) Component Acceleration Variances Step 2: Propagate Daily Definitive Ephemeris + Covariance using RIC Component Acceleration Variances Step 3: Collect Sets of Propagation Errors and Predictive Covariances Step 4: Compute the Chi- Square Statistic over multiple propagation points Step 6: Perform the 3- degree of freedom (DOF) Chi-Square Distribution Test to Determine Realism Pass Percentage Step 7: Tune Covariance if the Pass Percentage falls under a User Specified Threshold

• The acceleration variances in Step 1

can only be changed after the tuning process. Updated variances are configuration managed and require approval before they are deployed to operations.

• Step 2 is performed as part of the

nominal daily product delivery.

• Steps 3 to 6 represent the QA of the

covariance and are performed via automation using FreeFlyer and MATLAB.

• QA of Aqua and Aura covariances

is performed over a rolling 90-day timespan.

• Testing with a 3-day cadence is

statistically required in order to isolate the affects of the 2 ½ days worth of rolling Tracking and Data Relay Satellite (TDRS)

• bservations that go into daily

Flight Dynamics Facility (FDF)

• rbit determination runs.

Step 5: Use the Normalized Standard In-Track Errors to Determine Outlier Propagations

EOS Covariance Realism QA and Tuning Flowchart

4

Mission Operations Working Group

June 13-15, 2017

0.5 1 1.5 2 2.5 3 3.5 5 10 15 20 25 Propagation Time (Days) Cross-Track State Estimate Error (m), Set 1 0.5 1 1.5 2 2.5 3 3.5 100 200 300 400 500 600 700 Propagtion Time (Days) In-Track State Estimate Error (m), Set 1 0.5 1 1.5 2 2.5 3 3.5 1 2 3 4 5 6 7 8 9 Propagation Time (Days) Radial State Estimate Error (m), Set 1

• Example Aqua Set 1 QA Results for April 11, 2017 are given above.
• Component Estimate Error plots give an idea of how far each

component covariance is deviating from its mean root mean squared (RMS) component error.

• An empirical 3-DOF Chi-Square distribution for each propagation

point is assessed against its parent distribution.

• The Cramer-Von Mises empirical distribution function (EDF) test is

used to determine the likelihood each set of covariances represents a realistic distribution of the corresponding set of propagation errors tied to it – A “Pass Percentage” is used to determine Covariance Realism.

Covariance QA Automation Visual Aids Presented to Analyst (1 of 2)

5

01/07/17(01) 01/13/17(02) 01/16/17(03) 01/19/17(04) 01/22/17(05) 01/25/17(06) 01/28/17(07) 01/31/17(08) 02/03/17(09) 02/12/17(10) 02/15/17(11) 02/18/17(12) 02/21/17(13) 02/24/17(14) 03/02/17(15) 03/11/17(16) 03/14/17(17) 03/17/17(18) 03/23/17(19) 04/04/17(20) Mean Error Mean RIC Variance 04/01/17(Outlier)

5 10 15 20 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 C-1 3 DOF 2 CDF, Set 1 Ideal Estimated

Identified Outlier

Mission Operations Working Group

June 13-15, 2017

1 2 3 4

• 3
• 2
• 1

1 2 3 4 5 Propagation Time (Days) In-Track Standard , Set 1

• 4
• 2

2 4 6 0.2 0.4 0.6 0.8 1 C-1 Normalized In-Track Error Normal CDF, Set 1

Ideal Estimated

• Standard Component Errors are available for

Radial, In-Track, and Cross-Track directions. In-Track Standard Errors are utilized in Outlier Identification Process.

• Any propagations outside of the ± 1σ

bounds in the In-Track Component are tested for outlier identification

• Normal Gaussian distribution based on

Component Errors are also available.

Covariance QA Automation Visual Aids Presented to Analyst (2 of 2)

6

1 2 3 0.05 0.1 0.15 0.2 0.25 0.3 Propagation Time (Days) P-value, Set 1

• The Probability Value (P-Value) vs. Propagation Time chart gives

information regarding where in the propagation the covariances are passing the realism testing.

• A “Pass-Percentage” is calculated for all sets based on the P-values

calculated through the timeframe at every step. Based on seasonal covariance tuning from 2014 to 2016, FDS recommended this threshold be set to 60% – a statistically commendable result.

• Periodicity in the Radial Propagation Error is causing low levels of

realism between 0.5 to 1.25 days. The Covariance is oversized in this timeframe.

P-Value Threshold

Identified Outlier

Mission Operations Working Group

June 13-15, 2017

02/06/17 03/08/17 04/04/17 50 100 Solar Flux F10.7 50 100 Date Geomagnetic Index Estimated Solar Flux Geomagnetic Index Identified Outlier

1 2 3 4

• 3
• 2
• 1

1 2 3 4 5 Propagation Time (Days) In-Track Standard , Set 1

Covariance QA Automation Outlier Identification Confirmation

• Automation identifies potential outliers based on the In-

Track standard errors. Propagations with an In-Track standard error outside ± 1σ bounds after 3.5 days will be tested.

• Automation uses a Rosner Outlier Test on any deviant

normalized In-Track standard errors – the test will detect

• utliers that are either much smaller or larger than the rest
• f the data and is designed to avoid the problem of

masking, where an outlier close to another outlier goes undetected.

• The outliers are entered into the test in order of most to

least deviant.

• Naturally, the solar activity in the timeframe of the

propagation start date is used to determine if there was a peak or persistently high solar activity. See figure to the left.

Potential Outlier Propagations Note: Only the four most deviant propagations are tested using the Rosner Outlier Test.

7 April 1st, 2017

Mission Operations Working Group

June 13-15, 2017

0.5 1 1.5 2 2.5 3 3.5 5 10 15 Time,Days Uncertainty, (m) Cross-Track Covariance vs. Mean Error from QA Sets Mean RMS Set 1 Mean RMS Set 2 Mean RMS Set 3 0.5 1 1.5 2 2.5 3 3.5 50 100 150 200 250 300 350 Time,Days Uncertainty, (m) In-Track Covariance vs. Mean Error from QA Sets Mean RMS Set 1 Mean RMS Set 2 Mean RMS Set 3

• Aqua’s P-value Pass Percentage decreased below the FDS

imposed threshold (60%) on November 7, 2016. Aqua was tuned to improve covariance realism.

• The component acceleration variances are changed until

the Pass Percentages for all three sets of covariances exceed the user specified threshold.

• The current strategy is to tune the covariance to the largest

Mean RMS Component Error in the Radial and In- Track directions at the final propagation point and to the mid propagation point in the Cross-Track direction (to achieve the highest level of realism).

The Cross-Track covariance is tuned to the mid propagation point

Covariance QA Automation Tune the Covariance (Aqua)

8

Decrease in In-Track Acceleration Variance

Before Tuning After Tuning

0.5 1 1.5 2 2.5 3 3.5 1 2 3 4 5 Time,Days Uncertainty, (m) Radial Covariance vs. Mean Error from QA Sets Mean RMS Set 1 Mean RMS Set 2 Mean RMS Set 3

Before Tuning After Tuning Before Tuning After Tuning

Mission Operations Working Group

June 13-15, 2017

0.5 1 1.5 2 2.5 3 3.5 5 10 15 Time,Days Uncertainty, (m) Cross-Track Covariance vs. Mean Error from QA Sets Mean RMS Set 1 Mean RMS Set 2 Mean RMS Set 3 0.5 1 1.5 2 2.5 3 3.5 1 2 3 4 5 6 Time,Days Uncertainty, (m) Radial Covariance vs. Mean Error from QA Sets Mean RMS Set 1 Mean RMS Set 2 Mean RMS Set 3 0.5 1 1.5 2 2.5 3 3.5 50 100 150 200 250 300 350 Time,Days Uncertainty, (m) In-Track Covariance vs. Mean Error from QA Sets Mean RMS Set 1 Mean RMS Set 2 Mean RMS Set 3

• Aura’s Covariance was tuned in parallel with Aqua (on

November 9, 2017). Aura’s P-value Passing Percentage was improved after tuning and Aura did not fall under the FDS imposed threshold (60%).

• A similar tuning strategy was applied to Aura’s covariance.

The Cross-Track covariance is tuned to the mid propagation point

Covariance QA Automation Tune the Covariance (Aura)

9

Decrease In-Track Acceleration Variance

Before Tuning After Tuning Before Tuning After Tuning Before Tuning After Tuning

Mission Operations Working Group

June 13-15, 2017

Automated Covariance QA Results to Date (Aqua)

10

55% 60% 65% 70% 75% 80% 85% 90% 95% 100% 6/2016 8/2016 10/2016 12/2016 2/2017 4/2017

Covariance Realism Pass Percentage

Covariance QA Analysis Date

Group A Group B Group C Pass Limit 1 2 3 4 5 6 7 06/2016 08/2016 10/2016 12/2016 02/2017 04/2017

Number of Outliers

Covariance QA Analysis Date

Last Tuning Date (11/7/2017)

Mission Operations Working Group

June 13-15, 2017

Automated Covariance QA Results to Date (Aura)

11

1 2 3 4 5 6/2016 8/2016 10/2016 12/2016 2/2017 4/2017

Number of Outliers

Covariance QA Analysis Date

55% 60% 65% 70% 75% 80% 85% 90% 95% 100% 6/2016 8/2016 10/2016 12/2016 2/2017 4/2017

Covariance Realism Pass Percentage

Covariance QA Analysis Date

Group A Group B Group C Pass Limit

Last Tuning Date (11/9/2017)

Mission Operations Working Group

June 13-15, 2017

• The method that EOS FDS has adapted to account for maneuver execution error includes the following

stages:

• 1. Gaussian distribution testing and outlier investigation of the Delta-V component errors

Operational

• 2. Creation of a Error Covariance Matrix using the preceding Delta-V component errors

Operational

• 3. Error Covariance Matrix Scaling using RIC Scale Factors

Operational

• 4. Application of Dataset Biases (An Enhancement to the Maneuver Planning Process)

Future

• The preceding method is tested by propagating the Error Covariance Matrix through historical

maneuvers and performing a covariance realism analysis on the resulting predicted post-maneuver propagation errors.

• The mean of each of the Radial, In-Track, and Cross-Track ΔV components constitutes the maneuver

execution bias, μ.

• There are two approaches to make use of bias:

Approach 1 – Add bias to the planned maneuver plan. For example, if an In-Track ΔV component

• f 10 cm/s is planned and the bias is +2.5 cm/s then this bias is added to the satellite’s state estimate

propagation at the time of the maneuver. Approach 2 – Add bias to the commanded maneuver plan. For example, if an In-Track ΔV component of 10 cm/s is planned and the bias is +2.5 cm/s then the plan is changed to 7.5 cm/s prior to the propagation of the satellite’s state estimate.

Overview of Covariance Propagation through Maneuvers

12

EOS FDS Preferred Approach

Mission Operations Working Group

June 13-15, 2017

• EOS FDS is investigating new methods of adding the maneuver Execution Error

Sample Covariance to the propagated covariance throughout inclination adjust maneuvers (IAMs).

• Updated Linearized Covariance Propagation – The formula for linearly propagating

covariance through maneuvers: P tn = Φ tn, tn−1 𝑈 P tn−1 + Qm(t) P tn−1 ΦT tn, tn−1 + Q t where P tn = Initial Covariance Matrix Φ tn, tn−1 = State Transition Matrix Q t = Process Noise Matrix Qm(t) is non-zero only during the execution of the maneuver

• This method will be analyzed for Aqua and Aura DMUs for improved covariance
• realism. IAMs will be an extended case of DMUs.

Covariance Propagation through Maneuvers Future Work

13

Mission Operations Working Group

June 13-15, 2017

Future Work Polynomial Chaos Expansion (PCE)

• EOS FDS is exploring a new method of covariance propagation by using Polynomial

Chaos Expansion (PCE) methods. This method is based on the paper Conjunction Assessment Using Polynomial Chaos Expansions by Brandon Jones, Alireza Doostan, and George Born, in which PCEs were used to calculate conjunction Pc.

• PCE Methods maps stochastic inputs (in this case, some initial position/velocity state)

to a spectral polynomial solution space. That is to say, a spacecraft state can be approximated by: – 𝑌 𝑢, 𝜊 = 𝛽∈Λ𝑞,𝑒 𝑑𝛽(𝑢)𝜔𝛽(𝜊) where 𝑌 is the position/velocity state, 𝜊 is the stochastic input, 𝜔𝛽 is the basis polynomial being mapped to (in this case, Hermite Polynomials), and 𝑑𝛽 is the coefficient of the polynomial (to be solved).

14

Mission Operations Working Group

June 13-15, 2017

Polynomial Chaos Expansion (PCE) Method of Solving

• PCE Equations are solved for by mapping multiple stochastic inputs to corresponding
• utputs of the desired model. In this case, multiple propagations of the Aqua spacecraft

based on a Gaussian distributed initial states (such as graphed below).

Method: 1. Generate 𝑂 realizations (based on the number of coefficients to solve for) of 𝜊𝑗 which are Guassian distributed. 2. For each 𝜊𝑗, use initial 𝑌(𝑢, 𝜊) based on the random input 𝜊𝑗 and propagate 𝑌(𝑢, 𝜊) to some time t for each 𝑂 realizations (graphed

• n the left).

3. Solve for 𝑑𝛽(𝑢) based on the 𝑂 final states 𝑌(𝑢𝑔, 𝜊𝑔) (in this case, by using least- square regression)

15

Mission Operations Working Group

June 13-15, 2017

Polynomial Chaos Expansion (PCE) Advantages

• PCEs can be very useful when Gaussian uncertainties to the model input are

introduced (such as possibly during high solar activity, maneuvers, and spacecraft configuration changes).

• Once the coefficients of the PCE are solved, it is a complete state representation of

the system. Thus, one could use the PCE approximations in Monte Carlo type analysis where propagations could instead be replaced by evaluations of PCE Polynomials—a much less computationally demanding method.

1 2 3 0.05 0.1 0.15 0.2 0.25 0.3 Propagation Time (Days) P-value, Set 1 P-Value Threshold

• There is enough generalizations in

mapping input that it can be applied to more than just position/velocity state

• variations. Could be applied to other

type of inputs, such as yaw angle and burn time uncertainties during maneuvers.

• This may help improve low periods of

realism during 0.5-1.25 days of propagation, see right.

0.5-1.25 days

16

Mission Operations Working Group

June 13-15, 2017

• The EOS FDS team has been using tuned O/O covariance for Aqua and Aura:

– The automation of the covariance QA has been established, tested, and working as expected. – The automation of outlier identification has been established, tested, and working as expected.

• The EOS FDS team has been using O/O covariance with maneuver execution error

assuming zero-bias for DMUs: – Gaussian distribution testing of the maneuver component errors has been established and working as expected. – Maneuver Execution Error Covariances to be updated on a bi-annual basis.

• The EOS FDS team is looking into new methods of covariance propagation throughout

maneuvers and new covariance propagation methods using Polynomial Chaos Expansion (PCE). Conclusion

17