Air Force Institute of Technology Air Force Institute of Technology - - PowerPoint PPT Presentation

SLIDE 1

The AFIT of Today is the Air Force of Tomorrow.

Air Force Institute of Technology Air Force Institute of Technology

Dimensional Reduction Analysis Dimensional Reduction Analysis for Physical Layer Device for Physical Layer Device Fingerprints with Application to Fingerprints with Application to ZigBee and Z ZigBee and Z-

• Wave Devices

Wave Devices ZigBee and Z ZigBee and Z-

• Wave Devices

Wave Devices

Authors:

Trevor J. Bihl

Michael A. Temple Kenneth W. Bauer Benjamin Ramsey

US Air Force Institute of Technology Wright-Patterson AFB OH

26-28 Oct 2015

SLIDE 2

)

The AFIT of Today is the Air Force of Tomorrow.

Overview

• Problem Statement
• Background/Setup
• ZigBee and Z-Wave Devices
• Methodology
• RF-DNA Fingerprinting Feature Generation
• GRLVQI Device Discrimination
• GRLVQI Device Discrimination
• Dimensional Reduction Analysis (DRA)
• p-value vs Test Statistic DRA
• Results
• Classification and Verification Results
• Future Work
• Extend to Additional Classifiers
• Develop Additional DRA Methods for RF Fingerprinting

SLIDE 3

The AFIT of Today is the Air Force of Tomorrow.

Problem Statement

Investigate Suitability of p- Values and Test Statistic Based Dimensional Reduction Analysis Dimensional Reduction Analysis (DRA) Methods for Device Fingerprinting Using Radio Frequency Distinct Native Attribute (RF-DNA) Features.

SLIDE 4

• The AFIT of Today is the Air Force of Tomorrow.

ZigBee Z-Wave Standard IEEE Proprietary Frequency 2.4 GHz 906 MHz Bit Rate 250 Kbits/s 40 Kbits/s Security IEEE 802.15.4 Standard None: 200 and 300 Series

Background

ZigBee & Z-Wave Devices

Security IEEE 802.15.4 Standard AES 128: 400 Series Latency 50 to 100 mSec ~1000 mSec Range 10 to 100 m 30 to 100 m Message Size (Bytes) 127 (max) 64 (max)

SLIDE 5

)

The AFIT of Today is the Air Force of Tomorrow.

Methodology

ZigBee Emission Processing [2, 13, 14]

ZigBee Experimental Collection Setup for LOS (A) &

• Experimentally Collected

ZigBee Emissions

• 10 Like-Model Devices
• Collection Environments
• CAGE – Anechoic Chamber
• LOS – Hallway Line-of-Sight (LOS)
• WALL – Through Wall Propagation

ZigBee Experimental Collection Setup for LOS (A) & WALL (B) Environment Emissions [19,54] ZigBee Rogue Device ID and Collection Environments [19,54]

• WALL – Through Wall Propagation
• Authorized Devices
• Emissions Collected in CAGE, LOS, &

WALL for 4 of 10 Devs (Dev 1 – Dev 4)

• NC = 4 Like-Model Auth Devs, Different

Ser #s

• Rogue Devices
• NRog = 9 Like-Model Rogue Devs,

Different Ser #s (Dev 5 – Dev 10)

• Emissions Collected in Selected

Environments (See Table)

5

SLIDE 6

)

The AFIT of Today is the Air Force of Tomorrow.

Methodology

AFIT’s RF-DNA Fingerprinting Process [7]

10 Short OFDM Sym

≈ ≈ ≈ ≈ 8 µ µ µ µSec Duration

Fingerprint Region 1 2 Long OFDM Sym

≈ ≈ ≈ ≈ 8 µ µ µ µSec Duration

Fingerprint Region 2

Entire 802.11a Preamble

≈ ≈ ≈ ≈ 16 µ µ µ µSec Duration

Fingerprint Region 3

RF Statistical Fingerprint Generation

Variance (σ σ σ σ2) Skewness (γ γ γ γ)

Statistical Metrics

1D Non-Transformed

Amplitude (a) Phase (φ φ φ φ) Frequency (f)

Signal Fingerprint Regions

Region 1 Region 2

Burst Extraction Digital Filtering AWGN Generation

Signal Noise

(A) Agilent E3238S (RFSICS) (A) Nat’l Instruments (NI) (B) Riscure Inspector

Agilent E3238S

Bluetooth Access Code Mod: Binary GFSK TB ≈ ≈ ≈ ≈ 72 µSec 2 Spectral Comp 802.11a Preamble Mod: 64 - OFDM TB ≈ ≈ ≈ ≈ 16 µSec 52 Spectral Comp GSM Midamble Mod: Binary GMSK TB ≈ ≈ ≈ ≈ 96 µSec 2 Spectral Comp

t

RF Statistical Fingerprint (1D Non-Transformed) #Features (NF) = ( NR Regions X 3 Char X 3 Statistics )

( ( ( ( ) ) ) )

1 2 2 2 2 1 1

F F F F F

i i i i i i i Ri NR R i F F i a a a f N R R f N f R N φ φ φ φ φ φ φ φ φ φ φ φ

σ γ κ σ γ κ σ γ κ σ γ κ σ γ κ σ γ κ σ γ κ σ γ κ σ γ κ σ γ κ σ γ κ σ γ κ

× × × × × × × ×

        = = = =                         ⇒ = ⇒ = ⇒ = ⇒ =        

• M

M K M M K M M K M M K

Skewness (γ γ γ γ) Kurtosis (k) Region NR

• Cisco

Netgear Linksys RF DNA Markers

RF DNA Markers

CISCO LINKSYS NETGEAR

Representative Fingerprints

Statistical Fingerprint Generation

Post-Collection Processing (MATLAB)

Classification and/or Verification SNR Scaling Analysis Signal

Signal

Power Norm

Noise

Device Classification ROC Verification

SNR (dB) % Correct Classification

Pct Correct Dev 1 Dev 2 Dev 3 Dev 4 Mean

1 vs. M Assessment

1 .5 3 6 9 12 15 18

False Accept Rate (FAR) True Accept Rate (TAR)

Equal Error Rate (EER) SNR = 12 dB SNR = 15 dB SNR = 18 dB

1 vs. 1 Assessment

0 .2 .4 .6 .8 1 1 .5 2D T-F Transforms

Fourier Frac Fourier Wavelet Gabor Etc.

Model Development

MDA/ML Illustration

6

SLIDE 7

)

The AFIT of Today is the Air Force of Tomorrow.

Methodology

ZigBee Emission Processing [2, 13, 14]

Non-Transformed Instantaneous:

(a) Amplitude (b) Phase (c) Frequency ZigBee SHR Inst Amp Response

Time Domain (TD) RF-DNA Fingerprint Generation

Fingerprints Input to Classifier Model Development

1 2

2 1 4

thRegion

1 4

F Composite Fingerprint F F F

i NR R

i R i i i i R R R N

σ σ γ κ

× × ×

  =       M M L M

(U) Region of Interest (ROI)

7

SLIDE 8

)

The AFIT of Today is the Air Force of Tomorrow.

Methodology

Device Classification: GRLVQI

• LVQ-Based Classifiers
• Gradient Descent & Prototype Vector (PV)

Approach for Classification

• Gradient = 1st Derivative of Cost Function
• Iteratively Examines PV-to-Data Distances
• Correctly Classified PVs N Move Toward data
• Incorrectly Classified PVs N Move Away From Data
• GRLVQI N LVQ Extension [2, 9, 14]

LVQ Update

Artificial Neural Net (ANN) Learning Vector Quant. (LVQ)

• GRLVQI N LVQ Extension [2, 9, 14]
• G = Generalized N Sigmoidal Cost Function
• R = Relevance N Gradient Descent Feature

Relevance Ranking

• I = Improved N Improved Logic, PV Freq, Add’l

Learn Rate, Etc.

• No Explicit Assumption / Knowledge

Required for Data Distribution (PDF)

• Appropriate PV Initialization Required
• Normal PVs ⇒

⇒ ⇒ ⇒ Standardized Data

Cls 1 Cls 2 Cls 3

p3, j

8 LVQ Update [60,61]

GRLVQI

Cls 1 Cls 2 Cls 3

p3, j

Iteration 0 Iteration N

LVQ Update

SLIDE 9

)

The AFIT of Today is the Air Force of Tomorrow.

Methodology

Dimensional Reduction Analysis (DRA)

• Method #1: (Distribution Based): Two

Sample Kolmogorov–Smirnov (KS) [13,14, 17]

• Method #2: (Distribution Based): ANOVA

F-Statistics

( )

(x) F (x) F max KS

2 1

− =

300 400 500 600 ay ANOVA F-test Values

Amp Phz Freq F-Statistics [18]

• Method #3: (Classifier Based) GRLVQI

Relevance [9]

• Method #4: Dimensionality Assessment [18,

21]

100 200 300 400 500 600 700 750 100 200 One Way AN RF Fingerprint Component

9

) ( ) ( ) ( i Model i Feature i

MSE MS F =

Amplitude (a) : ZigBee Feats #1 - #243 Phase (φ φ φ φ) : ZigBee Feats #244 - #486 Frequency (f) : ZigBee Feats #487 - #729

SLIDE 10

)

The AFIT of Today is the Air Force of Tomorrow.

Methodology

DRA: Dimensionality Assessment

• Selecting quantity of features in

subsets non-trivial

• Qualitative DRA
• Previously Considered [13,14]
• NDRA, ZigBee = [25, 50, 243]
• Quantitative DRA
• Introduced Here

SNR (DB) METHOD

SIGNIFICANCE LEVEL

0.1% 1% 5% 10% F-TEST 196 264 350 402 KS-TEST (ΣP-VALUES) 37 74 130 160 10 F-TEST 589 639 674 688 KS-TEST (ΣP-VALUES) 337 414 512 557 18 F-TEST 706 713 720 722 KS-TEST (ΣP-VALUES) 666 692 711 716 F-TEST 718 725 727 728 ZigBee Dimensionality Assessment by Significance Level

• Introduced Here
• Removes Subjectively
• Intrinsic Data Dimensionality
• P-value and Data Eigenvalue

methods considered

• P-values Overestimate Required

NDRA

• Data Eigenvalue Methods Yield

NDRA Consistent with Prior Work

• NDRA, ZigBee = [17, 123]
• NDRA, Z-wave = [7, 34]

30 F-TEST 718 725 727 728 KS-TEST (ΣP-VALUES) 727 729 729 729

100 200 300 400 500 600 700 800 10

• 8

10

• 6

10

• 4

10

• 2

10 10

2

Eigenvalues Magnitude Covariance Eigenvalues for Training Data at SNR 18dB COV Eigenvalues Broken Stick Kaiser (> mean) Kaiser (> 1)

ZigBee Dimensionality Assessment by COV Eigenvalues

SLIDE 11

)

The AFIT of Today is the Air Force of Tomorrow.

• The mapping between test statistic

and p-value is typically nonlinear

• Simple F-Test Stat. [18]
• Complicated F-Test p-value [18]

Methodology

DRA: Test Statistics vs p-Values

• Recent RF-DNA DRA Research

Focused on p-values for feature relevance ranking [1, 2, 13-14, 28-29]

• Test Statistic to p-Value Conversion

Req’d

• Computing Test Statistic Values
• Ratio between quantities or a simple

relationship

0.75

u  

) ( ) ( ) ( i Model i Feature i

MSE MS F =

• The KS-test involves a similar

nonlinear mapping [17]

relationship

• Test Statistics vs. P-Values
• p-Values Represent Area Under a

Probability Curve

• Computing p-Values Requires [26]
• 1. Stated Hypothesis Test
• 2. Test Statistic Value
• 3. Degrees of Freedom
• 4. Distributional Assumption
• 5. Reference Distribution

(Not all are always considered / stated in DRA, e.g. [1, 2, 13, 14] )

1 2 3 4 5 6 7 8 9 10 0.25 0.5 0.75 Test Statistic Value

P-value=AUC

( )

( ) 2

/ 1 2 2

1 2 2 2 , |

v u u u

x v u v u x v u v u v u x f

+ −            

      +             Γ       Γ             + Γ =

11

SLIDE 12

)

The AFIT of Today is the Air Force of Tomorrow.

Methodology

ZigBee DRA: Test Statistics vs P-Values

p-values Test Statistic Values

0.02 0.04 0.06 0.08 0.1 0.12 Number 30 dB 18 dB 10 dB 0 dB 0.2 0.4 0.6 0.8 1 Number 30 dB 18 dB 10 dB 0 dB

KS-test

12

0.1 0.2 0.3 0.4 0.5 0.6 0.02 0.04 0.06 0.08 0.10 0.12 F-Test Value Number 30 dB 18 dB 10 dB 0 dB 0.1 0.2 0.3 0.4 0.5 0.6 0.2 0.4 0.6 0.8 1.0 P-value Number 30 dB 18 dB 10 dB 0 dB 0.1 0.2 0.3 0.4 0.5 0.6 0.02 Test Statistic Value 0.1 0.2 0.3 0.4 0.5 0.6 0.2 P-value

F-test

SLIDE 13

)

The AFIT of Today is the Air Force of Tomorrow.

Methodology

ZigBee DRA: Test Statistics vs P-Values

p-values Test Statistic Values

0.02 0.04 0.06 0.08 0.1 0.12 Number 30 dB 18 dB 10 dB 0 dB 0.2 0.4 0.6 0.8 1 Number 30 dB 18 dB 10 dB 0 dB

KS-test

• With large datasets, p-values tend towards zero

Hence resolution is lost when converting to p-values

13

0.1 0.2 0.3 0.4 0.5 0.6 0.02 0.04 0.06 0.08 0.10 0.12 F-Test Value Number 30 dB 18 dB 10 dB 0 dB 0.1 0.2 0.3 0.4 0.5 0.6 0.2 0.4 0.6 0.8 1.0 P-value Number 30 dB 18 dB 10 dB 0 dB 0.1 0.2 0.3 0.4 0.5 0.6 0.02 Test Statistic Value 0.1 0.2 0.3 0.4 0.5 0.6 0.2 P-value

F-test

• Hence resolution is lost when converting to p-values
• Interpretation/procedural issues also remain
• How to compare and rank equivalent values?

SLIDE 14

)

The AFIT of Today is the Air Force of Tomorrow.

• Test statistic methods offer comparable or better performance

to p-value based methods

0.8 1 0.8 0.9 1 F-test p-value F-test statistic KS-test p-value

ZigBee Classification, NDRA = 17 Z-Wave Classification, NDRA = 34

Results

Device Classification: ZigBee & Z-Wave

5 10 15 20 25 30 0.2 0.4 0.6 0.8 SNR (dB) Ave Pct Correct Baseline KS-Test p-value KS-Test statistic F-test p-value F-test statistic GRLVQI Relevance 5 10 15 20 25 0.4 0.5 0.6 0.7 0.8 SNR (dB) Ave Pct Correct KS-test p-value KS-test statistic GRLVQI Relevance Baseline

SLIDE 15

)

The AFIT of Today is the Air Force of Tomorrow.

Results

Device ID Verification: ZigBee

• Based on “one vs one” claimed identity scenarios
• Presented as:
• %TVR = True Verification Rate
• % RRR = Rogue Rejection Rate
• Bold Entry - Best or Statistically Equivalent Performance

DRA METHOD KS TEST STATISTIC KS ΣP-VALUE NF 17 50 123 17 50 123 TVR 0% 0% 0% 0% 0% 0% TVR 0% 0% 0% 0% 0% 0% RRR 8.33% 8.33% 0% 52.8% 2.78% 0% DRA METHOD F TEST STATISTIC F TEST P-VALUE NF 17 50 123 17 50 123 TVR 0% 0% 0% 25% 0% 0% RRR 8.33% 5.56% 0% 38.9% 19.4% 0% DRA METHOD GRLVQI NF 17 50 123 TVR 25% 50% 50% RRR 52.8% 66.7% 72.2%

SLIDE 16

耀

The AFIT of Today is the Air Force of Tomorrow.

Results

Device ID Verification: ZigBee

• Based on “one vs one” claimed identity scenarios
• Presented as:
• %TVR = True Verification Rate
• % RRR = Rogue Rejection Rate
• Bold Entry - Best or Statistically Equivalent Performance

DRA METHOD KS TEST STATISTIC KS ΣP-VALUE NF 17 50 123 17 50 123 TVR 0% 0% 0% 0% 0% 0% TVR 0% 0% 0% 0% 0% 0% RRR 8.33% 8.33% 0% 52.8% 2.78% 0% DRA METHOD F TEST STATISTIC F TEST P-VALUE NF 17 50 123 17 50 123 TVR 0% 0% 0% 25% 0% 0% RRR 8.33% 5.56% 0% 38.9% 19.4% 0% DRA METHOD GRLVQI NF 17 50 123 TVR 25% 50% 50% RRR 52.8% 66.7% 72.2%

• Distribution-based DRA offers poor verification

performance with non-linear GRLVQI classifier

SLIDE 17

£

The AFIT of Today is the Air Force of Tomorrow.

Conclusions & Future Work

Conclusions

• Introduction of F-test for DRA in RF Fingerprinting
• Test Statistic Methods vs P-values
• P-values Susceptible to Converge on 0 [26]
• Test Statistic DRA Offers Robustness
• Introduction Quantitative Dimensionality Assessment
• Introduction Quantitative Dimensionality Assessment
• NDRA = 123 (quantitative) better than NDRA = 243 (qualitative) of [14]
• Comparison of 5 DRA Methods for RF Fingerprinting
• First Look RF-DNA Fingerprinting Using Z-Wave Devices

Future Work

• Expand Z-Wave Assessments to Include Rogue Devices
• Reevaluate with an MDA-based classifier

SLIDE 18

]

The AFIT of Today is the Air Force of Tomorrow.

[1] B. W. Ramsey, B. E. Mullins, R. Speers and K. A. Batterton, "Watching for weakness in wild WPANs," Military Comm. Conf. (MILCOM), pp. 1404-1409, 2013. [2] B. W. Ramsey, M. A. Temple and B. E. Mullins, "PHY foundation for multi-factor ZigBee node authentication," IEEE Global Comm. Conf. (GLOBECOM), pp. 795-800, 2012. [3] Y. Zatout, "Using wireless technologies for healthcare monitoring at home: A survey," Int. Conf. e- Health Networking, Applicat. and Services (Healthcom), pp. 383-386, 2012. [4] J. Wright, "KillerBee: Practical ZigBee exploitation framework," in 11th ToorCon Conf., San Diego, 2009.

References

2009. [5] Y. Sheng, K. Tan, G. Chen, D. Kotz and A. Campbell, "Detecting 802.11 MAC layer spoofing using received signal strength," 27th Conf. on Comput. Comm., 2008. [6] B. Danev, D. Zanetti and S. Capkun, "On physical-layer identification of wireless devices," ACM Computing Surveys, vol. 45, no. 1, 2012. [7] W. E. Cobb, E. W. Garcia, M. A. Temple, R. O. Baldwin and Y. C. Kim, "Physical layer identification

• f embedded devices using RF-DNA fingerprinting," Military Comm. Conf. (MILCOM), pp. 2168-

2173, 2010. [8] M. D. Williams, M. A. Temple and D. R. Reising, "Augmenting bit-level network security using physical layer RF-DNA fingerprinting," IEEE Global Comm. Conf. (GLOBECOM), pp. 1-6, 2010. [9] P. K. Harmer, D. R. Reising and M. A. Temple, "Classifier selection for physical layer security augmentation in Cognitive Radio networks," IEEE Int. Conf. on Comm.(ICC), pp. 2846-2851, 2013. [10] T. Wu, J. Duchateau, J.-P. Martens and D. van Compernolle, "Feature subset selection for improved native accent identification," Speech Comm., vol. 52, no. 2, pp. 83-98, 2010.

SLIDE 19

£

The AFIT of Today is the Air Force of Tomorrow.

[11] A.-C. Haury, P. Gestraud and J.-P. Vert, "The Influence of Feature Selection Methods on Accuracy, Stability and Interpretability of Molecular Signatures," PLoS ONE, vol. 6, no. 12, 2011. [12] T. Kind, V. Tolstikov, O. Fiehn and R. H. Weiss, "A comprehensive urinary metabolomic approach for identifying kidney cancer," Analytical Biochemistry, vol. 363, 2007. [13] C. K. Dubendorfer, B. W. Ramsey and M. A. Temple, "An RF-DNA verification process for ZigBee networks," Military Comm. Conf. (MILCOM), pp. 1-6, 2012. [14] C. K. Dubendorfer, B. W. Ramsey and M. A. Temple, "ZigBee device verification for securing industrial control and building automation systems," Int. Conf. on Critical Infrastructure Protection

References

industrial control and building automation systems," Int. Conf. on Critical Infrastructure Protection (IFIP13), vol. 417, pp. 47-62, 2013. [15] S. Prabhakar, S. Pankanti and A. K. Jain, "Biometric recognition: Security and privacy concerns," IEEE Security and Privacy, pp. 33-42, March/April 2003. [16] A. K. Jain, R. P. Duin and J. Mao, "Statistical Pattern Recognition: a Review," IEEE Trans. on Pattern Anal. Mach. Intell., vol. 22, no. 1, pp. 4-37, Jan. 2000. [17] W. J. Conover, Practical Nonparametric Statistics, 2nd ed., New York: John Wiley & Sons, pp. 344-385, 1980. [18] W. R. Dillon and M. Goldstein, Multivariate Analysis Methods and Applications, New York: John Wiley & Sons, 1984. [19] J. D. Habbema and J. Hermans, "Selection of variables in discriminant analysis by F-statistic and error rate," Technometrics, vol. 19, no. 4, pp. 487-493, 1977. [20] M. Cowles and C. Davis, "On the Origins of the .05 Level of Statistical Significance," Amer. Psychologist, vol. 37, no. 5, pp. 553-558, 1982.

SLIDE 20

£

The AFIT of Today is the Air Force of Tomorrow.

[21] R. J. Johnson, J. P. Williams and K. W. Bauer, "AutoGAD: An improved ICA-based hyperspectral anomaly detection algorithm," IEEE Trans. Geosci. Remote Sens., vol. 51, no. 6, pp. 3492-3503, 2013. [22] C. J. Huberty and J. M. Wisenbaker, "Variable importance in multivariate group comparisons," J.

• f Education Stat., vol. 17, no. 1, pp. 75-91, 1992.

[23] A. Cord, C. Ambroise and J.-P. Cocquerez, "Feature selection in robust clustering based on Laplace mixture," Pattern Recognition Lett., vol. 27, no. 6, pp. 627-635, 2006. [24] P. Radivojac, Z. Obradovic, A. K. Dunker and S. Vucetic, "Feature selection filters based on the

References

[24] P. Radivojac, Z. Obradovic, A. K. Dunker and S. Vucetic, "Feature selection filters based on the permutation test," Mach. Learning: ECML 2004, pp. 334-346, 2004. [25] K. Schmidt, T. Behrens and T. Scholten, "Instance selection and classification tree analysis for large spatial datasets in digital soil mapping," Geoderma, vol. 146, no. 1-2, pp. 138-146, 2008. [26] L. G. Halsey, D. Curran-Everett, S. L. Vowler and G. B. Drummond, "The fickle P value generates irreproducible results," Nature Methods, vol. 12, no. 3, pp. 179-185, 2015.