Big Data Challenges and Opportunities Ira A. (Gus) Hunt Chief - - PDF document

▶
big data
SMART_READER_LITE
LIVE PREVIEW

Big Data Challenges and Opportunities Ira A. (Gus) Hunt Chief - - PDF document

Jan 25, 2024 450 likes •700 views

Big Data Challenges and Opportunities Ira A. (Gus) Hunt Chief Technology Officer Our Mission We are the nation's first line of defense. We accomplish what others cannot accomplish and go where others cannot go. We carry out our mission by:

slide-1
SLIDE 1

1

Big Data

Challenges and Opportunities

Ira A. (Gus) Hunt Chief Technology Officer

Our Mission

We are the nation's first line of defense. We accomplish what others cannot accomplish and go where others cannot go. We carry out our mission by:

Collecting information that reveals the plans, intentions and capabilities of our adversaries and provides the basis for decision and action. Producing timely analysis that provides insight, warning and

  • pportunity to the President and decisionmakers charged with

protecting and advancing America's interests. Conducting covert action at the direction of the President to preempt threats or achieve US policy objectives.

slide-2
SLIDE 2

2

2

It’s a

Big Data

World

Google

> 100 PB > 1T indexed URLs

3
slide-3
SLIDE 3

3

4

FaceBook

> 800M users

> 100PB

5

YouTube

> 750PB

>200,000 4TB drives

slide-4
SLIDE 4

4

6

World Population

> 6,987,139,094

7

Twitter

> 55B tweets/year

> 150M/day

>1700/sec

slide-5
SLIDE 5

5

8

Global Text Messages

> 6.1T per year

> 193,000 per second

> 876 per person per year

9

US Cell Calls

> 2.2 T minutes/year

> 19 minutes / person / day

(uncompressed~1 YouTube/year)

slide-6
SLIDE 6

6

10

3

Driving Forces

Social

11

Mobile Cloud

slide-7
SLIDE 7

7

12

+ +

=

+

13
slide-8
SLIDE 8

8

14

2 3

Our Job Leverage the Big Data world Find the Information that Matters Connect the Dots Understand the Plans of our Adversaries

Prevent an attack, Save lives, Safeguard our national security

1 4

slide-9
SLIDE 9

9

16

Why We Care

17

Why We Care

slide-10
SLIDE 10

10

18

Why We Care

19

Why We Care

slide-11
SLIDE 11

11

The Problem

20

2 3

Our Problem: Which 5K

Don’t know the future value of a dot today We cannot connect dots we don’t have The old collect, winnow, dissem model

fails spectacularly in the Big Data world

The few cannot know the needs of the many

1

Secure the data, Connect the data, Empower the user

slide-12
SLIDE 12

12

22

The

Challenge

23

Make

6,998,329,787

a small number

slide-13
SLIDE 13

13

24

Why

is this important?

Nano

25

Bio Sensors

slide-14
SLIDE 14

14

26 27

Sensors and The Internet of Things

slide-15
SLIDE 15

15

2 3

Sensors are BIG

Sensors are unbounded

1

Sensors are indiscriminate Sensors are promiscuous

2 3

The Internet of Things is BIG

Everything is Connected

1

Everything is a Sensor Everything Communicates

slide-16
SLIDE 16

16

30

The inanimate is rapidly becoming sentient

Smarter Planet Cars drive themselves Machines know your needs

31

That’s the

Really Big Data

challenge of our future

slide-17
SLIDE 17

17

32

Technology is moving faster than government

can keep up

33

How can we successfully navigate and operate in this new world??

slide-18
SLIDE 18

18

2 3

Our Approach Know the Business Set an overarching Strategy Establish a Framework for execution Fund and Implement with Intent

1 4 2 3

4 Big Bets

– Acquire, federate, and position for multiple constituencies to securely

  • exploit. Grow the haystack, magnify the needles.

Data

Big

Excellence

Operational

Serve CIA IC

by supporting the

Management

Talent

– Assume a leadership role in IC activities that matter to CIA – Build capabilities assuming they will be shared – Innovate infrastructure operations and provisioning, create an authoritative source on our asset base, and run IT like a business. – Focus on continuous learning and diversity of thought, experience, background

1 4

slide-19
SLIDE 19

19

2 3 5 Key Technology Enablers

– World-class abilities to discover patterns, correlate information, understand plans and intentions, and find and identify operational targets in a sea of data

Mission Analytics

Advanced

Widgets and Services

Enterprise

Security Service

as a

Data Harbor

Enterprise Data Management--the

– One environment, all data, protected and secure using common security services such as: ubiquitous encryption, enterprise authentication, audit, DRM, secure ID propagation, and Gold Version C&A. – A customizable, integrated and adaptive webtop that lets analysts, ops

  • fficers, and targeters to “have it their way”.

– An ultra-high performance data environment that enables CIA missions to acquire, federate, and position and securely exploit huge volumes data.

1 4

Cloud Computing

– Ruthlessly standardized, rigorously automated, dynamic and elastic commodity computing environment. Massive capacity ahead of demand. Speed for mission need.

5 2 3

Our Accelerated Technology Adoption Process

Discover the Opportunities (100) Evaluate claims versus Reality

(30)

Pilot with the Mission

(10)

Implement (5)

1 4

slide-20
SLIDE 20

20

Discover

Active External Engagement

VCs Commercial Labs Government Labs In-Q-Tel USG Contractors Tech Expo Showcase Mission Link Tech Connect IC Partners Other Agencies Universities Road Trips Contracts

Evaluate

Unclassified and Classified Evaluation Facilities

iLab—unclassified, lots of data, variable

hardware

Eval—high-side, on-desktop, real data, real

users, defined hardware

NEAT—contracting mechanism to bring in

capabilities from non-traditional vendors

slide-21
SLIDE 21

21

Pilot

Real Problems, Real Users, Focused Outcomes

I2—the original IC “Cloud” proof of concept pilot Mass Analytics Cloud (MAC)—high-side, big-

data, real problems

Training—Cloudera, Hadoop, Developing for the

Cloud

Road Trips—expose the pilot teams to best

practices across sectors

Implement

Becoming part of our DNA

It’s not just about Technology

  • People and skills
  • Architecture
  • Governance
  • Process
  • Ruthless Standardization
  • Com plete change in

Applications Developm ent— think sm all, think horizontal

  • Costing m odels
  • Contracting m odels
slide-22
SLIDE 22

22

42

Closing

Thoughts

Tectonic Technology Shifts

Traditional Processing

Data on SAN Move Data to Question Backup Vertical scaling Capacity after demand DR Size to peak load Tape SAN Disk RAM limited

Mass Analytics/ Big Data

Data at processor Move Question to Data Replication management Horizontal scaling Capacity ahead of demand COOP Dynamic/ elastic provisioning SAN Disk SSD Peta-scale RAM

It’s all about SPEED! Latency breeds contempt!!

slide-23
SLIDE 23

23

A Few Hard Problems

  • Pattern Discovery
  • Correlation not Search—people, events, dates,

locations, …

  • Boolean is broken
  • “Curiosity” Layer
  • Peta-scale in memory architectures
  • Continuous, recursive, peta-scale recomputation
  • Cloud encryption—key management
  • Secure computing—assurance end-to-end
  • Secure mobility

Challenges Ahead

  • It’s all about speed, latency breeds contempt
  • Build a continuous learning organization
  • Embrace continuous change
  • Agility--become an “Ahead of” organization
  • Software licensing—metered use, not ELAs