Building a Science of Open Source Analysis: Addressing the need for - - PowerPoint PPT Presentation

Building a Science of Open Source Analysis:

Addressing the need for real tradecraft using open source analysis

Current Context of Intelligence Analysis

• Customers and consumers demanding greater understanding of

political and national security contexts

• Demand for more and BETTER intelligence
• Customers: Greater fidelity on issues of the day
• Consumers: Just In Time products and services to help shape policy and

formulate action

• More visibility into quality of products and services they receive, including

precisely how analysts come to their conclusions

• Challenge for LAS: Identify ways to make analysts better by helping

them create more and better intelligence, in compliance with ICD 203 and the requirements of ODNI Analytic Integrity and Standards group

Before we begin – Words and how we will use them…

Term Our Operational Definition Science Science is more than the mechanics of careful observation, creating hypotheses, testing those hypotheses against data, replicating and validating those findings and/or reformulating hypotheses in the pursuit of understanding some phenomena. Science is also the mechanism of organizing a corpus of knowledge and turning a community of interest into a community of practice. Science is the the common language and formal communication that results in knowledge sharing within those communities of practice. Science is also an agreed standard against which practitioners and their findings are judged and incorporated into the corpus of knowledge. SOSA Science of Open Source Analysis

Term Our Operational Definition OSINT “Publicly available information appearing in print or electronic form including radio, television, newspapers, journals, the Internet, commercial databases, and videos, graphics, and drawings. While open-source collection responsibilities are broadly distributed through the IC, the major collectors are the DNI's Open Source Center (OSC) and the National Air and Space Intelligence Center (NASIC).”* Open Source Analysis Using open source data (including OSINT) to inform all source and single source analysis True Factually Accurate Confidence in Source Degree to which source is legitimate, reliable Confidence in Data Degree to which data are true, rigorously determined Confidence in Judgment Degree to which analytic findings are likely, based on examination of evidence trail; logical flow of thought processes

4

A few more words…

*As defined by ODNI at http://www.dni.gov/index.php/about/faq?start=2, retrieved 2/15/16

... and Intelligence Collection Disciplines*

Discipline Acronym Type of Data Collected Responsible Agency

Signals Intelligence SIGINT Derived from signal intercepts comprising - individually or combined: communications intelligence (COMINT), electronic intelligence (ELINT) and foreign instrumentation signals intelligence (FISINT). National Security Agency (NSA) Imagery Intelligence IMINT Representations of objects reproduced electronically or by optical means on film, electronic display devices, or other media. Can be derived from visual photography, radar sensors, and electro-optics. National Geospatial-Intelligence Agency (NGA) Measure- ment and Signature intelligence MASINT Technically derived data other than imagery and SIGINT. Locates, identifies, or describes distinctive characteristics of targets. It employs disciplines including nuclear, optical, radio frequency, acoustics, seismic, and materials sciences. Defense Intelligence Agency (DIA), Directorate for MASINT and Technical Collection (DT) Human Intelligence HUMINT Derived from human sources. Collection includes clandestine acquisition of photography, documents, and other material; overt collection by personnel in diplomatic and consular posts; debriefing

• f foreign nationals and US citizens who travel abroad; and official

contacts with foreign governments. Central Intelligence Agency (CIA), Directorate of Operations (DO) Open Source Intelligence OSINT Publicly available information -print or electronic -including radio, television, newspapers, journals, the Internet, commercial databases, and videos, graphics, and drawings. DNI Open Source Enterprise (OSE) National Air and Space Intelligence Center (NASIC). Geospatial Intelligence GEOINT Analysis and visual representation of security-related activities on the

• earth. Integration of imagery, IMINT, and geospatial information.

National Geospatial-Intelligence Agency (NGA)

*As defined by ODNI (http://www.dni.gov/index.php/about/faq?start=2), retrieved 2/15/16

5

Overview of Major Elements of Intelligence Analysis

Political Stability International Ambition Economic Stability Societal Issues Crime and Trafficking Political Stability International Ambition Economic Stability Societal Issues Crime and Trafficking Explain specific event Explain specific event Warn of situation change Warn of situation change Provide Situational awareness Provide Situational awareness

Intelligence Questions NIPF/PIP

Analysis

MASINT OSINT IMINT GEOINT HUMINT SIGINT Alternative Media Think Tanks Academia NGOs New/ Emerging Media Business Intelligence Intelligence Product

Collected Sources Collectible Sources

OSA Methods Can Vary Based on National Priorities and/or Intelligence Questions

6

OSA

A View of the Intelligence Analysis Production Process

7

Understanding analytic production processes is necessary but insufficient for understanding what happens ‘between the ears’ of the analyst. That is STILL not well understood.

Taking an illustrative look in the ”black box”

A good analyst is an accomplished learner.

Evaluation Synthesis Analysis Application Comprehension Knowledge

Conceptual Framework: Bloom’s Taxonomy of Learning Objectives (1956)

Understanding Self – The Analyst’s Lens

• Knowledge, Skills, Ability
• Education
• Experience
• Type of Analyst
• Agency
• On the job learning
• Professional Network
• National Security Worldview – purpose of intelligence and its relation

to policymakers

1.What do I need to know to answer this question? Iran What constitutes violating? What is in the agreement? Nuclear 2.What do I need to know about these topics to answer the question? Politics Culture Religion History Finance Deception Using loopholes Non-Compliance Is Iran violating the nuclear agreement?

Intelligence Question Question Decomposition

Military Leadership Economics Business International Policy Agreement Monitoring Enforcement Manufacturing Basic Science & Technology Weapons

• 4. What information do I need?

MASINT OSINT IMINT GEOINT HUMINT SIGINT Alternative Media Think Tanks Academia NGOs New/ Emerging Media Business Intelligence

• 3. What knowledge/skills do I bring

to this task vice the knowledge/skills that are needed?

Personal Assessment

Subject Matter Experts?

• 7. Are the data factually

accurate?

• 8. How confident am I that

the data sources are reliable?

Judgment Resource Identification

• 5. What information do I have?
• 6. What information don’t I

have?

• 9. What do I do with the data?

How do I decide - What methods? What tools? What techniques?

Conundrum

• 10. What are the results?

What do they mean?

• 11. How do I know the

results are reliable?

Evaluation

• 12. How do I tell

the story with the findings I have?

• 13. How do I tell the

story to address the audience who asked the question?

Synthesis

There’s a lot of work for us to do.

Our Research Agenda at LAS

Challenge: Identify ways to make analysts better by helping them create more and better intelligence Investigate what goes on ‘between the ears’ of analysts

• Analyst Assessment Support Program

– Using ”Expert in a Box” approach

• Building a Science of Open Source Analysis

14

Analyst Assessment Support Program

• Previous research
• Developed evidence-based approaches for social science problem sets
• Problem recognition and problem solving in complex environments
• Created method that functions as decision-aide for analysts by providing expert information

based on evidence from empirical research

• Includes guided approach to analysis, source citation collection, reporting function
• Trained, tested method for reliability (inter-rater) and validity (content)
• Successfully implemented for use with a number of different environments, requiring only

minimal adjustments

• Developed decision aids for health science problem sets
• Conducted numerous task and cognitive task analyses
• LAS research
• Bring expert aspect (research findings) in method up to date, redesign tool
• Identify how this method can be improved by or act as a complement to LAS projects, such as

Open KE, FSP, others

• Test the updated product

15

Using established academic disciplines to inform SOSA

• Previous research
• Ethnography of intelligence analysts
• Visualizing PMESII
• Socio-Cultural Influences
• Multi-layered/multi-disciplinary analytic methods
• Structured models, approaches, and techniques
• LAS research
• Supply Chain Analysis and Intelligence Analysis (DO5)
• Conduct a Use Case to identify the interaction of supply chain analysis and
• ther methods to analyze adversarial supply chains re: nuclear proliferation

16

Enriching a Community of Interest for OSA at LAS

• LAS Research
• Understand the development of recent academic disciplines
• What elements establish them as ‘official’
• How does a new discipline gain acceptance and grow?
• Outline the state of OSA type methods
• Identify gaps and needs
• Identify implications for scientifically sound OSA methods
• Facilitate dialogue on the work at LAS and the furtherance of this COI in the

interest of developing SOSA through quarterly workshops that feature practitioners from across the IC.

17

Questions?

18