Analytic Component System (ACS) Matt Schmidt and Andrew Crerar - - PowerPoint PPT Presentation

Analytic Component System (ACS)

Matt Schmidt and Andrew Crerar Presentation to LAS Weekly Research Meeting 13 April 2015

Collaborators

• Tonya Adelsperger (LAS-G)
• Joe Aguayo (LAS-G)
• Edward Browne (LAS-G)
• Samuel Christie (NCSU)
• Bill Elm (RCS)
• Rob Johnston (Johnston Analytics)
• Judy Johnston (Johnston Analytics)
• James Kaiser (LAS-G)
• Munindar Singh (NCSU)
• John Slankas (LAS-I2I)
• Colleen Stacy (LAS-G)
• Pankaj Telang (LAS-I2I)
• Eli Typhina (NCSU)
• Kathleen Vogel (NCSU)
• Lori Wachter (LAS-G)
• Alyson Wilson (NCSU)
• Mark Wilson (NCSU)
• Jorge Zuniga (RCS)

Outline

• Motivation and Overview

Tradecraft Storyboard Analytic Component Interface

Background and Motivation Approach Initial Results Benefits

Analytic Computing Environment

Background Demo #1: Execution of Composite Workflow Demo #2: Inspection of Workflow Demo #3: Modification of Workflow

Future Work and Summary

Goal of ACS

Provide analysts a library of analytic components that can be combined to create analytic workflows Provide analysts a library of analytic components that can be combined to create analytic workflows

Problem Problem

• Analysts have access to a wide variety of

analytic tools and methods

• Applying these tools and methods to a problem

can require a great deal of effort to:

• 1. Learn about the tool/method and;
• 2. Adapt the analyst’s workflow to

incorporate it

Desired Outcome Desired Outcome

An organized set of analytic tools and methods (components) that an analyst could use to construct their desired analytic workflow

Technical Challenge: Component Integration

The costs of integrating different components into a single workflow can overwhelm any benefits The costs of integrating different components into a single workflow can overwhelm any benefits

• Analytic components consume and produce

certain types of information

• Integration occurs when one component

consumes information produced by another

• Integration costs occur when there is a need to

translate the information before consumption

• Time
• Cognitive Load
• Mis-Communication
• Integration costs can arise at many different

places

• Component to Component
• Component to Human
• Human to Human

Integration Costs Integration Costs

VS

Technical Challenge: Workflows are not Applications

Analytic workflows do not look like procedural applications and therefore require a different computation environment Analytic workflows do not look like procedural applications and therefore require a different computation environment

• The analytic components used are not always known before execution
• Components can be computational, manual, or interactive
• Analysts may want to revisit and modify previously executed workflows

Technical Approach

The ACS effort consists of three related development efforts The ACS effort consists of three related development efforts

Goal Goal

Component Interface Component Interface Component Library Component Library Computing Environment Computing Environment

Enable compatible component interfaces Enable execution of workflows composed of analytic components Develop library of novel and existing analytic components Define standard analytic constructs Approach Approach Develop service-oriented computing platform Decompose and wrap components using ACI to execute in the ACE

Outline

Motivation and Overview

• Tradecraft Storyboard

Analytic Component Interface

Background and Motivation Approach Initial Results Benefits

Analytic Computing Environment

Background Demo #1: Execution of Composite Workflow Demo #2: Inspection of Workflow Demo #3: Modification of Workflow

Future Work and Summary

ACS Technology-Enabled Tradecraft Storyboard

Task of the analyst is to report on likely targets of the Anonymous hacktivist group Task of the analyst is to report on likely targets of the Anonymous hacktivist group

Backstory Backstory

• Single Analyst
• Components in ACS plus front-end
• Multiple open data sources
• Collected reporting

Question Question Who is the most likely next target of the Anonymous hacktivist group? Resources Resources

Storyboard Scene 1: Question to Data Source

Different components in ACS use different combinations of computation and manual processes to generate constructs Different components in ACS use different combinations of computation and manual processes to generate constructs

Scene Description Scene Description

Action Action Analyst selects a background report on Anonymous Entities Involved Entities Involved

• Analyst
• Analytic Component from ACL
• Produces Data Source given a

Question

• ACE
• Sends question to component and

sends data source to next component

• Background Report

Storyboard Scene 2: Data to Indicators

The ACI enables the data inspection component to generate constructs understood by question resolution components The ACI enables the data inspection component to generate constructs understood by question resolution components

Scene Description Scene Description

Actions Actions

• Analyst reads line in background report

saying Anonymous members often have strong libertarian beliefs

• Analyst creates an indicator question:

Who are the most visible anti-libertarian actors right now? Entities Involved Entities Involved

• Analyst
• ACL Components
• ACI Constructs
• Background report

Storyboard Scene 3: Indicators to Ranking

The ACL and ACE enables a question resolution workflow to be automatically composed and executed The ACL and ACE enables a question resolution workflow to be automatically composed and executed

Scene Description Scene Description

Action Action Analyst inspects automated workflow to see how list of anti-libertarian actors was produced Entities Involved Entities Involved

• Analyst
• ACL enables automated construction of

workflow to produce ranked list

• ACE logs all constructs produced in

workflow

• Front-end allows analyst to inspect

Storyboard Scene 4: Manually introduce new data source

The ACE enables analyst modification and control of an automated piece of the analytic workflow The ACE enables analyst modification and control of an automated piece of the analytic workflow

Scene Description Scene Description

Action Action

• Analyst adds a missing data source to the

workflow

• Workflow automatically runs

Entities Involved Entities Involved

• Analyst
• ACE enables workflow modification
• Front-end enables inspection of workflow

and addition of constructs

Storyboard Scene 5: New Hypothesis Alert

The ACS along with User Experience efforts supports transition of analysis effort to monitoring effort The ACS along with User Experience efforts supports transition of analysis effort to monitoring effort

Scene Description Scene Description

Action Action Analyst is alerted to new anti-libertarian actor because of a new data being added to the data source Entities Involved Entities Involved

• Front-end determines how to alert user
• ACE enables streaming ingest and

analysis of data using workflows originally for historical data

• ACL consists of components that can be

run in a streaming fashion

• ACI enables the use of either batch or

streaming methods of producing construct

Storyboard Scene 6: Workflow Capture and Analysis

The constructs logged by the ACS provide a rich source of instrumentation data for reporting and analysis The constructs logged by the ACS provide a rich source of instrumentation data for reporting and analysis

Scene Description Scene Description

Action Action

• Draft report generated automatically
• Workflow becomes part of report
• Analysis of workflow can provide estimate
• f analytic rigor

Entities Involved Entities Involved

• Elastic Search logs all analytic constructs
• Auto-report generation uses constructs to

produce draft report

• Analytic rigor maps production of

constructs to metrics of analytic rigor

Storyboard Slide 7: Develop new components

Previously captured workflows provide training data for the development of new components and workflow recommenders Previously captured workflows provide training data for the development of new components and workflow recommenders

Scene Description Scene Description

Action Action Researchers use logged pairings of constructs to develop automated method of generating the constructs Entities Involved Entities Involved

• Researchers
• Logged ACI constructs in Elastic Search
• Analyst production of constructs

Related Efforts

Multiple LAS efforts can benefit from or contribute to the ACS Multiple LAS efforts can benefit from or contribute to the ACS

Instrumentation Instrumentation ACI constructs define the resolution of workflow logs Analytic Rigor Analytic Rigor ACI constructs provide data from which to derive metrics on analytic rigor User Experience User Experience User Experience will be required enable analyst to observe and control workflows executed in the ACS Recommender Systems Recommender Systems Logged ACI constructs provide rich training data source for recommender system efforts Analytic Component Development Analytic Component Development ACS makes explicit the needs for new components for existing components that are inefficient Auto-Report Generation Auto-Report Generation Logged ACI construct should provide enough information to create a draft report automatically

Outline

Motivation and Overview Tradecraft Storyboard

• Analytic Component Interface
• Background and Motivation

Approach Initial Results Benefits

Analytic Computing Environment

Background Demo #1: Execution of Composite Workflow Demo #2: Inspection of Workflow Demo #3: Modification of Workflow

Future Work and Summary

Analytic Constructs

An analytic construct is a piece of information either required or provided by an analytic component An analytic construct is a piece of information either required or provided by an analytic component

• Analytic component produce one type of information using information

that has been provided

• Examples
• An analysts that provides a hypothesis for a given question
• A clustering algorithm that provides a mapping to the different cluster

IDs for all of the provided data points

• The different types of information either provided or produced by a

component are referred to as analytic constructs

Interfaces

In a modular workflow, analytic components are represented by their interfaces In a modular workflow, analytic components are represented by their interfaces

Interfaces Interfaces Impact Impact Benefits Benefits

• Makes modular components easier to reuse

because the client only needs to understand the interface

• Enables modular components to be

interchangeable as long as the share the same interface

• The method by which the component produces

the constructs has been abstracted

• From a systems perspective, the component

becomes a black box A component’s interface is the set of constructs that are either provided or required by the component Examples Examples

• Hypothesis Generation Component
• Required: Question
• Provides: Hypothesis
• Clustering Component
• Required: Vectorized Data Points
• Provides: Mapping of Data Points to

Clusters

Compatible Interfaces

Integrating components with incompatible interfaces leads to potentially overwhelming integration costs Integrating components with incompatible interfaces leads to potentially overwhelming integration costs

• Integrating two components means

generating the constructs required by one component from the constructs provided by another

• Components are compatible if the

constructs that are required by one component are provided by another

Integration Integration

• If two components are not compatible, then integrating those two components will require

an integration cost of transforming the provided constructs into the required constructs

• Paying this integration cost every time the workflow is executed can become an
• verwhelming burden

Compatibility and Integration Costs Compatibility and Integration Costs

Goal of the ACI

Ensure compatible component interfaces by defining a standard set of analytic constructs Ensure compatible component interfaces by defining a standard set of analytic constructs

Outline

Motivation and Overview Tradecraft Storyboard

• Analytic Component Interface

Background and Motivation

• Approach

Initial Results Benefits

Analytic Computing Environment

Background Demo #1: Execution of Composite Workflow Demo #2: Inspection of Workflow Demo #3: Modification of Workflow

Future Work and Summary

Highest-Level Abstraction of Analysts’ Task

Analysts provide information about the world as it was (reflect), as it is (observe), or as it will be (imagine) Analysts provide information about the world as it was (reflect), as it is (observe), or as it will be (imagine) Data Repositories and Feeds Analyst Information Request Information Request

Did it rain yesterday? Query yesterday.rainfall yesterday.rainfall = 0 “No, it did not rain yesterday”

Inference Task

Working Hypothesis: Inference from data, observations, and predictions is the primary task of almost all analysis Working Hypothesis: Inference from data, observations, and predictions is the primary task of almost all analysis Data Repositories and Feeds Analyst Information Request Information Request

Will it rain tomorrow? Query ‘radar’ Query ‘pressure’ “Yes, it will likely rain tomorrow” Query ‘humidity’ Query ‘temperature’

Other Domains of Inference

• An abstract analyst inference process assesses evidence to create

assessments of hypotheses likelihoods

• This inference process is conceptually similar to statistical and

computational inference processes Hypothesis: Constructs in statistical and computational inference will have analogs in intelligence analysis workflows Hypothesis: Constructs in statistical and computational inference will have analogs in intelligence analysis workflows

Outline

Motivation and Overview Tradecraft Storyboard

• Analytic Component Interface

Background and Motivation Approach

• Initial Results

Benefits

Analytic Computing Environment

Background Demo #1: Execution of Composite Workflow Demo #2: Inspection of Workflow Demo #3: Modification of Workflow

Future Work and Summary

Current Hypothesis for Constructs

This construct set is only a hypothesis and likely to change This construct set is only a hypothesis and likely to change

Construct Type Definition

Proposition Proposition A statement that can be either true or false Truth Function Whether or not a proposition is consistent with the state of the world(s) being analyzed Question Request A request for propositions and an assessment of their truth Assertion Assertion A proposition that is true for all worlds being analyzed Data Assertion Assertions that serve as the foundation for the analysis process Query Request A request for data that satisfy a certain criteria Validity Function Whether or not the proposition is of the type requested by the question Hypothesis Proposition A proposition that is valid for the given question Decision Assertion An asserted hypothesis of Q Relationship Function The degree to which the truth of P1 is affected by asserting the proposition(s) P2 Corollary Proposition A proposition that is strongly supported by the given proposition(s) Evidence Assertion An assertion that is relevant to the given hypothesis(es) or question(s) Indicator Request A question whose hypotheses can serve as evidence to the given question Likelihood Function The truth of the hypothesis H when the evidence(s) E is asserted Assessment Function The relative likelihoods of the hypotheses of a question Q given the same set of evidence E Model Function The set of all relationships between hypotheses of the set of questions Q Scenario Assertion A set of assertions that generated according to model M

Current Challenge to Specifying Constructs

Need to find a mechanism by which to describe and validate constructs across different fields Need to find a mechanism by which to describe and validate constructs across different fields

Description ACI Construct Statistical Analog Classification Analog

The piece of information to be inferred Question Response Variable Class Variable All potential values for the information requested Hypothesis Set Response Variable Domain Label Set The information determined to be most likely Decision / Assessment Most Likely Estimate Assignment

Outline

Motivation and Overview Tradecraft Storyboard

• Analytic Component Interface

Background and Motivation Approach Initial Results

• Benefits

Analytic Computing Environment

Background Demo #1: Execution of Composite Workflow Demo #2: Inspection of Workflow Demo #3: Modification of Workflow

Future Work and Summary

Benefit of Having Standard Constructs

Workflows can be described in terms of the constructs that will be produced instead of the components that will be used Workflows can be described in terms of the constructs that will be produced instead of the components that will be used

Benefit of Having Standard Constructs

Workflows can then be used to determine what components should be integrated into the workflow Workflows can then be used to determine what components should be integrated into the workflow

Benefit of Having Standard Constructs

Recursive workflows arise naturally Recursive workflows arise naturally

Outline

Motivation and Overview Tradecraft Storyboard Analytic Component Interface

Background and Motivation Approach Initial Results Benefits

• Analytic Computing Environment
• Background

Demo #1: Execution of Composite Workflow Demo #2: Inspection of Workflow Demo #3: Modification of Workflow

Future Work and Summary

Components and Services

Analytic components are the methods used to enact the service of producing analytic constructs Analytic components are the methods used to enact the service of producing analytic constructs

Example Example Using Services Using Services

• A service is a task being performed for others
• A component is the method by which the task

is performed Services can be composed in different ways to accomplish larger tasks Services Services Components Components

Atomic Services

The enactment of atomic services only requires executing the underlying component The enactment of atomic services only requires executing the underlying component

Composite Services

The ACE enables the enactment of composite services The ACE enables the enactment of composite services

Outline

Motivation and Overview Tradecraft Storyboard Analytic Component Interface

Background and Motivation Approach Initial Results Benefits

• Analytic Computing Environment

Background

• Demo #1: Execution of Composite Workflow

Demo #2: Inspection of Workflow Demo #3: Modification of Workflow

Future Work and Summary

Application as a Composite Service

Demo #1 is an enactment of a composite service built to mirror the functionality of a previous monolithic application Demo #1 is an enactment of a composite service built to mirror the functionality of a previous monolithic application

ACE: Providing Services

This pattern exists for each construct in the ACI This pattern exists for each construct in the ACI

• 1. A request message is received by the

manager

• 2. The manager identifies a component in

the component repository that can handle the request

• 3. The manager instantiates the component

using a thread from the thread pool

• 4. The handler executes
• 5. Upon completion, the manager runs the

callback function to return the construct produced by the handler

• 6. The manager publishes the construct on

the response queue

ACE: Requesting Sub-Services

Temporary message queues enable handlers to receive constructs from other services Temporary message queues enable handlers to receive constructs from other services

• 1. A request message is received by manager A
• 2. Manager A instantiates handler A
• 3. Handler A begins execution and sends request

message to Manager B

• 4. Manager B instantiates Handler B
• 5. Handler B executes and produces construct B
• 6. Manager B returns construct B on Handler A’s

temporary queue

• 7. Handler A completes execution and produces

construct A

• 8. Manager A responds with construct A

ACE: Choreographing Services

Changing the response queues allows composite services without a centralized controller Changing the response queues allows composite services without a centralized controller

• 1. A request message is received by

Manager A

• 2. Manager A instantiates Handler A
• 3. Handler A completes and produces

construct A

• 4. Manager A puts message on Manager

B’s request queue with construct A

• 5. Manager B instantiates Handler B
• 6. Handler B completes and produces

construct B

• 7. Manager B puts message on Initiator’s

response queue

Outline

Motivation and Overview Tradecraft Storyboard Analytic Component Interface

Background and Motivation Approach Initial Results Benefits

• Analytic Computing Environment

Background Demo #1: Execution of Composite Workflow

• Demo #2: Inspection of Workflow

Demo #3: Modification of Workflow

Future Work and Summary

Workflow Observability

Logging the analytic components produced by an enactment allows analyst to inspect automated portions of their workflow Logging the analytic components produced by an enactment allows analyst to inspect automated portions of their workflow

• All messages are logged in an Elastic Search DB
• Through careful querying we can reproduce what messages were

generated during the workflow and by which component

• It does not matter how the component produced the message

Outline

Motivation and Overview Tradecraft Storyboard Analytic Component Interface

Background and Motivation Approach Initial Results Benefits

• Analytic Computing Environment

Background Demo #1: Execution of Composite Workflow Demo #2: Observability of Workflow

• Demo #3: Modification of Workflow

Future Work and Summary

Sensitivity Analysis

Analysts can observe the sensitivity of the analytic product to modifications of intermediate results Analysts can observe the sensitivity of the analytic product to modifications of intermediate results

• All services are designed to be independent of other services
• The only thing that ties the services together in a workflow is the

messages sent between them

• Modifying the messages can modify the workflow and enable the analyst

to restart the workflow without re-executing unnecessary components

Outline

Motivation and Overview Tradecraft Storyboard Analytic Component Interface

Background and Motivation Approach Initial Results Benefits

Analytic Computing Environment

Background Demo #1: Execution of Composite Workflow Demo #2: Inspection of Workflow Demo #3: Modification of Workflow

• Future Work and Summary

Ongoing Work

Ongoing work seeks to enable wider range of workflows supported by ACS Ongoing work seeks to enable wider range of workflows supported by ACS

• (ACI) – Refine and validate constructs
• Need more consensus on what the standard constructs should be
• (ACE) – Incorporation of a planner service
• Planner service enables user or algorithm to compose a service as

determined by the problem

• (ACS) – Demonstrate interactive composition and execution of workflows
• Interactive composition and execution of components is necessary

for ACS to support analyst workflows

Future Work

Future work will focus on enabling ACS to better support existing tools and tradecraft Future work will focus on enabling ACS to better support existing tools and tradecraft

• (ACL) – Incorporate existing analytic components
• Need to incorporate large set of existing manual and automated

analytic techniques

• (ACS) – Integrate with a User Experience framework
• Need to determine how ACS supports an analyst’s user experience
• (ACS) – Demonstrate on mission exemplar
• Need to determine how ACS affects mission tradecraft

Summary

Have made initial progress on a foundational development effort for how analytic workflows are composed and executed Have made initial progress on a foundational development effort for how analytic workflows are composed and executed

• Proposed initial set of standard analytic constructs to decompose and

modularize analytic components

• Developed prototype service-oriented computing environment to enable

analytic components to be executed as services

• Demonstrated how an analytic tool can be transitioned to the ACS and

the benefits that come from doing so

Questions