SLIDE 1
경희대학교 이 승 룡
/
Agenda
- Evolution of Mining Minds
- Mining Minds V2.5 Service Scenario
- Mining Minds V2.5 Architecture
- Platform Operations, Uniqueness and Contribution
- Mining Minds V2.5 Microdemos
- Future Plan
- Conclusions
May 2016 - - PowerPoint PPT Presentation
May 2016 - - PowerPoint PPT Presentation
May 2016 Agenda 2 / Evolution of Mining Minds Mining Minds V2.5 Service Scenario Mining Minds V2.5 Architecture Platform
SLIDE 2
SLIDE 3
Evolution of Mining Minds
3
SLIDE 4
/
Mining Minds Platform Overview
4
Personalized well-being and health-care Platform
Big data Storage Real time Analytics Domain Expert End user
SERVICES SERVICES
Privacy & Security Behavior Quantification Induced Habituation Personalized Recommendation Knowledge creation Lifelog LLM
Mining Minds Platform Mining Minds Inputs
SLIDE 5
/
History of Mining Minds
5
MM V1.0 Dec 2014
Weight Management App Management Tool Personalized Lifestyle Coaching App Rule Authoring Tool Behavior Inspection Tool Real time activities simulation tool Data driven knowledge acquisition tool
MM V1.5 May 2015
Personalized Lifestyle Coaching App Rule Authoring Tool Behavior Inspection Tool
MM V2.0 Dec 2015
SLIDE 6
/
History of Mining Minds
6
Supporting Layer
Visualization Security and Privacy User Interface
Service API
Credentials and Authorized StorageService Curation Layer
Information Curation Layer Low Level Context-Awareness
Activity Recognizer
Data Curation Layer
Data Acquisition
Low-level Context Preserver Data Curation Socket Server Data Curation Client Service Curation Service ClientLife-log Representation and Mapping
CRUD Operations DCL Object ModelSensory Data Processing and Life-log Persistence Microsoft Azure Public Cloud
Data Curation Service Client Data Curation Service Client UI Container Send Recv. Msg Buf. Life-log MonitoringData Curation Restful Service
Data Curation Service Contract Data Curation Service Client Access Validator Oblivious Evaluator Information Modification and Reply DetectionKnowledge Curation Layer
Knowledgebase Knowledge Acquisition Editor Rule Builder Descriptive Analytics Query Creation Interface Trend Analyzer
Visualization Enabler Service Curation Restful Service Service Curation Service ContractRecommendation Manager
Recommendation Builder Knowledge Interface Recommendation Interpretor Context Interpreter Reasoner Explanation ManagerService Orchastrator
Event Handler Input / Output Adapter Intermediate Database Life-log User-Profiles Big Data Processing ClientLife-log Monitor
Event Generator Object to Relational MappingMM V1.0 Dec 2014
Primitive Data Curation (Representation and Mapping) Smartphone-based Activity Recognition (5 activities) Prebuilt Recommendations Generation (static) Static Admin Tool (raw data and activities)
UCLab Private Cloud Personalized Big Data HDFS HDFS HDFS Big Data Socket Server Send / Recv. Messages Message Buffer fica filMM V1.5 May 2015
Big Data (only persistence) & Lifelog (representation and mapping) Smartphone + Smartwatch-based Activity Recognition (8 activities) Rule-authoring tool (no executable knowledge) Situation-triggered Personalized Recommendations (local knowledge) Expert Inspection Tool (single/multi-user stats)
Supporting Layer
Visualization Security and Privacy User Interface
Service API
Encrypted Authorized Storage
Service Curation Layer
Recommendation Manager
Information Curation Layer Low Level Context-Awareness
Activity Recognizer
Data Curation Layer
Parameter Classification Data Acquisition Data Preserver Data Curation Webservice Reasoning and Prediction Data Curation Service Contract Intermediate Database Life-log User-Profiles Data Curation Webservice Client Data Curation Service Client Data Representation and Mapping CRUD Operations DCL Object Model Object to Relational Mapping
Sensory Data Processing and Persistence
Service Curation Webservice
Microsoft Azure Public Cloud
Data Curation Service Client Content Representation Data Model Rendering Data Curation Service Client Rule-based Reasoning Knowledge-base Service Curation Service Contract UI Container Explanation Generator Recommendation Generator SLIDE 7
/
History of Mining Minds
7
Big Data (only persistence) & Lifelog (representation and mapping) Smartphone + Smartwatch-based Activity Recognition (8 activities) Rule-authoring tool (no executable knowledge) Situation-triggered Personalized Recommendations (local knowledge) Expert Inspection Tool (single/multi-user stats) Real time activities monitoring (low level and corresponding high level) Data driven knowledge acquisition (utilizing big data approach)
MM V2.0 Dec 2015
SLIDE 8
/
Mining Minds Evolution
8
MM V 1.0
MM V 1.5 MM V 2.0 MM V 2.5
Physical Activities
End-user app view
PHYSICAL ACTIVITIES PHYSICAL ACTIVITIES PHYSICAL ACTIVITIES PHYSICAL ACTIVITIES
user app view Expert view
Knowledge base
RULE AUTHORING TOOL
Jan 2015 May 2015 Dec 2015 May 2016
user app view Expert view
UI/UX SNS Trends
NUTRITION
+
Expert view user app view
Physical Activities Physical Activities Physical Activities
SLIDE 9
Mining Minds V2.5 Service Scenario
9
SLIDE 10
/
Personalized well-being and health-care Platform
Overall Service Scenario for MM2.5
Expert View Inputs
Physical Sensors Logical Sensors User
Domain Expert Big data Storage lifelog Intermediate Database Kinect Camera Smart Phone Smart Watch
10
Expert End user Twitter Facebook
COLD SERVICES WARM SERVICES
MM V3.0 Inputs Personalized Recommendation Human Behavior Quantification Expert Recommendation Privacy & Security Induced Habituation
SLIDE 11
/
Mining Minds Platform and Services
11
INPUTS
Knowledge (physical activities & nutrition) Direct Recommendations
Expert Input
Tagging Taking Picture Feedback
- Expert direct
recommendations and education
- New knowledge
evolution
User Input Service Requirements
Expert-based Services User Behavior Quantification Personalized Services Induced Habituation
- Physical
activities analysis
- Diet Control
and Food analysis
- Lifeline
- User aware
- Situation
aware
- Context
Aware
- Educational
facts
- Gamification
- Multimedia
contents
Platform Requirement
Multimodal Sensory BigData Processing Context Determination Knowledge Acquisition & Maintenance Personalized Recommendations
- Real Time
- Volume,
velocity and variety
- Life-log
representation & Monitoring
- Context
fusioning
- Context
- ntology
- Semantic
Reasoing
- Data & Expert
Driven Knowledge
- Validation &
Verification
- Knowledge
Engineering Toolkit
- Rule based
Reasoning
- Situation
Evaluation
- Cross-domain
recommendation
- UI/UX
Knowledge Authoring
- Descriptive
Analytics
- Security and
Privacy
UI/UX
End User Services Expert Services
- Knowledge Authoring
Tool
- Knowledge creation and
maintenance
- User Behavior
Inspection Tool
- Visualization
- UI/UX Authoring Tool
- UI adaption rules
- Personalized
recommendations
- Expert
Recommendations
- User Behavior
Quantification
- Healthy habit
induction
- Security and
privacy
Accelerometer Gyro Audio Video (camera) GPS SNS
Sensors
WARM SERVICES
Warm Services Cold Services
Security and Privacy Aware Services
- Anonymization
- Oblivious
Matching
- Encryptions
SLIDE 12
/
Cold Service Scenario-[1/2]
12
Activity data User Information Information Curation Data Curation Nutrition data
Big data Storage LLM Data synchronization Intermediate Database
Mining Minds Platform
Preferences Demographics Risk Factors Feedback High Level Context OfficeWork, Amusement, Sleeping, Gardening, Inactivity, HouseWork, Exercising, Commuting, Snacks, Nuts, Vegetables, SeaFoods, Fruit, NoHLC Low Level Context Eating, LyingDown, Running, Sitting, Standing, Stretching, Sweeping, Walking
SLIDE 13
/
Cold Service Scenario-[2/2]
13
Information Curation Data Curation
Big data Storage LLM Data synchronization Intermediate Database
Mining Minds Platform
High Level Context Low Level Context
Descriptive Analytics SNS
Service Curation
Service Orchestrator Recommendation Builder Recommendation Interpreter
Supporting Layer
SLIDE 14
/
Warm Service Scenario-[1/4]
14
Lifelog
Intermediate Database Domain Expert
Expert View Knowledge Curation
Knowledge base Rule Builder Domain Model Manager Rule Validator
Knowledge Authoring Tool
Knowledge Authoring tool
Rule Condition Low cholesterol is recommended[Whole -wheat pasta with peas and spinach (Fiber enrich) ]and 20 minutes Brisk walk Recommendation
Save Rule
Weight Status Meal Time Food Intake Condition Key = = = Condition Key Is Situation! Over Weight Lunch Cholesterol Enrich Condition Key v v v
Rule Condition
Meal Time Food Intake Condition Key = = Condition Key Dinner Fiber Enrich Condition Key
Mining Minds Platform Big data Storage
Data Curation
Big-data Descriptive Analytics
LLM
SLIDE 15
/
Warm Service Scenario-[2/4]
15
Information Curation Data Curation
Mining Minds Platform
High Level Context Low Level Context
Descriptive Analytics SNS
Service Curation
Service Orchestrator Recommendation Builder Recommendation Interpreter
Supporting Layer
Data synchronization
Activity data User Information Nutrition data
Preferences Demographics Risk Factors Feedback
Lifelog Intermediate Database Lifelog Monitoring
Physical Activity Monitoring Nutrition Monitoring Monitoring Situation
SLIDE 16
/
Warm Service Scenario-[3/4]
16
Supporting Layer Data Curation
Big data Storage Lifelog Data synchronization Intermediate Database
High Cholesterol Expert Feedback to User Sedentary Behavior
Mining Minds Platform
Holistic View
Visualization UI/UX
Knowledge Authoring Tool
Light food with good fiber amount is recommended for your health in dinner, as you have taken cholesterol enriched food in lunch and try to have 20 min brisk walk as you remained sedentary whole day.
SLIDE 17
/
Warm Service Scenario-[4/4]
17
Light food with good fiber amount is recommended for your health in dinner, as you have taken cholesterol enriched food in lunch and try to have 20 min brisk walk as you remained sedentary whole day. Expert View: Expert feedback to end user Send End User View: User get the expert recommendation
Personalized well-being and health-care Platform
Light food with good fiber amount is recommended for your health in dinner, as you have taken cholesterol enriched food in lunch and try to have 20 min brisk walk as you remained sedentary whole day. Expert Recommendation
SLIDE 18
18
Induced Habituation
Personalized Recommendation Human Behavior Quantification Expert Recommendation Privacy & Security Induced Habituation
SLIDE 19
/
Habituation-MMV2.5
19
Existing
well-being Platforms
Activity data User Information Nutrition data
Preferences Demographics Risk Factors Feedback
Even if you exercise each day, studies show that sitting for long periods will increase health risks.
Educational Facts
Personalized well-being and health-care Platform
SLIDE 20
/
Habituation- Example (Educational Facts & Rewards)
20
9:05am - 10:05 am 10:06 am 12:00 pm 3:17 pm 11:06 pm
You were sitting for the past 1 hour, please stretch your legs, arms and back
WORKING AT OFFICE GOING TO OFFICE AT NIGHT
1 2 3 4
11:07 am 11:50 am It has been 3 hours of sitting now, please walk for few minutes
Light food with good fiber amount is recommended for your health, as you have taken cholesterol enriched food items, and try to have 20 min brisk walk as you remained sedentary whole day. It has been 2 hours of sitting now, please walk for few minutes Sitting is the new smoking. It can increase your rate of contracting lung cancer more than 50%
Sitting is the new smoking. It can increase your rate
- f
contracting lung cancer more than 50%
4 3 2
AT RESTAURANT WORKING AT OFFICE
Light food with good fiber amount is recommended for your health, as you have taken cholesterol enriched food items, and try to …………………….. Done all the offers? There is still the Daily Bonus for you to collect!
+6
Credits
1
You were sitting during 1 hour, please stretch your legs, arms and back
SLIDE 21
21
Personalized Recommendations
Personalized Recommendation Human Behavior Quantification Expert Recommendation Privacy & Security Induced Habituation
SLIDE 22
/
Personalized Recommendation-MMV2.5
22
Personalized well-being and health-care Platform
User in meeting
x
Existing
well-being Platforms
Activity data User Information Nutrition data
Preferences Demographics Risk Factors Feedback
“You have been sitting for more than 3 hours now, please take a walk and refresh your mind for at least 5 minutes”
Context aware 1
Context Environ- mental Special Conditions
Personalization
Generate
Recommendations
SLIDE 23
/
Personalized Recommendation-Context aware
23
“you are recommended 5 minutes running to relax and improve your health ”
Office Location
In-Active
“You have been sitting for 30 min, you are recommended 5 minutes running to relax and improve your health ” “You have been sitting for 30 min, please take a walk and refresh your mind for at least 10 minutes” You have been sitting for 30 min, please take a walk and refresh your mind for at least 10 minutes
Office Work
SLIDE 24
/
Personalized Recommendation-MMV2.5
24
Personalized well-being and health-care Platform
Generate
Recommendations
Activity data User Information Nutrition data
Preferences Demographics Risk Factors Feedback
“You have spent whole day in office, please enjoy your favorite exercise cycling for about 25 minutes to relax and improve you’re your muscles health ”
Situation aware 2
“Today weather is not suitable, Kindly take exercise in Gym for 30 minutes”
Context Environ- mental Special Conditions
Existing
well-being Platforms
Personalization
SLIDE 25
/
Personalized Recommendation-Environmental Context
25
“Today weather is not suitable, Kindly take exercise in Gym for 30 minutes” Low fat (chicken soup) is recommended “You have spent whole day in office, please enjoy your favorite exercise cycling for about 25 minutes to relax and improve your muscles health ” “Today weather is not suitable, Kindly take exercise in Gym for 30 minutes” Low fat food (vegetable rice) is recommended Low fat (chicken soup) recommended
Weather Condition
Food Activity
SLIDE 26
/
Personalized Recommendation-MMV2.5
26
Personalized well-being and health-care Platform
Personalization
Activity data User Information Nutrition data
Preferences Demographics Risk Factors Feedback
“You have been sitting for whole days, please take a walk and refresh your mind for at least 5 minutes”
Special Conditions 3
“You were sitting whole day, please stretch your arms and back and neck for 20 min
Context Environ- mental Special Conditions
Existing
well-being Platforms
Generate
Recommendations
SLIDE 27
/
Personalized Recommendation-Special Condition
27
Special Conditions
You were sitting during 1 hour, please stretch your legs, arms and back “We recommend you to use low fat foods (steam cooked fish, noodles cooked in unsalted water)”
End user
Health Condition = Disable Health Condition = Pregnant
“You have been sitting for whole days, please take a walk and refresh your mind for at least 5 minutes” “You were sitting whole day, please stretch your arms and back and neck for 5 min” “We recommend you low fat foods (fish, noodles). “We recommend you to use low fat foods (steam cooked fish, noodles cooked in unsalted water)”
SLIDE 28
28
Human Behavior Quantification
Personalized Recommendation Human Behavior Quantification Expert Recommendation Privacy & Security Induced Habituation
SLIDE 29
/
Descriptive Analytics 2.5
29
Personalized well-being and health-care Platform
Activity data User Information Nutrition data
Preferences Demographics Risk Factors Feedback
Existing
well-being Platforms
Nutrition Log Activity Log SNS
20 Minutes
Knowledge Authoring Tool
Expert View
Domain Expert
SLIDE 30
/
Proteins/Fats /Carbs Consumed overall based on food SNS: Meat /Fish Trends Activities
Descriptive Analytics – Activities, Nutrition, & SNS
30
Nutrition Log SNS Activity Log
SLIDE 31
31
Privacy & Security
Personalized Recommendation Human Behavior Quantification Expert Recommendation Privacy & Security Induced Habituation
SLIDE 32
/
Privacy and Security -MMV2.5
32
Personalized well-being and health-care Platform
Activity data User Information Nutrition data
Preferences Demographics Risk Factors Feedback
Message integrity check
^ ^ % & @ @ # #
Man in the middle modification
Domain expert 1 3 4 2 5 6
1
SLIDE 33
/
Privacy and Security -MMV2.5
33
Personalized well-being and health-care Platform
Activity data User Information Nutrition data
Preferences Demographics Risk Factors Feedback
Privacy and Security 2 User information that affects the MM recommendation (User profile)
acknowledgement 1 User profile (registration) Acknowledgement Profile confirmation
Last updated
2 3 Profile update confirmation 1 2
current updated
SLIDE 34
/
Privacy -MMV2.5
34
Personalized well-being and health-care Platform
Activity data User Information Nutrition data
Preferences Demographics Risk Factors Feedback
Privacy (identity theft) 3 Identification of susceptible theft of user identity
1 User activity recorded at time T1
Previously recorded Locations data
2 3 Initiate multiway authentication User activity recorded at time T2 Location distance > (T2-T1) New location 4
SLIDE 35
Mining Minds V2.5 Architecture
35
SLIDE 36
/
Software Engineering Principles
36
Agile Software Development Process
SLIDE 37
/
MMv2.5: Architecture
37
Delivers timely and accurate personalized cross-domain recommendation based on domain knowledge and users preferences/context Creates and maintains health and wellness knowledge using expert-driven and data- driven approaches Provides real-time data acquisition from multimodal data sources and its persistence using big data technologies. Context data are mapped for life-logging and personalized predictions from life-log Facilitates information to the users in the most intuitive manner, in a secure environment reflecting their personal needs and preferences Converts the data obtained from the user interaction with the real and cyberworld, into abstract concepts or categories, such as physical activities, emotional states, locations and social patterns, which are intelligently combined to determine and track context and behavior
SLIDE 38
/
MMv2.5: Operation Flow
38
SLIDE 39
/
MMv2.5: Distributed Database Model
39
Intermediate Database
Data Curation Layer Information Curation Layer Knowledge Curation Layer Service Curation Layer Supporting Layer
HDFS High Level Context Ontology Knowledge Base Rule Index Heuristics Adaptive Rules
RecognizedLow and High Level Context Sensory Data (Analytics) Lifelog (Data Driven) Lifelog (Data Driven) Lifelog and Profile Knowledge Base Rules Rules Index Lifelog and Profile Situation Index Classification Dataset Recommendation
SLIDE 40
Platform Operations, Uniqueness and Contributions
40
SLIDE 41
/
Platform Operation
41
SLIDE 42
/
MM Service Uniqueness
42
Personalized well-being and health-care Platform Big data
Knowledge base Knowledge Curation Data Curation Domain Expert
End user Knowledge Authoring tool
Cold Warm
Rule Builder Domain Model Manager Rule Validator LLM Lifelog
Historical evidence Authoring Guidance Reduce Tediousness Reduce Complexity Expert View
Benefits
Quality of Services
Wellness Model Clinical Model Existing legacy System Wellness & Clinical Guidelines Expert Input
SLIDE 43
/
Overall Technical Uniqueness
43
Unique Features
Push Service Model Multimodal Heterogeneous Data Acquisition Low Level Context Fusion Situation based Knowledge Maintenance User Experience Quantification Multi level recommendation interpretations SNS & Lifelog based Big Data Analytics Data Curation Layer Information Curation Layer Knowledge Curation Layer Services Curation Layer Supporting Layer Automatic High Level Context Awareness Knowledge based Recommendations Shareable Knowledge Creation
SLIDE 44
/
Technical Contribution
44
Data Curation Layer Information Curation Layer Knowledge Curation Layer Services Curation Layer Supporting Layer
Multimodal, Multidimensional and Multilevel Context Inference Machine-learning- driven Activity, Location and Emotion Recognition Ontology-based High-Level Context Identification Situation-based Knowledge Acquisition and Reasoning Data-driven Knowledge Acquisition from LifeLog Big Data Real-time Life-log Monitoring with Dynamic Situations Big Data Storage with Active/Passive Reads for Analytics and Data-Driven Knowledge Learning Cross-context Interpreted Personalized Recommendations Lifelog & SNS Analysis with Push-based Expert Notifications UX Analytics Tool for UI Adaptation
SLIDE 45
Mining Minds Versions Microdemos
45
SLIDE 46
/
46
Real-time Life-log Monitoring with Dynamic Situations
DCL
SLIDE 47
/
47
Big Data Storage with Active and Passive Reads for Analytics and Data-Driven Knowledge Model Learning
DCL
SLIDE 48
/
Multimodal, Multidimensional and Multilevel Context Inference
48
10 subjects 12 activities 8 locations 4 emotions 98 sensor signals
>2GB
- f data
ICL
SLIDE 49
/
Machine-learning-driven Activity, Location and Emotion Recognition
49
ICL
SLIDE 50
/
Ontology-based High-Level Context Identification
50
ICL
SLIDE 51
/
Situation-based Knowledge Acquisition and Reasoning
51
KCL
SLIDE 52
/
Data-driven Knowledge Acquisition from LifeLog Big Data
52
KCL
SLIDE 53
/
SCL Cross-context Interpreted Personalized
Recommendations
53
SLIDE 54
/
Lifelog & SNS Analysis with Push-based Expert Notifications
54
SL
SLIDE 55
/
UX Analytics Tool for UI Adaptation
55
SL
SLIDE 56
Comparison with Existing Systems
56
SLIDE 57
/ Applications Platforms
Comparison with Existing Systems 57
FEATURES CATEGORIZATIONS storage & Security
Google Fit Samsung S Health Microsoft Health Apple Health Kit Open mHealth NoomCoach Fitbit Argus Runtastic RunKeeper ZombieRun
1. Sensory Data 2. User profile 3. IOT 4. Other apps 5. Clinical data 6. Social media
Data Soruce
1. User device storage 2. Cloud storage 3. Bigdata storage 4. Encrypted data storage 5. Anonymized access
Storage & Security
1. Open knowledge 2. Knowledge acquisition 3. Knowledge evolution
Knowledge Maintenace
1. With other apps 2. Social media sharing 3. Other users (authorized circle)
Information Sharing
1. Activity Recognition 2. Expert Services 3. Wellness services 4. Personalized recommendations 5. Clinical services 6. SDK/API
Services
1. User Experience 2. User Modelling 3. Adaptation of UI
UI/UX
SLIDE 58
/
Data Source Storage & Security UI/UX Services Information Sharing Knowledge Maintenance
Sensory Data User profile IOT Other apps Clinical data Social media User device Cloud storage Big data storage Encryp storagey Anon access User Experience User Modelling Adaptation of UI Activity Recog Expert Svc Wellness svc Personal recom Clinical svc SDK/API With other apps Social media Other users Open Know Know acq Know evolution Google Fit
Samsung S health Microsoft health
Apple healthkit
Open mhealth
Fitbit
NoomCoach
Argus
Runtastic
Runkeeper
Zombie Run
Mining Minds
SLIDE 59
Future Plan
59
SLIDE 60
/
Future Plan: MM V3.0 Milestones
60
May 2016 June 2016 July 2016 Aug 2016 Sep 2016 Oct 2016
V 2.5 V 3.0
Nov 2016 Dec 2016
Physical Activities and Nutrition based MM V2.5 Finalization MM V3.0 - Chronic Disease Scenario Requirements Analysis MM V3.0 – Platform Features Evolution Analysis MM V3.0 – Chronic Disease Scenario based System Design MM V3.0 – Chronic Disease Scenario based System Design MM V3.0 - Chronic Disease Scenario Finalization MM V3.0 – Chronic Disease Scenario Implementation MM V3.0 – Chronic Disease Scenario Testing and Integration
SLIDE 61
/
Future Plan: MMv3.0 Roles
61
Infrastructure Level
Data Curation Layer Information Curation Layer Knowledge Curation Layer Service Curation Layer Supporting Layer
Platform Level Services Level
SLIDE 62
/
Conclusion
62
V3.0 (Dec 2016)
Physical Activity + Nutrition + Chronic Disease scenario
Demonstrated platform-oriented and layer- wise technical contributions Established overall uniqueness compared to similar existing systems Evolution of Mining Minds platform from V1.0-1.5 (Physical Activities) & V2.0-2.5 (Physical Activities & Nutrition) Developed the infrastructure to support current and future services
SLIDE 63