A Stock Prediction System using open-source software Fred Melo - - PowerPoint PPT Presentation

▶
a stock prediction system using open source software
SMART_READER_LITE
LIVE PREVIEW

A Stock Prediction System using open-source software Fred Melo - - PowerPoint PPT Presentation

Dec 29, 2023 268 likes •680 views

A Stock Prediction System using open-source software Fred Melo William Markito fmelo@pivotal.io wmarkito@pivotal.io @fredmelo_br @william_markito It's all about DATA Prediction Data Sources Look for patterns Machine Learning is the answer

slide-1
SLIDE 1
slide-2
SLIDE 2

A Stock Prediction System using

  • pen-source software

Fred Melo fmelo@pivotal.io @fredmelo_br William Markito wmarkito@pivotal.io @william_markito

slide-3
SLIDE 3
slide-4
SLIDE 4

It's all about DATA

Data Sources Look for patterns Prediction

slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7

Machine Learning is the answer

Neural Networks Clustering Genetic Algorithms

slide-8
SLIDE 8

Train with historical dataset Apply model to the new input

Applying Machine Learning

slide-9
SLIDE 9

Hard to add new data sources

Why?

Hard to scale

Why so hard?

Hard to make it real-time

slide-10
SLIDE 10

HDFS

Data Lake

Store Analytics Hard to change Labor intensive Inefficient No real-time information ETL based Data-source specific

Traditional models are reactive and static

slide-11
SLIDE 11

HDFS

Data Lake

Expert System / Machine Learning In-Memory Real-Time Data

Continuous Learning Continuous Improvement Continuous Adapting

Data Stream Pipeline

Multiple Data Sources Real-Time Processing Store Everything

Stream-based, real-time closed-loop analytics are needed

slide-12
SLIDE 12

Info Analysis Look at past trends

(for similar input)

Evaluate current input

Score / Predict

Neural Network

How can it be addressed?

slide-13
SLIDE 13

Info Analysis

Filter [ json ]

Neural Network

How can it be addressed?

slide-14
SLIDE 14

Info Analysis

Filter Enrich

Neural Network

How can it be addressed?

slide-15
SLIDE 15

Info Analysis

Neural Network

Filter Enrich Transform

How can it be addressed?

slide-16
SLIDE 16

Info Analysis

Filter Enrich Transform

Neural Network

How can it be addressed?

slide-17
SLIDE 17

Info Analysis

Filter Enrich Transform Transform

Neural Network

How can it be addressed?

slide-18
SLIDE 18

Neural Network

In-Memory Data Grid

Real-time scoring

How can it be addressed?

Train

slide-19
SLIDE 19

Neural Network

In-Memory Data Grid

Front-end

Update Push

How can it be addressed?

slide-20
SLIDE 20

Ingest Transform Sink SpringXD

Store / Analyze

Fast Data

Distributed Computing Predict / Machine Learning

Other Sources and Destinations JMS

Streaming real-time analytics architecture

slide-21
SLIDE 21

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning Fast Data

Filter Predict Sink

HTTP

Split Dashboard

Push

Demo Architecture

slide-22
SLIDE 22

SpringXD

shell - R Transformer geode-json client geode-json client http-client http-server

  • bj-to-json

splitter splitter Simulator tap

slide-23
SLIDE 23

SpringXD

INGEST / SINK PROCESS ANALYZE

  • Little or no coding required
  • Dozens of built-in connectors
  • Seamless integration with Kafka,

Sqoop

  • Create new connectors easily

using Spring

  • Call Spark, Reactor or RxJava
  • Built-in configurable filtering,

splitting and transformation

  • Out-of-box configurable jobs for

batch processing

  • Import and invoke PMML jobs

easily

  • Call Python, R, Madlib and other

tools

  • Built-in configurable counters and

gauges

Data Stream Pipelining

slide-24
SLIDE 24

SpringXD XD Nodes XD Nodes XD Nodes XD Nodes

Ingest

SpringXD

Split Filter Transform Sink

XD admin XD Nodes

Ingest Split Filter Transform Sink

Stream Deployment Messaging

Scale-Out and HA Architecture

slide-25
SLIDE 25

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning Fast Data

Filter Predict Sink

HTTP

Split Dashboard

Push

Demo Architecture

slide-26
SLIDE 26

Geode client-server architecture

slide-27
SLIDE 27

Partitioned Regions

slide-28
SLIDE 28

Event handling

slide-29
SLIDE 29

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning Fast Data

Filter Predict Sink

HTTP

Split Dashboard

Push

Demo Architecture

slide-30
SLIDE 30

Neural Networks

slide-31
SLIDE 31

Neural Networks

slide-32
SLIDE 32

medium avg (x+1) relative strength (x)

medium avg (x) price(x)

Neural Network

slide-33
SLIDE 33

Neural Network

slide-34
SLIDE 34

Transform Sink

SpringXD

Extensible Open-Source Fault-Tolerant Horizontally Scalable

HTTP

Machine Learning Fast Data

Filter Predict Sink

HTTP

Split Dashboard

Push

Demo Architecture

slide-35
SLIDE 35
slide-36
SLIDE 36

Demo Time

slide-37
SLIDE 37

SpringXD

shell - R Transformer geode-json client geode-json client http-client http-server

  • bj-to-json

splitter splitter Simulator tap

slide-38
SLIDE 38

SpringXD

http://projectgeode.org http://projects.spring.io/spring-xd http://www.r-project.org

slide-39
SLIDE 39
slide-40
SLIDE 40

Follow-up: In-Memory Unconference
 "A place for all things in-memory: projects, people, ideas, roadmaps, discussions."
 Location: Hill Country A/B”
 Weds 4:15pm - 6pm. (after this talk)

The demo code is on GitHub! @fredmelo_br @william_markito