A Stock Market Prediction System based on Learning from Historical Prices and Customer Reviews
Supervised By
Dr. Reda El-Basiony
Abdallah Ibrahim El-Badry
Abdelrahman Mohamed Emam
Aya Mohsen El-Kilany
Habiba Khaled Ibrahim
Ibrahim Mohamed Yousef
Mona Mardy El-Araby
Reem Emad Abdelaziz
Team Members
Outlines
01
Problem & Solution
02
System Components And Architecture
04
Front End Development
03
Back End Development
05
Machine Learning For Time Series Forecasting
06
Conclusions & Future Work
Problem Statement
Problem Statement
Motivation
Stock Market Applications
Allows users to sync portfolios and quotes across multiple devices � Tracking stocks, currencies, commodities � Notifications allow you to stay current on breaking news
Users can play a simulated version of the real U.S. stock market and invest virtual funds without taking on any real risk. � Shouldn't be used as a decision-making tool for actual trading � A great way for first-time investors to get their feet wet and learn how the stock� market works without losing any actual money. �
Proposed Solution
Proposed Solution
Historical Data
Tweets
Proposed Solution
Outlines
01
Introduction
02
System Components And Architecture
04
Front End Development
03
Back End Development
05
Machine Learning For Time Series Forecasting
06
Conclusions & Future Work
System Model
Goals:
System Model
This project built on Google Cloud infrastructure..
Why Cloud!?
1. Cost efficiency
2. Private Global Fiber + Tiered Network�3. Increase number of users�4. Improved Performance �5. More Security�6. Continued Expansion
7. Scalability
8. Flexibility
What is Google Cloud Platform!?
Google Cloud Platform is a suite of public cloud computing services offered by Google
System Architecture
Outlines
01
Introduction
02
System Components And Architecture
04
Front End Development
03
Back End Development
05
Machine Learning For Time Series Forecasting
06
Conclusions & Future Work
System Architecture
Html, css and..React js!
React (also known as React.js or ReactJS) is a JavaScript library for building user interfaces.
React allows developers to create large web applications which can change data, without reloading the page.
Why React js?
What is React js?
Data Visualization
The Tool We Used:
React Stockcharts: creates highly customizable stock charts built with React js and d3.
Home page
Featured Companies
Company Page
Historical Prices
Company Page
Daily Prediction
Company Page
Monthly Prediction
Company Page
Twitter Analysis
News(Twitter)
News(Financial News)
Outlines
01
Introduction
02
System Components And Architecture
04
Front End Development
03
Back End Development
05
Machine Learning For Time Series Forecasting
06
Conclusions & Future Work
Back End Development
What is app engine & features?
Back End Development
by download SDK, we can start the project.
Back End Development
Back End Development
Compute Engine
Big query
App Engine
App engine
(website)
Big query
Send request for data
Response Send back json file
Back End Development
This API call could be local like this
This API call for Facebook stocks prediction.
Call query from bigquery , the result json file
Back End Development
Then after deploy the backend folder on google cloud:
Back End Development
In Front-End Use Axios to call API
Axios is a JavaScript library used to make HTTP requests
Back End Development
http://www.tanta-stocks.appspot.com
Build React
Deploy on
App Engine
Front-end
+
Back-end
Back End Development
Home page
Outlines
01
Introduction
02
System Components And Architecture
04
Front End Development
03
Back End Development
05
Machine Learning For Time Series Forecasting
06
Conclusions & Future Work
Machine Learning For Time Series Forecasting
Phase1
01
Phase2
02
Phase3
03
Phase1
Historical Data
Phase2
Social media Data(Twitter)
Phase3
Historical + Twitter Data
Machine Learning For Time Series Forecasting
Phase1
01
Phase2
02
Phase3
03
Phase1
Historical Data
Phase2
Social media Data(Twitter)
Phase3
Historical + Twitter Data
Phase 1
ARIMA Model
FBProphet
For Short term Prediction
For Long term Prediction
Time Series Data
Time-series data is a sequence of observations stored in time order.
Time series analysis is an approach to analyze time series data to extract meaningful characteristics of data and generate other useful insights applied in business situation.
Time series analysis helps understand time-based patterns of a set of metric data points which is critical for any business.
ARIMA Model
ARIMA stands for Auto Regressive Integrated Moving Average.
The AutoRegressive (AR) model
specifies that the output variable depends linearity on its own previous values.
Yt = β0 + β1 Yt-1 + β2 Yt-2 + .. + βpYt-p
Moving average Model
In statistics, a moving average is a calculation to analyze data points by creating a series of averages of different subsets of the full data set.
MA=(A1+A2+…+An)/n
ARIMA is a combination of Auto Regression Model and moving Average Model with difference to make data stationary.
The parameters of the ARIMA model are defined as follows:
• p is the number of autoregressive terms(AR Model)
• q is the number of lagged forecast errors in the prediction equation(MA Model)
• d is the number of nonseasonal differences needed for stationarity(Integration level)
Historical Data API
Data Schema
Visualizing Close time series data
Validating forecasts
To help us understand the accuracy of our forecasts, we compare predicted Close to real close price of the time series
Producing and visualizing forecasts
FBprophet Model
Fbprophet is an open source released by Facebook in order to provide some useful guidance for producing forecast at scale.
fbprophet package in Python or R can help us look deeper into the hidden cycles in stock market.
The Prophet uses a decomposable time series model with three main model components: trend, seasonality, and holidays. They are combined in the following equation:
y(t)= g(t) + s(t) + h(t) + εt
• g(t): piecewise linear or logistic growth curve for modeling non-periodic changes in time series.
• s(t): periodic changes (e.g. weekly/yearly seasonality).
• h(t): effects of holidays (user provided) with irregular schedules.
• εt: error term accounts for any unusual changes not accommodated by the model.
The important idea in Prophet is that by doing a better job of fitting the trend component very flexibly, we more accurately model seasonality and the result is a more accurate forecast
Apple stock price from 2014/10/10 to 2019/2/15
We can also visualize our data according to different components of FBprophet model: weekly, yearly, trend and user-defined cycle
Model validation
In our model we try to use last 12-month data to predict the next 10 days.
Machine Learning For Time Series Forecasting
Phase1
01
Phase2
02
Phase3
03
Phase1
Historical Data
Phase2
Social media Data(Twitter)
Phase3
Historical + Twitter Data
Phase 2
Sentimental Analysis
Nature Language Processing
Sentiment Analysis
Sentiment Analysis is the most common text classification tool that analyses an incoming message and tells whether the underlying sentiment is positive, negative our neutral.
Naïve Bayes
Bayes theorem calculations :
Data Set Sample
We used training dataset from sentiment140.com. The dataset comes with prelabeled columns containing one of two binary targets. Zero(0) for Negative sentiment and Four(4) for Positive sentiment. The data is made up of about 1.6 million random tweets with corresponding sentiment labels.
�
Twitter API
Tweets
Training Data Preprocessing
Pre-processing steps:
Feature Extraction
We will transform data into tokens and then tokens into features. This is done by Term Frequency, Inverse Document Frequency (TF-IDF).
We'll convert each message which is represented by a list of tokens into a vector that a machine learning model can understand.
Feature Extraction steps:
Naïve Bayes Training
Naïve Bayes score
Phase 2
Sentimental Analysis
Nature Language Processing
What is deep learning ?
Deep Learning is :
*input layer .
*hidden layer.
*output layer .
NLP
Natural Language Processing (NLP) is the field of Artificial Intelligence concerned with the processing and understanding of human language.
Since its inception during the 1950s, machine understanding of language has played a pivotal role in translation, topic modeling, document indexing,
information retrieval, and extraction.
NLP using Deep learning
CNN can extract an area of features from global information, and it is able to consider the relationship among these features. The above solution can achieve a higher accuracy in analysis and classification. For natural language processing, texts data features also can be extracted piece by piece and to consider the relationship among these features, but without the consideration of context or whole sentence, the sentiment might be understood wrong.
CNN has a convolutional layer to extract information by a larger piece of text, so we work for sentiment analysis with convolutional neural network
Model Architecture
Model Architecture
Model Layer description:
1- Embedding Layer:
we can think of the Embedding layer as a dictionary
that maps an index assigned to a word to a word vector.
This layer is very flexible and can be used in a few ways:
a-The embedding layer can be used at the start of
a larger deep learning model.
b-Also, we could load pre-train word embeddings into
the embedding layer when we create our model.
Model Layer description:
2- Convolution layer:
After we come out of embedding layer, we want to get feature
descriptor for whole of text, so we use neural network over this
embedding. To analyze n-grams in our text, we just convolve
the word vectors that are near, you can think of it as sliding
window walk on our word vector matrix. You need to track
many n-grams, that's why you need a lot of convolutional filters.
And these filters are called 1D convolutions because we
slide the window only in one direction.
Model Layer description:
3- Maxpooling layer:
is a sample-based discretization process.
The objective is to down-sample an input
representation (image, hidden-layer
output matrix, etc.), reducing its
dimensionality , we do maximum pooling
overtime and, we apply multiple filter with
different length and finally we got vector is
a kind of embedding of our input sequence.
Model Layer description:
4- Flatten layer:
Does not affect the batch size. The purpose of this argument is to preserve weight ordering when switching a model from one data format to another
5- Dense layer :
Two dense layer are used in our architecture one with relu activation function , another one with sigmoid
6- dropout layer:
This layer prevent overfitting .
Data Cleaning
Text preprocessing is technique that transforms raw data into an understandable format. Raw data(real world data) is always incomplete and that data cannot be sent through a model. That would cause certain errors. That is why we need to preprocess data before sending through a model.
Preprocessing steps:
1- lowering words.
2- remove punctuation.
3- remove hyperlinks.
4-remove unused fields.
Feature Extraction
We will transform tokens(words) into features. This is done by word embedding. Word Embedding is the texts converted into numbers and there may be different numerical representations of the same text. many Machine Learning algorithms and almost all Deep Learning Architectures are incapable of processing strings or plain text in their raw form They require numbers as inputs to perform any sort of job, be it classification, regression etc. A Word Embedding format generally tries to map a word using a dictionary to a vector.
Feature Extraction steps:
1- Tokenize sentence into words or tokens. Each word take an integer id . Start indexing from1.
2- Convert sentence into vector of numbers by replacing each word by its integer id .(To_sequence_function)
3- padding sequence mean when there are multiple sentence with different length ,we are in need to make them equal in length vector.
Model Training
Comparison
| sentiment analysis (with machine learning) | Nlp using deep learning |
Data dependency | Machine learning algorithms often work well even if the dataset is small | Deep learning is Data Hungry the more data you have, the better it is likely to perform |
computation power | Traditional machine learning algorithm can be implemented on a CPU | For the deep learning network training, you need a graphical processing unit (GPU) which have thousands of cores compared to a CPU that has very minimal cores. |
Time consuming | Whereas traditional machine learning algorithms often train very fast ranging from few minutes to a couple of hours, but during the test time, some algorithms can also take quite a bit of time. | The training time of a deep learning network can range from anywhere between a few hours to months |
Machine Learning For Time Series Forecasting
Phase1
01
Phase2
02
Phase3
03
Phase1
Historical Data
Phase2
Social media Data(Twitter)
Phase3
Historical + Twitter Data
Phase 3
Sell or buy model
Polynomial model
CNN model
1- Sell/Buy Model
We used SVM model to suggest whether a person should buy or sell a share. We applied some preprocessing on the data as we created new columns to present the class if it is 1 or 0 (buy or sell) so we can train the model.
SELL
BUY
SVC Model
We achieved 98% accuracy
2- Polynomial Regression Model
In statistics, polynomial regression is a form of regression analysis in which the relationship between the independent variable x and the dependent variable y is modelled as an nth degree polynomial in x. Polynomial regression fits a nonlinear relationship between the value of x and the corresponding conditional mean of y, denoted E(y | x). Polynomial regression is considered to be a special case of multiple linear regression.
This is the general equation of a polynomial regression is:
Y=θo + θ₁X + θ₂X² + … + θₘXᵐ + residual error
We developed a polynomial regression model to enhance our daily prediction using the score we get from NLP analysis of tweets and customer reviews. We used a second-degree model to fit our data. We used 70% of the data for training the model and 30% for the evaluation.
2- Polynomial Regression Validation
3- CNN Model
We use 3 hidden layers (Conv2D, MaxPooling2D and Dense). And input of size (1,7,7). We use RELU function for all of them.
Conv2D of 64 neurons.
Pool size = (1, 2).
Dense of 30 neurons
3- CNN Model Architecture
Input Layer 49 neurons
Dense�7*7*32
Output Layer
Conv2D 7*7*64
MaxPooling2D 1*2*64
3- CNN Model Architecture
3- CNN Model Validation
Evaluation
Actual / predicted | True | False |
True | 35 | 1 |
False | 0 | 63 |
Evaluation
Model | MSE | RMSE | EVS | R2 |
CNN |
4.567077 |
6.52544 |
0.86388 |
0.86266 |
Polynomial |
8.4125 |
2.9 |
0.9731 |
0.9724 |
FBprophet |
6.6753 |
9.219508 |
0.73108 |
0.73107 |
Outlines
01
Introduction
02
System Components And Architecture
04
Front End Development
03
Back End Development
05
Machine Learning For Time Series Forecasting
06
Conclusions & Future Work
Conclusions
Future Work
Demo
Thank You
Insert the Sub Title of Your Presentation