New York Stock Exchange Price Prediction
Produced by Bingqing (Claire) Li, Fan Hong, Kerui Chen, Tingyi Hung, Yuejia Feng

Predicting stock price has been a popular topic in both financial and academic fields, and increasing number of people is trying to dig deeper in this problem. There are different ways to approach this problem. In the past, the fundamental analysis, technical analysis, and quantitative analysis have been conducted, but building predictive model using machine learning knowledge has become more and more popular. In this project, we have tried different models, such as linear regression, LSTM, and ARIMA in this project, and this blog will describe the process in detail.

I. Project Summary

Goal: Build time series models for New York Stock Exchange to predict tomorrow’s stock price in a continuous format, and plot a trendline for future understanding. Also, we extract business insights from different modeling predictions.

Data: Our dataset for this project is from a Kaggle competition named “New York Stock Exchange”. For our analysis, we mainly used two data files. One is stock price data and the other one is “fundamentals”, which presents company’s yearly financial data. The link for this competition is ​https://www.kaggle.com/dgawlik/nyse

Method: In summary, we separated our modeling analysis into two categories. The first one is fundamental analysis using linear regression, trying to find the relationships between company’s financial data and its stock price in a yearly manner. Another category is utilizing time series model, Auto ARIMA and LSTM, to predict future stock price in certain time window, and plot the trend line for a specific company.

Applications: The potential application of our models are plentiful. From individual investors to business portfolio investment, it could be used as a essential tool to help people make right decisions.

Performance: The results are mostly positive. We explored the relationships between financial variables and stock prices even though the prediction error is higher. Also for the the time series model, LSTM achieved the least error, and we believed that is the most accurate model in this project.

Github Link: https://github.com/yjiafeng117/DataMiningFinalProject

II. Data Collection

Our dataset for this project is from a Kaggle competition named “New York Stock Exchange”. There are four different data sources in this dataset: price.csv, prices-split-adjusted.csv, securities.csv, and fundamentals.csv. Among all the data source, our team will be mainly explore price and fundamentals.csv to find the relationship between stock prices and company’s financials and estimate future trend of stock prices.

Here is a short presentation of our two main dataset:

Google’s stock price related items by date in 2012

American Airlines financial data trends from 2012 to 2015

III. Fundamental Analysis — Linear Regression

A. Data Preprocessing

Features and labels

We used features from “ fundamental.csv” as the independent variables. The file contains the financial indexes of 348 companies from 2010 to 2016. There are 75 features as following:

Header of Fundenmantal.csv

For the dependent variable Y, we calculated the average stock price for each company per year from “prices-split-adjusted.csv” table, because the X variables are yearly based. And then, two tables were merged together according to two indexes “Ticker” and “Year”.

2. Checking collinearity

For linear model, one of the assumption is there are no obvious correlation among independent variables, but for financial indexes, we can find such relationship among them. So, it is necessary to check the collinearity of data. By calculating Pearson correlation coefficient, we deleted 15 variables whose coefficients are over 90% as following:

Collinearity within X variables

B. Modeling

Ordinary least square model (OLS)

In statistics, OLS is a type of linear least squares method for estimating the unknown parameters in a linear regression model. OLS chooses the parameters of a linear function from a set of explanatory variables by the principle of least squares: minimizing the sum of the squares of the differences between the observed dependent variables (values of the variables being predicted) in the given dataset and those predicted by the linear function. We train a OLS model, fitting in X and Y. The R-squared of the whole model is 0.6, which means 60 percentage of security price’s movements that can be explained by movements in benchmark indexed. However, what we want was to figure out what indexes would be more important when we make investment decisions. P-values can tell us about that. For those variables with less than 0.05 P-value, we can say they have significant influences on the target, which are as following:

P-values < 0.05 in OLS model

Also, the covariance will tell us it is a positive influence or negative one.

2. PanelOLS model

Anthor model we tried is PanelOLS. It used the same formula with OLS, but the amazing thing it has is it can automatically cluster by entities and time. We get a higher R-squared value for PanelOLS model. It made sense because stock prices has a nature that it changes by entities and time. We got some different P-values for independent variables as following:

P-values < 0.05 in PanelOlS model

C. Results

We use Root Mean Squared Error(RMSE) as the metric to evaluate our model. Though the result of two models are 51 which is not good compared to more complex models, we believe PanelOLS are much reliable when speaking of feature importance.

Our prediction shows the as similar trend as true labels.

IV. Time Series Models

A. Auto ARIMA

Data Preprocessing

There are several columns in this “prices-split-adjusted.csv” dataset, including date, symbol, open, close, low, high and volume.

open and close: Starting and final stock price.

high and low: The maximum and minimum stock price.

volume: The number of shares bought or sold in the day.

First, we load the stock price dataset and the data needs to be transformed into a format containing time series index for Auto Arima model to consume. Therefore, we sort the column “date” and set it as the index of our dataset. Then, in this blog, we are going to predict Microsoft’s future stock price, so we extract the data with the value, MSFT, in “symbol” column. At the same time, we take the column “close” as our target variables because the closing price can determine whether it is profit or loss. The following picture shows the historical stock price of Microsoft:

After that, we split the preprocessed data into training dataset and validation dataset in which training dataset is 75% of the preprocessed data and test dataset is 25% of the preprocessed data. Now, these two datasets are ready for checking the performance of Auto Arima model.

2. Modeling

Arima is known as a popular tool in making predictions on time series data. Before implementing Arima model, we need to firstly make the series stationary, and determine the values of parameters. Auto ARIMA makes this task really simple, the data preparation and parameters tuning processes end up being really time consuming. This model is able to automatically find the best combination of the parameters based on information criterions (AIC and BIC) which estimate the quality of the model. There are several important parameters in this model and the followings are the meanings of the parameters and the values we set for them.

p: The order of the autoregressive (AR) model which will incorporates past values. We use `start_p=1` and `max_p=3` and let the model to search best “p”.

d: The degree of differencing. We use `test=“adf”` which is adftest to find optimal “d”.

q: The order of the moving average (MA) model. We use `start_q=1` and ` max_q=3` and let the model to search best “q”.

P, D & Q: Describe the same associations as p,d, and q, but correspond with the seasonal components of the model. We use `start_P=0` and `D=1` for finding best values.

m: The number of observations per seasonal cycle. m = 12 (for monthly data, it suggests a yearly seasonal cycle)

seasonal: It makes the model handle in seasonal data. We set it as “True” because stock price has seasonal variation.

3. Results

To check the performance of the Auto Arima model, the root mean square error (RMSE) can be calculated by using the predictions and actual values on the validation set. A lower value of the RMSE implies better accuracy. The RMSE of the Auto Arima model is close to 7:

In order to quantify the uncertainty of the predictions, we plot the confidence intervals which provide an upper and lower expectation for the real observation. The grey shade in the chart shows the confidence interval of this model. From the chart, most of real observations lie within the 95% confidence band.

B. LSTM

Data Preprocessing

To make comparison with two models, we used MSFT historical stock price again. We split the data by 75% and make prediction upon test data. To make our data in the same scale, we used MinMaxScaler from sklearn to scale down our data from 0 to 1. Also we create a data structure with 60 timesteps and 1 output and reshape it to meet Keras 3D tensor input shape, which includes batchsize, time step, and input dimension.

2. Modeling

LSTM stands for Long Short Term Memory networks. It is an improvement of RNN and very effective on sequence prediction because it can store past information that is important, and forget the information that is not. The architecture of LSTM is displayed below.

Architecture of LSTM

A typical LSTM network is comprised of different memory blocks called cells. There are two states that are being transferred to the next cell; the cell state and the hidden state. The memory blocks are responsible for remembering things and manipulations to this memory is done through three major mechanisms, called gates.

The forget gate removes the information that is no longer required by the model

gate removes the information that is no longer required by the model The input gate adds information to the cell state

gate adds information to the cell state The output gate selects the information to be shown as output

Now let’s see how time series data is implemented in LSTM.

1st, importing the Keras libraries and packages.

2nd, initialising the RNN using Sequential model.

3rd, adding the 1st to 4th LSTM layer with 50 units and Dropout to avoid overfitting

4th, adding the output layer using Dense layer.

5th, compiling the RNN with optimizer ‘adam’

6th, fitting the RNN to the training data with epoch is 10

7th, getting the real stock price of test data

8th, getting the predicted stock price of test data

9th, visualising the results

3. Results

The RMSE of LSTM model is 5.85 and the graph of stock price changes are displayed below . In general we believe, the RMSE will decrease as epoch increases when we training the data.

To make it clear, let’s zoom in and see how the predicted stock price differentiate from real stock price.

C. Model Evaluation

The following results present how the LSTM and Auto ARIMA model perform when predicting stock price after 441 days (No specific reason, we just choose a random day. We can also predict stock after one day or after a week). Each model predicted the stock price for the nine products used as datasets, resulting in a error value in RMSE. To compare the models and their error-values, a t-test was conducted, comparing if there exists a significant difference between the models. The difference between the models for each individual product, the overall difference in error as well as the p-value from the t-test is presented in table 4.1. For both the individual products as well as for the total difference in error, a negative value in the tables indicates that the LSTM model has a lower error, while a positive value indicates that the ARIMA model has a lower error.

In our scenario, H0 was that the LSTM model has a smaller prediction error than the Auto ARIMA model, and Ha (alternative hypothesis ) is LSTM model has a prediction error as small as or greater than the ARIMA model.

The following table shows the results for the prediction scenario:

The P-value we got is 0.02 (<0.05), which indicates that the LSTM model in this prediction scenario does have higher accuracy than the Auto ARIMA model.

V. Conclusion

As part of our objectives for this project, we want to generate some business insights from it that can be used in the future and furthermore. From our linear regression and fundamental analysis, based on the significance level of variables (p-value) and coefficients (beta), we found that some of the features have positive influence on the company’s stock price, such as Gross Profit, Long-Term Investments, Profit Margin, Earnings Per Share. Non-Recurring Items, Operating Margin. Some others may have negative influence like Deferred Asset Charges, Interest Expense, Cash and Cash Equivalents, Inventory, Total Current Asset. The interpretation of the negative impact from Cash and Cash Equivalents and Total Current Asset is that if company keeps many cash on hand instead of investing, there will be no value added to the company as time passed, which could lower its stock value in the future.

In addition, we could use the data to calculate stock percentage change to see how stocks ebb and flow. We also plot stock return so that stockholders can know when is the best time to purchase a particular stock. More importantly, we plotted Google and Microsoft stock price curves through 2010 to 2017 by normalizing data (by dividing each time series element of all time series by the first element). Companies can make strategies to outperform against their competitors. Stockholders can compare stocks among companies and make better purchase decisions.

Stock Percentage Change of Agilent Technologies

Stock Return of Agilent Technologies

Stock Prices of Google and Microsoft

Reference

https://github.com/bashtage/linearmodels/tree/master/examples

https://www.machinelearningplus.com/time-series/arima-model-time-series-forecasting-python/

https://towardsdatascience.com/inferential-statistics-series-t-test-using-numpy-2718f8f9bf2f

https://pdfs.semanticscholar.org/e58c/7343ea25d05f6d859d66d6bb7fb91ecf9c2f.pdf

https://github.com/krishnaik06/Stock-Price-Prediction-using-Keras-and-Recurrent-Neural-Networ
['Bingqing Li']
['Lstm', 'Time Series Forecasting', 'Auto Arima', 'Stock Price', 'Machine Learning']