DEEP REINFORCEMENT LEARNING IN ALGORITHMIC TRADING (Part- II)
DEEP REINFORCEMENT LEARNING IN ALGORITHMIC TRADING (Part- II) Athekunal Follow Dec 20 · 9 min read

So now we will discuss the paper, An Application of Deep Reinforcement Learning to Algorithmic Trading.

First, what happens in trading activity?

All our trading activities are compiled in a portfolio and at any time t, our portfolio has some cash and shares. Buying and selling operation is simply cash and shares exchange. So the main aim is to increase your LHS through yield and avoid losses.

So what are we expecting from our model? We expect our model which has been trained on historical OHCLV data, to go through the order book of present-day and then come up the price, number of shares and whether to Ask (sell) or Bid (buy).

REINFORCEMENT LEARNING PROBLEM FORMATION?

For Reinforcement learning, we need three things → STATE, ACTION SPACE AND REWARD. Let’s go through each of them

STATE SPACE

For state-space, the agent has to look at two things, its internal state and market observations.

Its internal state is defined by the current portfolio value, how many shares it owns and whether it has a net long or short position. You can learn about short selling in this video. The takeaway is, going long means going with the market. So if the market falls you lose money and if it blooms, then you gain. Going short means betting against the market.

Market observations give the current state of the market. So the entire state space looks like the following

STATE-SPACE

BUT encompassing the entire state space would lead to complex design and would induce latency. Latency is a key factor in algorithmic trading. So in the paper, they have a state-space reduction to internal state and OHCLV data ONLY.

STATE-SPACE AFTER REDUCTION

ACTION SPACE

At each time step, the agent executes an action. Whether to go long, short or do nothing (hold). Also, we expect or agent to come up with the number of shares to trade. SO NEED A WAY TO COMBINE OUR POSITION AND NUMBER OF SHARES

Action Space

So if your action is +10,000 for a company XYZ, then you want to hold a long position and want to buy 10,000 shares.

BUT THERE IS A LIMIT TO NUMBER OF SHARES THAT CAN BE TRADED

CONSTRAINT — 1

YOU SHOULD NOT RUN OUT OF CASH!!!. At any time step t, your cash value should be more than 0.

After you buy Q_t shares, your cash value changes but your cash should always be positive. So this gives an upper bound to the number of shares that you can buy.

CONSTRAINT — 2

When the market is rising even if you went short, you should have enough money to payback share lenders in short selling. In short selling, you borrow shares in the promise that you will pay them back at the current market price in the future.

So this constraint the number of shares that we can trade when the market is moving against you. Your cash value should be sufficient to pay back share lenders

When the market is volatile, then you are in a risk to recoup your losses. So another parameter (ϵ) was defined which accounts for the change in market price. Here negative means, you are in a short position.

Enough cash value to pay back share lenders

where ϵ is

SO IF THE MARKET IS TOO MUCH VOLATILE, BETTER HAVE SOME CASH ASIDE TO PAYBACK SHARE LENDERS

More is the ϵ → More is the price volatility → Better have MORE cash to payback share lenders.

The paper goes into the mathematical derivation of action space, you can refer it under the Appendix section.

REWARD

This paper only takes discounted rewards, that is daily returns that we get from the strategy.

Daily return of the portfolio value

Total Reward as the discounted sum

We take discounted rewards and the γ factor plays a key role here. If γ tends to 0 then it values recent returns or prices more. Also, small γ can’t capture the cumulative sum of trading costs and would lead to losses.

If γ tends to 1, then trading costs are considered. But too high γ can lead to passive strategies like Buy and Hold or Sell and Hold. Hence for a balance 0.4 was used. It strikes a balance between active strategy and transaction cost.

Other forms like labelling gain as 1 and loss as 0, would lead to sparse reward setup and the agent won’t learn better. Also, the agent won’t capture the margin of gain or loss. Hence daily return is better

Also, you can specify some explicit rewards like in the paper, rewarding -0.1 when the agent does nothing to promote active strategies instead of passive strategies. -10 if it ran out of money and -100 if it ran out of money and have 0 stocks

Also, you need an objective function like Sharpe Ratio which takes the risk involved in our strategy. You need to maximize the objective function. It is not used as rewards, but Sharpe ratio is used as a comparison tool between different strategies.

Sharpe ratio gives the OPPORTUNITY COST for taking the extra risk and investing in stocks instead of risk-free bonds like treasury and government bonds. Blind reward chasers are not optimized models

TDQN ALGORITHM (TRADING DEEP Q-NETWORK)

We discussed states, action and rewards of the Reinforcement learning problem. Now we will discuss the algorithm.

The trading costs were fixed to 10% of the current market price. Higher trading cost reduces the trading frequency and the agent slowly adopts passive strategies as trading costs increases.

Here we use DDQN (Double Deep Q-Network) instead of DQN (Deep Q-Network). DDQN is a modification to DQN which tackles the overestimation bias.

In DQN, we have an online learning network and a target network. Decoupling these two helps us to stabilize our targets. The online learning network optimizes its weights in each time step and after some fixed number of time steps (here 1000 steps), its weights are cloned to target network.

The target network chooses the action and finds the max Q-value. This leads to overestimation and unstable training. Whereas DDQN unloads the work of choosing action by epsilon greedy policy to the online learning network and target network is only concerned with selecting the max value. This ameliorates the problem of overestimation and leads to stable training.

Huber loss was used instead of MSE loss, because of the Avalanche effect. In Avalanche effect, a slight change in the input can lead to a significant change in output. MSE loss penalizes the outliers, which is common in volatile stock markets. But the prices generally normalize after the hype dies down. Hence as a stable loss function, Huber loss is used which mildly penalizes large errors.

Huber loss

HOW TDQN TACKLES EXPLOITATION AND EXPLORATION DILEMMA?

At each time step, the network executes an action (long or short) in an environment (E) and executes an opposite action in a copy of the environment E.

of the environment E. This trick does not entirely solve the challenge but helps the agent in exploration

DON’T CHANGE HISTORY DURING TRAINING

In the order book, we saw supply and demand play out. If the supply of shares increases, the price will go down as more orders are fulfilled.

So the number of shares traded by the agent while training in simulated environments is LESS so that it does not change the history. The number of shares is less or the trade is executed in multiple batches so that it does not affect the market.

RESULTS AND DISCUSSION

TRAINING SET : 01/01/2012 to 31/12/2017

: 01/01/2012 to 31/12/2017 TESTING SET: 01/01/2018 to 31/12/2019

01/01/2018 to 31/12/2019 The RL agent was compared to both passive (Buy and hold, Sell and hold) and active strategies (Trend following and mean reversion).

Trend following means to go with the trend and have a constant position . The stock price won’t return to its normal value hence you would buy if you think it would go up

. The stock price won’t return to its normal value hence you would buy if you think it would go up Mean reversion means the Stock price is believed to come back to normalcy after some downturn or upturn. You would have two different positions. If you think there is an upturn, but it is a weak trend, so prices will fall then you would sell and then buy.

Mean reversion after price rises and then falls

The trading algorithm was trained on APPLE (“AAPL”) stock which is fairly stable . The TDQN surpassed both passive and active trading strategies.

. The TDQN surpassed both passive and active trading strategies. But TDQN is reactive, not proactive. Reactive means, watching the trend and then taking a decision so you are late to the party . Proactive means predicting the trend and taking decisions before the market follows you. This is due to the limited observation space. It just watches volatility and its direction.

. Proactive means predicting the trend and taking decisions before the market follows you. This is due to the limited observation space. It just watches volatility and its direction. BUT TDQN failed to perform well on highly volatile stocks like TESLA (“TSLA”). A simple B&H (Buy and Hold) performed much better. The lack of proactivity and limited state space leads to poor results.

The most important feature of TDQN is versatility, we don’t need to explicitly say which strategy to use. If there are too much speculation and uncertainty in active markets, TDQN adopts passive strategies else it adopts profitable active strategies.

SOME FUTURE IMPROVEMENTS

Widening the state space would help our model to be BETTER INFORMED and also ameliorate overfitting. News plays a major role in stock markets. Sentiment analysis of news headlines can determine the ensuing trend.

Like in the paper, the authors scraped tweets of Official Reuters account, then extracted the named entities. The named entities were feed to Google Knowledge Graphs to know about the corporation it was talking about.

Like the headline, “Cloud computing will grow two-fold in this year”. The named entity “Cloud Computing” → “Azure” → “MICROSOFT”. So it has a positive sentiment towards Microsoft.

Sentiment analysis has 3 labels, POSITIVE, NEGATIVE and NEUTRAL. But a sentiment score can say about the extent of positiveness or negativeness. Many news won’t affect the stock market, so a score can be tied to the number of shares and issued price.

SOME OTHER NUANCES IN STOCK MARKET

The news, “TESLA is set to join S&P500” came on November 17 and after the announcement, Tesla stock rose to $649.88 from $408.09 — almost 60 per cent. This news has high positiveness and we expect to buy TESLA stocks immediately. So the extent of positiveness can’t be captured by labels alone, hence sentiment score would perform well.

and we expect to buy TESLA stocks immediately. So the extent of positiveness can’t be captured by labels alone, hence sentiment score would perform well. Also, the news “XYZ company cuts 10,000 jobs due to pandemic” has a negative sentiment. But job cuts can lower expenditure and the company can give out more dividends (extra profit handed over to shareholders). This would rally the prices higher. SO YES, STOCK MARKET AND ECONOMY ARE TWO DIFFERENT THINGS.

So in this two-part series, we went over Stock market basics, why RL would make sense in trading and discussed a recent paper based on RL in Algorithmic trading. Hope that this article can be a cornerstone in your future research in this area.
[]
['Deep Learning', 'Finance', 'Reinforcement Learning', 'Algorithmic Trading', 'Financial Engineering']