title: Market-making with reinforcement-learning (SAC)
authors: Alexey Bakshaev
text: 
  
  plain
  
  =-10pt
  
  
  
  
  Market-making with reinforcement-learning (SAC)
      Alexey Bakshaev 
   alex.bakshaev@gmail.com 
      
  ===============================================
  
  
  
  
  The paper explores the application of a continuous action space soft actor-critic (SAC) reinforcement learning model to the area of automated market-making. The reinforcement learning agent receives a simulated flow of client trades, thus accruing a position in an asset, and learns to offset this risk by either hedging at simulated "exchange" spreads or by attracting an offsetting client flow by changing offered client spreads (skewing the offered prices). The question of learning minimum spreads that compensate for the risk of taking the position is being investigated. Finally, the agent is posed with a problem of learning to hedge a blended client trade flow resulting from independent price processes (a "portfolio" position). The position penalty method is introduced to improve the convergence. An Open-AI gym-compatible hedge environment is introduced and the Open AI SAC baseline RL engine is being used as a learning baseline.
    
  
  
  
  
  § INTRODUCTION
  
  Let's assume that our goal is to train our agent in a way that it can perform market making effectively. In this trading mode our agent puts out both the price it is willing its clients to buy at ("client" ask) and sell at ("client" bid), thereby accepting incoming flow of client orders at those prices and profiting from the resulting spread (ask-bid > 0). Client order flow would depend on a number of factors (explained in Environment section) making it non-symmetric, which results in our agent accruing a variable position in a traded asset with time. As the market price of the asset changes with time, this results in revaluation of the position, introducing uncertainty about the price the position would be closed at and hence the final profitability of transactions made over the lifetime of this position. 
  
  To give an intuitive example of this, consider Figure 1 which displays the dynamics of market mid price as well as bid and ask prices our agent is making in the market. 
  
  [14]l0.5
  	Example of unmanaged market risk
  	
      < g r a p h i c s >
  
   
  At a time t=47 the agent buys from a client at a discount to mid market price (at "client bid"), makes a half-spread s1 and takes on a long position in an asset. If the agent was to get out of this position instantaneously at the same time t=47 by selling to another client, it would have made the entire spread of ask-bid. 
   
  Instead, however, the agent closes the position at time t=50  at a lower market at a half-spread s2, and the sum of those half-spreads s1+s2 is not sufficient to offset the drop in mid market price that causes the negative revaluation of the position.
  
  
  
  This raises the need in risk-managing which our hedging agent can achieve by:
  
  
    * putting offsetting orders out to exchange and hereby "hedging" the position at the cost of paying exchange bid/ask spreads
  
    * decreasing client bid/ask spreads to attract more client flow on a chosen side and hereby decreasing the position more quickly. This is known as "skewing" the price under condition of client flow ("demand") being sufficiently elastic to price changes.
  
  Hedging comes at a cost of decreased profitability whereas not hedging introduces the risk of incurring losses in case of unfavorable price movements. Thus, given the net incoming client order flow and the resulting accrued client position, the agent needs to learn to reason out its risk appetite: how much of the position to carry (in a hope it gets offset by the future client order flow) and how much of it to hedge either via an exchange or by offering clients more attractive prices to attract more offsetting client flow. 
  
  In a driftless environment where the asset price is a martingale, the expectation of the market price move with time is zero:
  
      E[S_t_2 - S_t_1] = 0     (t_2 ≥ t_1 ≥ 0)
  
  From this follows the first challenge in setting up an effective market-making ML framework: the agent needs to learn that there is no sense in entering speculative hedge positions that would try to profit from market moves. Instead, in a very basic setting, the agent needs to learn to offset the accrued client position with an opposite exchange (hedger) position. This is an inventory management problem that is very similar to a popular https://gym.openai.com/envs/InvertedPendulum-v2Pendulum environment where ML agent learns to balance the pendulum, or to https://gym.openai.com/envs/MountainCar-v0Mountain car environment where the agent needs to learn to swing the car out of the bottom of the pit.
  
  Both observation space (the size of a position accrued by the agent) and action space (choice of an amount to hedge) are chosen to be continuous, and the asset price is driven by the drift-adjusted Brownian motion. This allows for an infinite action search space and presents a challenging task. An extension of this problem is presented when price skew is added as an additional action available for an agent to take making the overall search space a box in R^2.
  
  Assuming the agent has successfully learnt  to balance the position to zero via choosing hedge / skew actions, we can now pose the following problem. In an environment where hedge spreads and client flow are unknown, can the market making agent learn to offer such spreads that would compensate it for the risk taking? In other words, can an agent learn the price of market risk, measured in terms of offered client spreads?
  
  Another problem that naturally follows is that of portfolio management. The dynamics of the portfolio value is determined by the blended asset price process as well as the blended client trade flow, both of which are driven by the underlying asset price processes, correlation coefficients and asset weights. The learning agent does not know what correlations and asset weights have been used to build the portfolio and needs to work out the effective hedge strategy that will maximize profitablity and minimize the position involved. This makes it an inventory management problem with not just one asset but a portfolio of assets.
  
  
  
  § REINFORCEMENT LEARNING AND SOFT ACTOR-CRITIC (SAC)
  
  Reinforcement learning is becoming increasingly popular in the area of robotics and control automation. In the reinforcement learning setting there is a learning agent that can perpetually (i ∈ [1, N]) interact with the environment by performing an  action_i and then observing the resulting state_i of the environment  and the associated reward r_i. The main purpose of the reward function is to define the end goal of the learning process by rewarding "good" actions and penalizing for "bad" ones. As a result, the agent learns how to sample from the action space in a way that maximizes cumulative discounted return over all steps within a simulation trajectory
  
      R(τ)=∑_i^γ^i r_i
  
  where γ is a discount factor which puts more weight on nearest rewards. The learned probability distribution π for action sampling given reward increment and observation space state is called a policy. The goal of the learning process is to maximize the expected return over all simulation trajectories and to converge onto the optimal policy π^*:
  
      π^* = max_π E_τ∼π[ ∑_i^γ^i r_i ]
  
  If an agent chooses actions according to some fixed policy π starting from a given initial pair of state and action, the resulting expected return is known as "value function", representing the value of this policy. Naturally, as the network learns, the policy gets updated, resulting in a new value function on each step of a trajectory. A Q-function, or "action-value" function is a value function where the first action is being fixed (e.g. sampled off-policy) before the policy is applied on consecutive steps of the trajectory.
  
  Soft actor-critic (SAC) algorithm introduced in <cit.>, <cit.> and <cit.> offered a few methods of improving stability of convergence to the optimal policy in case of continuous action and observation spaces:
  
  	
    * entropy maximization: entropy H(π(s_t)) of the action-generating distribution is being used as a part of the reward function. This leads to joined maximization of expected policy and return which encourages stochastic exploration and stability of convergence:
  	
      π^* = max_π E_τ∼π[ ∑_i^γ^i (R(s_i,a_i,s_i+1) + α H(π(s_t))) ]
  
  	where α is the temperature parameter which determines the trade-off between maximizing return vs maximizing entropy (exploration)
  	
    * actor-critic: the use of separate policy ("actor") and value function ("critic") networks. The policy π and two value (Q(s,a)) functions j=1,2 are being concurrently learned so as to minimize the respective loss functions
  	
      L_j=E[(Q_j(s_i, a_i)-y(r_i,s_i+1))^2]
  
  	where the target y is approximated by sampling actions ã_i+1 from the policy π(s_i+1), conditioned on state return r and next state s_i+1 coming from the replay buffer:
  	
      y(r, s_i+1) = r_i + γ(1-d)(min_j=1,2Q^targ_j(s_i+1,ã_i+1)-αlogπ(ã_i+1|s_i+1))),   ã_i+1∼π(s_i+1)
  
  	The target value function (Q_j^targ) networks are being periodically updated from current value networks Q_j in an exponential average fashion. Within the learning loop (see SAC learning loop) Q-functions are updated by gradient descent on the loss function <ref> and the policy is updated by gradient ascent on the entropy-adjusted Q functions.
  	
    * experience replay: the use of the replay buffer to simulate next states in an off-policy manner. Next actions are simulated from the current policy.
  
  
  These methods alongside with others were used for baseline implementation of SAC algorithm   by https://spinningup.openai.com/en/latest/algorithms/sac.htmlOpenAI. It serves as a baseline RL engine used in this paper. The boxed network is a multi-layered ReLU perceptron with a linear input and output layers. Linear input layer transforms dimensionality from input (observation) space onto the first perceptron layer. Linear output layer transforms dimensionality from the last perceptron layer onto the action space. E.g. when we have the action space defined as position in an asset (dim = 1), and action space defined as hedge amount (dim = 1) and 2 ReLU layers of 2048 units, we will write this as MLP Linear (1) x ReLU (2048) x ReLU (2048) x Linear (1). An additional dimension to all those would be the batch size.
  
  
  
  § ENVIRONMENT
   
  Hedge environment is implemented to be https://gym.openai.comOpen AI Gym-compatible. It makes use of Pytorch https://pytorch.org/docs/stable/tensors.htmlTensor framework for generating price processes and client flows and is intended to be used with a Pytorch-based implementation. Hedge environment provides the standard members  for updating the hedger state based on the action selected by the agent,  for re-setting the simulation and  for displaying the dashboard with hedger statistics. Visualization of the hedging agent performance serves as a very important tool in understanding the progress (or lack thereof) of the hedging agent as well as assessing the overall correctness of the environment set-up.
  
  Project code for this paper is available under: https://github.com/bakalex/autohedger
  
  
  
   §.§ Market data
  
  Asset price process is generated as a standard Euler discretization of a log-normal price process:
  
      Δlog(S_i)=μΔt - 1/2σ^2 Δt + σϵ√(Δt)
  
  
      S_N = S_0 exp∑_iΔlog(S_i)
  
  For the inventory management problem we are trying to solve we don't want our reinforcement learning agent to focus on market drift which would result in the hedger taking speculative directional positions in an asset. Instead, we want it to focus on managing the position arising from the incoming client order flow, so we assume no drift (μ=0) which makes our price process a martingale (eq <ref>).
  
  For a given normally distributed variable ϵ_1 and uncorrelated i.i.d variable ϵ a correlated normally distributed variable ϵ_2 with correlation coefficient ρ is generated as follows:
  
      ε_2 = f(ε_1, ρ) = ρε_1 + (√(1-ρ^2)) ε
  
  Generation of https://pytorch.org/docs/stable/distributions.html#normalnormally distributed variables as well as above discretization is done within Pytorch Tensor framework so that at the beginning of the learning process we have the market data set ready for the entire simulation.
  
  
   §.§ Stochastic spread model
  
  
  There are two types of offered spreads δ^bid, δ^ask defined in the model: "client" spreads that our market-making agent offers clients to trade at (hence, "client" bid / ask) and "hedge" spreads that our agent trades at with an exchange (hence, "hedge" bid/ask). Both those spreads are applied on the top of the same mid price process <ref>. Generated spreads are based on the rolling mean volatility σ_roll_avg of the mid price process plus log-normal stochastic spread add-on ε^L.
  
      S^bid = S - νδ^bid,    S^ask = S + νδ^ask
  
  
      δ^bid=σ_roll_avg+ε^L_1,   δ^ask=σ_roll_avg+ε^L_2
  
  
      σ_roll_avg = StDev(S_i, window)/√(n_steps)
  
  
      ε^L = LogNormal(0, γ S_0 σ/√(n_steps))
  
  where ν is spread multiplier determining the overall magnitude of the spread, γ is spread multiplier that determines the magnitude of the stochastic spread add-on, σ is flat volatility driving the mid process and ε^L_1 and ε^L_2 are log-normal i.i.d variables. 
  
  Given the fat-tailed nature of the log-normal distribution, stochastic spreads δ^bid, δ^ask are being clamped between 0.1 and 2.5 of mean simulation volatility. This helps to avoid unreasonable spikes and bid/ask prices.
  
  
  
   §.§ Client trade flow model
  
  
  Client trade sizes hitting our offered bid and ask prices are simulated as bid and ask https://pytorch.org/docs/stable/distributions.html#torch.distributions.poisson.PoissonPoisson processes. Magnitude of intensity of those processes is assumed to be a function of rolling volatility of the price process. Net trade flow that determines how position changes on each step of the simulation is a function of imbalance of bid/ask https://pytorch.org/docs/stable/distributions.html#torch.distributions.poisson.PoissonPoisson intensities which is assumed to be correlated with the log-price process.
  
      TradeSize^net = TradeSize^bid - TradeSize^ask
  
  
      TradeSize^bid = Poisson(ClientTradeRate * λ̅^bid)
  
  
      TradeSize^ask = Poisson(ClientTradeRate * λ̅^ask)
  
  where ClientTradeRate determines the overall magnitude of the size Poisson process, and λ̅^bid and λ̅^ask are trade flow intensity multipliers. 
  
      ClientTradeRate = C (α + σ_rolling/σ_mean)
  
  where α allows to define the vol-independent "mean" client trade flow, σ_rolling is the average volatility within a rolling window and σ_mean is mean volatility for the entire simulation. C is the flow scaling factor and may be taken to be equal to the initial price of the asset S_0.
  Trade flow intensity multipliers are defined as:
  
      λ̅^bid=max(1-λ, 0)
  
  
      λ̅^ask=max(1+λ, 0)
  
  where λ is the net intensity. It is defined to be correlated with the log-price process as per eq <ref>: 
      λ=β * ε_2 (logS_i, ρ)
   
  where β is sensitivity of net client trade flow towards  log-returns and ρ defines correlation between net intensity process and log-price process. To smoothen out variance in intensity, a rolling average version of the log-price process may be used.
  
  
  
   §.§ Profit and loss (PNL) model
   
  As explained in introduction, our hedging agent accepts client trades at "client" spreads δ^client it offers to clients and then it may choose to hedge the accrued position out exchange "hedge" spreads δ^hedge. Let's assume a client sells to our agent at our offered ask price and then the agent hedges out (sells) the accrued position to the exchange at exchange bid price. In a hypothetical setting where our hedger could have hedged instantaneously, it could have made the client half-spread and paid exchange half-spread, locking in the profit.
  
      PNL = TradeSize (S^client_ask - S^mid) - TradeSize (S^hedge_bid - S^mid) = ( δ^client_ask - δ^hedge_bid) TradeSize
   
  In reality, having accrued the client trade at step t_1, our hedger can only choose to hedge out on the next step t_2 of the simulation, or even later, which results in revaluation of the position at the market mid price between those steps.
  
      PNL = TradeSize (S^client_ask_t_1 - S^mid_t_1) + TradeSize (S^mid_t_2 - S^mid_t_1) - TradeSize (S^hedge_bid_t_2 - S^mid_t_2)
  
  
      = ( δ^client_ask_t_1 - δ^hedge_bid_t_2 + ΔS^mid_t_2-t_1) TradeSize
  
  Given the martingale nature of the simulated price process E ( ΔS^mid_t_2-t_1) = 0 our agent can only be profitable if average offered client spreads are greater than average paid hedge spreads. Note that our reinforcement agent does not know that the price process is a martingale, so as it begins to explore the action space initially, it would be sensitive to realized directional market moves of the mid process. With time, however, the agent needs to learn to focus on managing positions resulting from the client flow instead of trying to capture directional moves of the mid process. Martingale formulation of the price process helps to achieve that. 
  
  
  
   §.§ Elasticity of demand (skewing) model
  
  
  When demand is elastic, we may affect the trade size a client is willing to trade with us by offering a more attractive ("skewed") price than the rest of the market. This can be used as means of position management: e.g. given a long position in an asset, we could lower our offered price to entice the client to buy from us thus decreasing the position. Depending on the marginal profitability of the transaction given the skewed price, such an action may be less expensive compared to executing the similar transaction on the exchange. We will be using the linear model in our experiments:
  
      Δsize = skew * βMaxHedgeSize/ClientTradeRate
  
  where ClientTradeRate is the mean trade rate from TradeFlowModel <ref>. MaxHedgeSize is the maximum hedge trade size constant defined for the bounded action space for the hedger and skew is a bounded action skew ∈ [-1.0,1.0] resulting in the following client price adjustment:
  
      S_adj^bid=S^bid-skew (S^mid - S^bid),   skew ∈ [-1.0,0.0]
  
  
      S_adj^ask=S^ask-skew (S^ask - S^mid),   skew ∈ (0.0,1.0]
  
  So, negative skew means our agent buys more expensively and attracts more client flow on bid side. Positive skew means our agent sells cheaper and attracts more client flow on ask side.
  
  β constant is specified to be large enough for the learning agent to prefer skewing over hedging. Setting β = 0 makes the client flow non-elastic and can be used for price discovery purposes such as finding market price of risk spreads in Market price of risk. 
  
  
  
   §.§ Dummy hedger and simulation dashboard
  
  It is of interest to set up the environment in such a way that the choice to offset the accrued client position with hedge trades should follow naturally from the observed reward. An obvious choice would be to use strategy profitability (PNL) as a basis for the reward function. Hence, we'd be using such parametrization of the environment that wrong hedging choices would result in adverse PNL. In particular, the ratio of mid process volatility σ and stochastic spread multiplier ν is such that for a given realization of the mid price process the decision not to hedge quickly becomes apparent in strategy PNL.
  
  To test this, let's use a dummy hedging strategy that on each step targets to offset the position accrued on the previous step. See 
  
  To compare performance of such a strategy to an alternative not to hedge at all we will be using a dashboard that shows the price process, the net position alongside hedge and client positions, net PNL of the strategy and its constituent components: client PNL, market PNL and hedge PNL. With our dummy strategy on Figure <ref> it is seen that we are offsetting the outstanding client position with the opposite hedge position and are making the difference of client and hedge spreads. It is seen that the strategy looses on unfavorable spikes of market data between consecutive steps which is a result of position revaluation between steps. Compared to unhedged case this strategy exhibits low variance in PNL since market exposure is limited to only unhedged position increment (net client size flow) between steps.
  
  In the case of unhedged client position the variance of strategy PNL is driven by the combined variance of the simulated position and market data process. PNL becomes hostage to directional market moves resulting in worst Sharpe ratios.
  
  
  
  
  
  
  § AUTO-HEDGING A SINGLE ASSET
  
  
  We want the learning agent to carry a position in time only if it is justified by the resulting PNL. As a matter of risk management we don't want the agent to take on excessive positions, so we need to set up a risk limit system. To achieve this goal a position penalty component is introduced to the reward function. This penalty is defined to be an exponential function of the position size, and is intended to make taking bigger positions increasingly more expensive for the agent in reward terms, forcing the agent to justify those by the resulting PNL. For small positions relative to MaxPosLimit the charge would be relatively small but would increase exponentially when position nears or overshoots MaxPosLimit. We can see this penalty as a "spring" that would force the agent to bring the position back to zero unless having this position is profitable enough.
  
  
      Penalty = γ S_0  (e^abs(Position)/MaxPosLimit - 1 ) MaxPosLimit
  
  where γ constant determines how harshly the position penalty w.r.t MaxPosLimit is enforced. To further discourage exploring unpromising trajectories the simulation is terminated when MaxPositionLimit breach exceeds a certain multiple, and additionally penalized reward is recorded.
   
  The following reward function is defined:
  
      Reward_t = ClientPNL_t + HedgePNL_t + MarketPNL_t - Penalty_t
  
  Another technique to improve the convergence to the optimal policy is to limit the action space to a closed interval
  
      a_i = hedgeAmount_i ∈ [-MaxHedgeAmount, MaxHedgeAmount]
  
  where MaxHedgeAmount is defined to be of the same order as mean client trade size. This naturally limits the capacity of the learning agent to explore policies related to speculating on asset price movements and makes it focus on managing net position size instead. 
  
  Network architecture:
  
  	
    * action space: agent action a_i = HedgeSize ∈ [-MaxHedgeSize, MaxHedgeSize]
  	
    * observation space: cumulative net position of client trades and the hedger up until and including the last step NetPosition_i = ∑_0^i(NetClientTradeSize_j+HedgePositionSize_j) ∈ R
  	
    * network layers: MLP of Linear (1) x ReLU (1024) x ReLU (1024) x Linear (1) layers (x batch_size)
  
  The agent successfully learns to hedge the client flow as it could be seen from rewards statistics and the simulation dashboard:
  
  
  [language=bash, label=python-env-setup2,caption=Training the single asset autohedger]
  cd stable-baselines3-autohedger-single/autohedger-single
  python3 autohedger_ml.py
  
  
  
  § AUTOHEDGING WITH SKEW
  
  
  Let's now consider the setting where in addition to hedging on the market the agent can also apply skew i.e. adjust the offered price on one side to attract more offsetting client flow instead of hedging all of the position on the market. For simplicity we will assume that the offsetting client trade flow size is a linear function of skew (see Skewing model <ref>). Architecture layout:
  
  	
    * action space: agent action is a pair of {HedgeSize, Skew} rational numbers within a box bounded by HedgeSize ∈ [-MaxHedgeSize, MaxHedgeSize] , Skew ∈ [-1, 1]
  	
    * observation space: net position
  	
    * network layers: MLP of Linear (1) x ReLU (2048) x ReLU (2048) x Linear (2) layers
  
  It can be seen that the agent learns to make use of the skew in addition to utilizing hedge amounts to hedge the residual. This is useful as it allows us to raise the question of Market price of risk.
  [language=bash, label=python-env-setup2,caption=Training the skew]
  cd stable-baselines3-autohedger-single/autohedger-single
  python3 autohedger_ml_skew.py
  
  To gauge the performance of model with skew vs the model without skew it is useful to introduce a risk-weighted measure of reward. We'll be using a measure similar to Sharpe-ratio of returns, where a return of a single step i is defined as step_return=PNL_i - PNL_i-1:
  
      Sharpe_ratio = E[returns_array]/stdev(returns_array) = ∑_i=1^N_steps(PNL_i-PNL_i-1)/stdev(returns_array)
  
  To compare a non-skew and a skew model side-by-side those models are trained independently, saved, and then used on the same generated market data to compare the performance. 
  
  Depending on the magnitude of the chosen β in the Skewing model it can be seen that skew-based model shows higher Sharpe ratios than a non-skew one. You can run the pre-trained models side-by-side on the same generated market data sets as per listing below:
  [language=bash, label=python-env-setup2,caption=Comparison of models with skew and without]
  cd stable-baselines3-autohedger-single/autohedger-single
  python3 autohedger_ml_learned.py
  
  
  
  
  § MARKET PRICE OF RISK
  
  
  Let's assume a setting where our market-making agent chooses which spreads to charge clients. It is of interest than to determine such minimum mean spread that would compensate the agent for the need of risk-managing a position in an asset resulting from the accumulation of the client trade flow. Such spread would include the cost of heding the position in the market as well as the premium for any non-Brownian behavior of the market (e.g. jumps, volatility clustering, drifts etc). Having those "hurdle" spreads calculated would allow the researcher to build the elasticity of demand spreading model on the top of them and to have a more clear idea of how to price the spreads being charged.
  
  Within our market model the price process is a martingale, so the expected revaluation of a position between any two future timestamps is zero:
  
      Position_t_1 * E[S_t_2 - S_t_1] = 0     (t_2 ≥ t_1 ≥ 0)
  
  Within our PNL model this means that mean client spreads the agent should charge should be equal to mean hedge spreads paid on the exchange to offset the position. Note however that our agent does not know what those hedge spreads are and needs to work them out by interacting with the market model. 
  
  To achieve this goal the following reward function is defined:
  
      Reward_t = max(-abs(ClientPNL_t + HedgePNL_t), 0) + MarketPNL_t - PositionPenalty_t
  
  This reward function makes the learning agent to come up with such client spreads that would offset the cost of hedging the position on the exchange. As before, we introduce a charge on carrying a directional position via PositionPenalty. As the agent learns, at first it incurs higher penalties due to exploring actions leading to larger positions. Once the agent learned to control the position, PositionPenalty becomes a lesser factor letting the agent to bring client spreads in line with the cost of hedging.
  
  We allow the learning agent to control client spreads via skew (see <ref>). In the context of this experiment we assume that client trade flow is not elastic to price changes so β = 0 in eq. <ref>.
   
  
  Architecture layout:
  
  	
    * action space: agent action is a pair of {HedgeSize, Skew} rational numbers within a box bounded by HedgeSize ∈ [-MaxHedgeSize, MaxHedgeSize] , Skew ∈ [-1, 1]
  	
    * observation space: net position
  	
    * network layers: MLP of Linear (1) x ReLU (2048) x ReLU (2048) x Linear (2) layers
  
  As a result of the learning process it is seen how rolling average spread charged by our agent ("maker" spread) converges onto the rolling average hedge spread ("taker" spread).
  
  On a sample PNL dashboard of the learned agent it is seen how the agent tries to keep flat net PNL, at the same time continuing to offset accrued client position with the hedger position.
  
  [language=bash, label=python-env-setup2,caption=Training Price of risk model]
  cd stable-baselines3-autohedger-single/autohedger-single
  python3 autohedger_ml_price_of_risk.py
  
  
  
  
  § MODELING PORTFOLIOS
  
  Let's assume we have two assets S^1 and S^2 which log increments <ref> are correlated as in <ref> with correlation coefficient ρ. For simplicity let's assume that log-increments have the same flat volatility σ, so that the resulting log-normal price processes would have the same variance characteristics. Each of S^1 and S^2 processes would drive its own TradeFlowModel and Stochastic spread model. Naturally, each of those processes would imply its own Poisson intensity of arriving client trade sizes, as well as net imbalance of the arriving client trade flow. "Portfolio" is formed by blending these asset prices and net client trade size flows <ref> with a fixed weight w:
  
      S^blended_mid,bid,ask = w S^1_mid,bid,ask + (1 - w) S^2_mid,bid,ask
  
  
      TradeSize_net^blended = w TradeSize_net^1 + (1 - w) TradeSize_net^2
  
  As a result, the reinforcement learning agent accrues a position which gets incremented with the blended size and the blended price. The agent does not know what asset weights are being used (w and 1-w), neither does it know the correlation coefficient ρ. The agent needs to learn the optimal hedging strategy using hedge amounts for S^1 and S^2. 
  
  At any step the net position ("portfolio") value is defined as the sum of values of the blended client position and its hedges:
  
      Portfolio_value(t) = S^blended_mid(t) Position_Client^blended(t)  + S^1_mid(t) Position_Hedge^S^1(t) + S^2_mid(t) Position_Hedge^S^2(t)
  
  The goal of the learning agent would be to maximize policy PNL whilst keeping the position value in check. The action space is now a R-valued square which bounds the sampled hedge amounts S^1 and S^2. As before, we will be using a penalty function to facilitate risk-management by the learning agent. However, a straightforward application of the exponential penalty function (see single-asset penalty function <ref>) to Portfolio_value would not be as effective. The reason is that by having the two-dimensional action space we are now allowing HedgeAmount^S^1 and HedgeAmount^S^2 to offset each other, thus introducing an additional degree of freedom to the action space. This shows up as a lower net portfolio value and contradicts the convex definition of the blended price <ref>. This could be remediated in a few ways. 
  
  The first approach of making the actions convex is to change the parametrization of the action space:
  
      HedgeAmount^S^1 = w_action * hedge_amount_action
     
  
      HedgeAmount^S^2 = (1 - w_action) * hedge_amount_action
  
  Here actual actions are the weight w and hedge_amount, and the respective S^1 and S^2 hedge amounts are being calculated in a convex manner from those.
  
  A better approach would be to leave it for the learning agent to reason out but to introduce an additional "over-hedge" penalty for when an individual hedge value overshoots the value of the blended client position or has the same sign (so leverages instead of offsetting):
  
      Overhegdge=
      abs(hedge_value + client_pos_value)     hedge overshoots or same sign
      
      0              other cases
  
  Then Overhedge is used as a part of the overall penalty function, where ϕ constant determines penalty tolerance to overhedge and γ determines the tolerance to overall portfolio value closing on MaxPosLimit or overshooting it:
  
      Penalty = γ S_0  (e^abs(Portfolio_value) + ϕabs(Overhedge)/S_0 MaxPosLimit - 1 ) MaxPosLimit
  
  Note that three positions are accrued over the simulation trajectory: the blended client position, S^1 hedge position and S^2 hedge position, where hedge positions are supposed to be offsetting the blended client position. For each of those positions at each step we are tracking spread PNL (paid or received) as well as market revaluation PNL.
   
  Portfolio PNL is defined as trajectory PNL consisting of blended client position PNL (client spread PNL and client position market revaluation) and the sum of S^1 and S^2 hedge positions PNL (hedge spread PNL and hedge position market revaluation). Reward function is  defined as portfolio PNL minus the penalty function:
  
      Reward_portfolio = BlendedClientPosition_PNL + HedgePosition^S^1_PNL + HedgePosition^S^2_PNL - Penalty
  
  Architecture layout:
  
  	
    * action space: agent action is a pair of {HedgeSize_S^1, HedgeSize_S^2} rational numbers within a square box bounded by HedgeSize ∈ [-MaxHedgeSize, MaxHedgeSize] 
  	
    * observation space: S^1 hedger position value S^1_mid Position_Hedge^S^1, S^2 hedger position value S^2_mid Position_Hedge^S^2 and net Portfolio_value(t) (see <ref>) 
  	
    * network layers: MLP of Linear (input: 3) x ReLU (8000) x ReLU (8000) x Linear (output: 2) layers
  
  
  Given the initial correlation ρ=0, the agent is seen to learn to offset the blended incoming client size flow with the hedge positions. The agent seems to be able to deduce the implied weighting of assets within the blended client trade flow, however, more work needs to be done on improving stability on farther epochs. One can find sample dashboards for portfolio learning agents in Portfolio hedging dashboards.
  [language=bash, label=python-env-setup2,caption=Training portfolio hedge model]
  cd stable-baselines3-autohedger-portfolio/autohedger-portfolio
  python3 autohedger_ml_portfolio.py
  
  A slower, more stable version:
  [language=bash, label=python-env-setup2,caption=Training portfolio hedge model - slow]
  cd stable-baselines3-autohedger-portfolio/autohedger-portfolio
  python3 autohedger_ml_portfolio_slow.py
  
  
  
  
  § CONCLUSION
  
  Techniques explored in this paper could be used as a baseline for setting up a position management framework within an automated market-making system. More importantly, however, is that it could be used as a way of blending alpha-generation and position management into one reinforcement learning system. Let's assume that we have a signal generating unit that is listening to the real-life price process as well as other information sources like order book, news feeds etc. Then the signal it generates could be plugged into our reinforcement learning agent. If (as a result of the learning process) our agent finds this signal to be beneficial to the reward function, it would automatically learn to adjust its hedging decisions in line with that. Bridging signal-generation capacity with reinforcement-learning inventory management poses an interesting area of research.
  
  Automating portfolio management decisions by letting the agent infer correlations and weights while learning also proves to be a challenging area to be explored. In a more realistic setting for the portfolio management task we could define a risk-weighted reward function (e.g. based on Sharpe ratio <ref>) and let the agent pick from tens or hundreds of assets which would translate into the same dimensionality for observation and action spaces and would increase the complexity of stochatic policy search problem.
  
  Even in the current set-up more work needs to be done to increase learning stability and avoid intermittent oscillations of HedgeSize action between -MaxHedgeSize and MaxHedgeSize between consecutive steps on farther epochs. This is most likely related to diminished scale of rewards on farther epochs compared to large scale of penalties incurred while starting to train. Methods like normalization of rewards and entropy regularization could be used for this purpose, which would boil down to fine-tuning the entropy maximization parametrization within SAC.
  
  
  
  § APPENDIX
  
  
  
   §.§ Project repository
  
  Project code is available under https://github.com/bakalex/autohedger
  
  
   §.§ Physical environment
  
  Modeling was performed on Amazon EC2 g4dn.2xlarge machine with the use of Deep Learning AMI (Ubuntu 16.04) Version 30.0 (ami-02379288a3b4cbe7b). PyTorch 1.5 CUDA 10.1 is activated with:
  [language=bash, label=python-env-setup,caption=Setting up PyTorch environment on EC2 AMI]
  source activate pytorch_latest_p36
  
  g4dn.2xlarge allowed for research on MLPs as large as 10000 x 10000 x 8000 (x default batch size), but typically nets of no more than 8000 x 8000 units were used. 
  
  Open AI baseline installed with:
  [language=bash, label=python-env-setup2,caption=Installing Open AI baseline]
  cd stable-baselines3
  pip install .
  
  For the convenience of reproducing the results raised in this paper, a version of Open AI is supplied alongside the project code. It is a choice of the researcher to use it or to try a more recent version.
  
  
  
   §.§ SAC algo
  
  
  Soft actor-critic algorithm as listed in https://spinningup.openai.com/en/latest/algorithms/sac.htmlOpen AI Spinning Up. A more succinct version is available in <cit.>.
  
  
  
  
  
  
   §.§ Portfolio hedging dashboards
  
  
  
  
  *
  
  
  
  