Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The non-stationarity characteristic of the solar power renders traditional point forecasting methods to be less useful due to large prediction errors. This in increased uncertainties in the grid operation, thereby negatively affecting the reliability and ing in increased cost of operation. This research paper proposes a unified architecture for multi-time-horizon solar forecasting for short and long-term predictions using Recurrent Neural Networks (RNN). The paper describes an end-to-end pipeline to implement the architecture along with methods to test and validate the performance of the prediction model. The demonstrate that the proposed method based on the unified architecture is effective for multi-horizon solar forecasting and achieves a lower root-mean-squared prediction error compared to the previous best performing methods which use one model for each time-horizon. The proposed method enables multi-horizon forecasts with real-time inputs, which have a high potential for practical applications in the evolving smart grid. Today's power grid has become dynamic in nature mainly because of three changes in the modern grid: 1. Higher penetration level of renewables, 2. Introduction (and rapidly increasing deployment) of storage devices, and 3. Loads becoming active (by participating in demand response). This dynamic modern grid faces the challenge of strong fluctuations due to uncertainty. There is a critical need of gaining real time observability, control, and improving renewable generation forecast accuracy to enhance the resiliency and keep the operational costs sustainable. Independent system operators (ISOs) with higher renewable penetration on the grid have already been facing challenges with the uncertainties associated with short-term forecasting errors. In year 2016, California ISO doubled its frequency regulation service requirements (causing a sharp rise in the cost of requirements) to manage the recurring short-term forecasting errors in renewable generation BID0. The Western Electricity Coordinating Council (WECC) could achieve $5 billion savings per year by integrating wind and solar forecasts into unit commitment, according to the study conducted by Lew et al BID1. Thus, it is clear that the increased grid penetration levels of solar with its inherent variability (a combination of intermittence, high-frequency and non-stationarity) poses problems with grid reliability and cost of operating the grid on various time-scales. For example, day-ahead solar forecast accuracy plays a significant role in the effectiveness of Unit Commitment (UC); very-short-term solar forecasts errors due to fluctuations caused by the passing clouds lead to sudden changes in PV plant outputs that can cause strain to the grid by inducing voltage-flickers and real-time balancing issues. Thus, solar power generation forecast becomes an area of paramount research, as the need for robust forecast for all timescales (weekly, day-ahead, hourly and intra-hour) is critical for effectively incorporating increasing amount of solar energy resources at a global level and contributing to the evolution of the smart grid. Moreover, improving the accuracy of solar forecast is one of the lowest cost methods of efficiently integrating solar energy into the grid. The rest of the paper is organized as follows. The literature is reviewed and the significant shortcomings of the current forecasting approaches are recognized in Section II. Section II further introduces the capabilities of the proposed unified architecture and the novel algorithm to fill in the gap between the need to improve the forecasting techniques and the existing approaches. Section III introduces the proposed unified architecture based on RNN and the training algorithms utilized for implementing the neural network. Exploratory data analysis, evaluation metric and structure of input data, and the proposed algorithm are presented in Section IV. Section V discusses the and their interpretation. The paper is concluded with Section VI, which also identifies the future avenue of research in this method of solar forecasting.. Forecasting methods which have been used for renewable generation and electric load forecasting prediction can be mainly classified into five categories: 1) Regressive methods, such as Autoregressive (AR), AR integrated moving average (ARIMA), and exponential smoothing (ES) models BID2, BID3, BID4, nonlinear stationary models; 2) Artificial Intelligence (AI) techniques, such as Artificial Neural Networks (ANN) BID5 - BID9, k-nearest neighbors BID10 - BID13, fuzzy logic systems (FLSs) BID14 - BID16; 3) Numerical Weather Prediction (NWP) BID17; 4) Sensing (remote and local) BID18. 5) Hybrid models, such as neuro-fuzzy systems BID19 - BID20, ANN and satellite derived cloud indices BID21, to name a few. Numerical Weather Prediction (NWP) models are based on physical laws of motion and thermodynamics that govern the weather. For the places where ground data is not available, NWP models are powerful tools to forecast solar radiation. However, they pose significant limitations in predicting the precise position and extent of cloud fields due to their relatively coarse spatial Multi-time-horizon Solar Forecasting Using Recurrent Neural Network T resolution. Their inability to resolve the micro-scale physics associated with cloud formation renders them with relatively large error in terms of cloud prediction accuracy. In order to mitigate this limitation, NWPs are simulated at regional level (called Regional NWP) models, downscaled to derive improved site-specific forecasts. NWP has another limitation of temporal-resolution. The timescale of output variables of NWP model is from 3 hour -6 hours for the Global Forecast System (GFS) and 1-hour (for mesoscale model), which is not useful for predicting the ramp-rate and very-short-term output fluctuations. For the areas where the previous ground-based measurement are not available, satellite based irradiance measurement proves to be a useful tool BID21. The images from satellite are used to analyze the time evolution of air mass by the superimposition of images of the same area. Radiometer installed in the satellite records the radiance, states of the atmosphere (clear sky to overcast) impacts the radiance. Satellite sensing has the main limitation of determining an accurate set point for the radiance value under clear sky conditions and under dense cloudiness condition from every pixel and every image. Another limitation of solar irradiance forecasting using remotes sensing with satellite is the algorithms that are classified as empirical or statistical BID22 - BID23. These algorithms are based on simple statistical regression between surface measurements and satellite information and do not need accurate information of the parameters that model the solar radiation attenuation through the atmosphere. So the ground-based solar data is required for these satellite statistical algorithms anyway. The aforementioned limitations of NWP and sensing models have steered the short-term solar forecasting research towards timeseries analysis using statistical models and more recently AI techniques. Statistical techniques can mainly be classified as BID24: 1) Linear stationary models (Autoregressive models, Moving Average models, Mixed Autoregressive Moving Average Models, and Mixed Autoregressive Moving Average models with exogenous variables); 2) Nonlinear stationary models; 3) Linear nonstationary models (Autoregressive integrated moving average models and Autoregressive integrated moving average models with exogenous variables). Though these conventional statistical techniques provide a number of advantages over NWP and sensing methods, but these are often limited by strict assumptions of normality, linearity, variable independence. Artificial Neural Networks (ANN) are able to represent complex non-linear behaviors in higher dimensional settings. When exogenous variables like humidity, temperature and pressure are considered in the process of solar forecasting -ANNs act as universal function approximators to model the complex non-linear relationships between these variables and their relationship with the Global Horizontal Irradiance (GHI). An ANN with multiple hidden layers can be called A Deep Neural Network (DNN). With the advancements in computational capabilities, DNNs have proven to be effective and efficient in solving complex problems in many fields including image recognition, automatic speech recognition and natural language processing etc BID25. Although, feed-forward neural network models have been used for solar forecasting problem, use of Recurrent Neural Networks (RNN) models have not been explored yet, to the best of author's knowledge. RNN is a class of ANN that capture the dynamics of sequences using directed cyclic feedback connections BID26. Feedforward neural networks rely on the assumption of independence among the data points or samples. The entire state of the network is lost after processing each data point (sample). Unlike vanilla feedforward neural networks, recurrent neural networks (RNNs) exhibit dynamic temporal behavior by using their internal memory to process arbitrary sequences if inputs, which can be harnessed in predicting the irradiance for the next time step by considering the input from many previous times steps. Recent advances in parallelism, network architectures, optimization techniques, and graphics processing units (GPUs) have enabled successful large-scale learning with RNNs overcoming their traditional limitations of being difficult to train due to having millions of parameters. Several methods have been proposed for solar forecasting in the past but most of them were modeled for a particular timehorizon and no single model performed well compared to others for multi-time-horizon forecasting/prediction. In addition, the state-of-the-art methods used for solar forecasting primarily focus on averaged rather than instantaneous forecasts. This paper proposes two approaches using RNNs. 1). A single algorithm that is capable of being trained to output solar forecast for 1-hour or 2-hour or 3-hour, or for 4-hour time horizons. ii) A unified architecture that can predict/forecast the solar irradiance for multitime-horizons; for example, the trained model can predict/forecast the solar irradiance values for the 1-hour, 2-hour, 3-hour and 4-hour. Our proposed method is capable of taking a time-series data as the input and provides predictions with a forward inference time in the order of milliseconds, enabling real-time forecasts based on live measured data. This offers a great value for industrial applications that require real-time multi-time-horizon forecasting for overcoming the current operational challenges with high penetration of renewable source of energy. The RNN resembles a feedforward neural network except for an additional directed edges. These edges span adjacent time steps, introducing the notion of temporal component to the model. Theoretically, the RNN architecture enables the network to make use of past information in sequential data. The input to an RNN is a sequence, and its target can be a sequence or a single value. An input sequence is denoted by (x BID0, x BID1,...x (T) ), where each sample/data-point x (t) is a real valued vector. The target sequence is denoted by (y BID0, y BID1,...y (T) ) and the predicted target data-point is denoted by (y BID0, y BID1,...y (T) ). There are three dimensions to the input of the RNN (shown in FIG0): 1) Mini-batch Size; 2) Number of columns in the vector per time-step; and 3) Number of time-steps. Mini-batch size is the samplelength (data-points in the time-series). Number of columns are the input features in the input vector. The number of time-steps is the differentiating factor of RNN, which unfolds the input vector over time. In a typical multilayer feedforward neural network, the input vector is fed to the neurons at the input layer, which then gets multiplied by the activation function to produce the intermediate output of the neuron, this output then becomes the input to the neuron in the next layer. The net input (denoted by input_sumi) to this neuron belonging to the next layer is the weight on connections (W) multiplied by previous neuron's output with the bias term, as shown in Equation BID0. An activation function (denoted by g) is then applied to the input_sumi to produce the output from the neuron Equation 2 and 3. DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 where, Whx is the conventional weight matrix between the input and the hidden layer and Whh is the recurrent weights matrix between the hidden layer and itself at adjacent time steps. bh and by are bias parameters. The proposed architecture uses Rectified Linear Units (ReLU) as the activation function. The network unfolds the given input at time t as shown in FIG0. The unfolded network is trained across the time steps using an algorithm called backpropagation through time (BPTT) BID27. The loss function used for this regression problem is Mean Squared Loss (MSE). The loss function finds the error between the target output and the predicted output from the network. Gradients are computed using back-propagation-through time BID27 and the stochastic gradient descent optimizer is used to update the weights so as to minimize the loss. The RMSE is calculated for benchmarking purposes. The motivation to use a RNN is to identify and learn the complex relationship between sequences of various exogenous variables and their combined impact on the solar irradiance. This, in the author's view, enables the algorithm to recognize non-linear contributing factors for example the atmospheric conditions, which may lead to cloud formation in nearby time horizon. This is one of the reasons the prediction RMSE is lower in the proposed approach compared to other reported approaches. The algorithm and the unified architecture developed in this paper were trained and tested on data from the NOAA's SURFRAD BID30 sites similar to the previous works in the literature BID28 BID31. The input features are: downwelling global solar (Watts/m^2), upwelling global solar (Watts/ m^2), direct-normal solar (Watts/ m^2), downwelling diffuse solar (Watts/ m^2), downwelling thermal infrared (Watts/ m^2), downwelling IR case temp. (K), downwelling IR dome temp. (K), upwelling thermal infrared (Watts/ m^2), upwelling IR case temp. (K), upwelling IR dome temp. (K), global UVB (milliWatts/ m^2), photosynthetically active radiation (Watts/ m^2), net solar (dw_solar -uw_solar) (Watts/ m^2), net infrared (dw_ir -uw_ir) (Watts/ m^2), net radiation (netsolar+netir) (Watts/ m^2), 10-meter air temperature (C), relative humidity (%), wind speed (ms^1), wind direction (degrees, clockwise from north), and station pressure (mb). According to Dobbs BID28, Global downwelling solar measurements best represent the Global Horizontal Irradiance (GHI) at the SURFRAD sites, which was validated in this paper through exploratory data analysis.[Figure 3] shows the daily averages of Clear Sky GHI and global downwelling solar at SURFRAD site for a year, both the variable follow the same trend. [Figure 4] shows that both these variables are positively correlated. Bird model is used to calculate clear sky GHI BID29. At time t, clear sky GHI is denoted by, representing the theoretical GHI at time t assuming zero cloud coverage. At time t, the ratio between the instantaneously observed and the theoretical maximum is called clear sky index, denoted by , this parameter is introduced in BID28. The algorithm uses Mean Squared Error (MSE) as a measure to find the difference between the target and the output neural network is producing during the training process, this is shown in Equation 7. Later in the process, for the purpose of benchmarking, the Root Mean Squared Error is calculated by taking square root of the MSE values. DISPLAYFORM0 Where is Y a vector of target values and Y is vector of n predicted values. The flowchart of the proposed Unified Recurrent Neural Network Architecture algorithm is shown in Figure 4. The overall algorithm can be divided into three main blocks. The site-specific data is imported and clear sky global horizontal irradiance values for that site are obtained from the Bird Model. The two are merged. Dataset is split into train and testing sets. The clear sky index parameter is created as the ratio of observed global downwelling solar (Watts /m^2) and GHI (Watts /m^2). Kt is a dimensionless parameter. The missing values in the data are replaced by the mean and/or the neighborhood values. Exploratory data analysis is conducted to identify and eliminate extreme outliers in order to normalize the data. Recurrent Neural Network model is defined. The model is instantiated by specifying the architectural parameters: input dimension (number of nodes at the input layer), hidden dimension (number of nodes in the hidden layer), layer dimension (number of hidden layers) and output dimension. Sequence length, which unfolds as time-steps is also defined here along with the batch size. The model is trained and test by iterating through the whole dataset based on pre-set number of epochs. Once the training and testing is over, the stored MSE is first de-normalized and then it is used to calculate RMSE. If the RMSE is not satisfactory the hyperparameters (learning rate and number of epochs) are tuned and the model is trained again. When a satisfactory (or as expected) RMSE is achieved, the algorithm terminates. The algorithm is trained using the data for the year 2010 and 2011 from the SURFRAD observations sites in Boulder, CO; Desert Rock, NV; Fort Peck, MT; Sioux Falls, SD; Bondville, IL; Goodwin Creek, MS; and Penn State, PA. The test year for each respective site was chosen to be 2009 for the purpose of benchmarking against BID28 and other previously reported in the literature. Results from the two methods proposed in this paper are presented below: The first method uses the proposed RNN architecture and algorithm to predict for 1-hour, 2-hour, 3-hour and 4-hour time horizons, independently. In other words, four independent models are developed for 1-hour, 2-hour, 3-hour and 4-hour predictions for each of the seven SURFRAD sites. The RMSE values in TAB1 show that the proposed architecture and algorithm has lower RMSE values for all four forecasting horizons and all the seven sites, compared to the best RMSE values reported in BID28 from a suite of other machine learning algorithms (Random Forests, Support Vector Machine, Gradient Boosting and vanilla FeedForward). In this method, the architecture predicts for all four forecasting time horizons (1-hour, 2-hour, 3-hour and 4-hour) in parallel; i.e. one model per SURFRAD site is developed which makes predictions for all the four time horizons. This method is the multitime-horizon implementation of the proposed architecture. None of the methods discussed in the literature have been shown to be capable of producing multi-time-horizon predictions. TAB1 and 2017. To quantify the overall performance of the predictive model in terms of its combined forecasting accuracy for all four forecasting horizons, the mean of the RMSE values is calculated. Although, even the best RMSE values reported in the literature (for example in BID28) were for a single time horizon forecast at a time, the proposed method achieves a significantly lower RMSE in predictions across all the short-term (1 to 4 hours) forecasting time-horizons as seen in Table 2. Capability to predict for multi-time-horizons makes the proposed method very relevant for industry applications. The real-time data can be fed to the RNN and due to its lower forward inference time, predictions can be made for multiple time horizons. Short-term solar forecasting is of great importance for optimizing the operational efficiencies of smart grids, as the uncertainties in the power systems are ever-increasing, spanning from the generation arena to the demand-side domain. A number of methods and applications have been developed for solar forecasting, with some level of predictive success. The main limitation of the approaches developed so far is their specificity with a given temporal and/or spatial resolution. For predictive analysis problems, the field of AI has become promising with the recent advances in optimization techniques, parallelism, and GPUs. AI (especially deep neural networks) thrives on data, and with decreasing cost of sensor and measurement equipment, plethora of solar data is getting available. Data availability is only going to keep increasing in the coming years. The proposed novel Unified Recurrent Neural Network Architecture harnesses the power of AI to form a high-fidelity solar forecasting engine. This architecture has the potential to be implemented as a complete forecasting system, which spans the entire spectrum of spatial and temporal horizons with a capability to take real-time data as input to produce multi-time-scale (intra-hour, hourly and day-ahead scales) predictions. In addition, the proposed algorithm outperforms traditional Machine Learning methods in terms of quality of the forecast and its low forward inference time makes it a robust real-time solar forecasting engine. Although a deeper neural network will have more capacity, we experimentally observed that it leads to high variance in the model and therefore a reduced generalization power for the particular problem dealt in this paper. The performance of the proposed method can be further improved in several ways including hyper-parameter tuning and architectural changes like the activation functions used or the type of layers. Extension of the proposed architecture with LSTM cells and intra-hour forecasting horizons are potential future research avenues in this domain.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper proposes a Unified Recurrent Neural Network Architecture for  short-term multi-time-horizon solar forecasting and validates the forecast performance gains over the previously reported methods

Introduce an information theoretic viewpoint on the behavior of deep networks optimization processes and their generalization abilities


Question: Represent the scientific article to find a short TLDR

Question: The ResNet and the batch-normalization (BN) achieved high performance even when only a few labeled data are available. However, the reasons for its high performance are unclear. To clear the reasons, we analyzed the effect of the skip-connection in ResNet and the BN on the data separation ability, which is an important ability for the classification problem. Our show that, in the multilayer perceptron with randomly initialized weights, the angle between two input vectors converges to zero in an exponential order of its depth, that the skip-connection makes this exponential decrease into a sub-exponential decrease, and that the BN relaxes this sub-exponential decrease into a reciprocal decrease. Moreover, our analysis shows that the preservation of the angle at initialization encourages trained neural networks to separate points from different classes. These imply that the skip-connection and the BN improve the data separation ability and achieve high performance even when only a few labeled data are available. The architecture of a neural network heavily affects its performance especially when only a few labeled data are available. The most famous example of one such architecture is the convolutional neural network (CNN) BID6. Even when convolutional layers of CNN were randomly initialized and kept fixed and only the last fully-connected layer was trained, it achieved a competitive performance compared with the traditional CNN BID5 BID14. Recent other examples are the ResNet BID3 and the batch-normalization (BN) BID4. The ResNet and the BN are widely used in few-shot learning problems and achieved high performance BID8 BID9.One reason for the success of neural networks is that their architectures enable its feature vector to capture prior knowledge about the problem. The convolutional layer of CNN enable its feature vector to capture statistical properties of data such as the shift invariance and the compositionality through local features, which present in images BID13. However, effects of the skip-connection in ResNet and the BN on its feature vector are still unclear. To clear the effects of the skip-connection and the BN, we analyzed the transformations of input vectors by the multilayer perceptron, the ResNet, and the ResNet with BN. Our show that the skip-connection and the BN preserve the angle between input vectors. This preservation of the angle is a desirable ability for the classification problem because the last output layer should separate points from different classes and input vectors in different classes have a large angle BID11 BID10. Moreover, our analysis shows that the preservation of the angle at initialization encourages trained neural networks to separate points from different classes. These imply that the skip-connection and the BN improve the data separation ability and achieve high performance even when only a few labeled data are available. We consider the following L layers neural networks, which transform an input vector x ∈ R D into a new feature vector h L ∈ R D through layers. Let h 0 = x and ϕ(·) = max(0, ·) be the ReLU activation function. ResNet BID12 BID1: DISPLAYFORM0 ResNet with batch-normalization (BN): DISPLAYFORM1 where the expectation is taken under the distribution of input vectors in the mini-batch of the stochastic gradient descent (SGD). Without loss of generality, we assume that the variance of input vectors in the mini-batch is one, Var (DISPLAYFORM2 We analyzed the average behavior of these neural networks when the weights were randomly initialized as follows. In the MLP, the weights were initialized by the He initialization BID2 because the activation function is the ReLU function. DISPLAYFORM3 In the ResNet and the ResNet with BN, the first internal weights were initialized by the He initialization, but the second internal weights were initialized by the Xavier initialization BID0 because the second internal activation function is the identity. DISPLAYFORM4 We analyzed the transformation of input vectors through hidden layers of the neural networks. Now we define the quantity studied in this paper., we define the angle and the cosine similarity, DISPLAYFORM0 where DISPLAYFORM1 is the length of the feature vector and DISPLAYFORM2 is the inner product between the pair of the feature vectors. Note that the expectation is taken under the probability distribution of initial weights. We derived the recurrence relation of the angle TAB0 . Its plot FIG0 shows that the MLP contracts the angle between input vectors, which is an undesirable property for the classification DISPLAYFORM0 DISPLAYFORM1 Figure 2: Transformation of the angle between input vectors. We plotted the mean and the standard deviation of the angle over 10 randomly initialized parameters.problem, and that the skip-connection in ResNet and the BN relax this contraction. Numerical simulations (Fig.2) on the MNIST dataset validated our analysis. TAB0 gives us the clear interpretation how the skip-connection in ResNet and the BN preserve the angle between input vectors. The ReLU activation function contracts the angle because the ReLU activation function truncates negative value of its input. The skip-connection bypasses the ReLU activation function and thus reduces the effect of the ReLU activation function to the half. Moreover, the BN reduces the effect of the ReLU activation function to the reciprocal of the depth. We derived the dynamics of the angle through layers (Table 2) by applying the recurrence relation of the angle TAB0 iteratively and using the fact that, if θ is small, arccos(ψ(θ)) can be well approximated by the linear function, a · θ where a < 1 is constant. Table 2 shows that, in the MLP with randomly initialized weights, the angle between input vectors converges to zero in an exponential order of its depth, that the skip-connection in ResNet makes this exponential decrease into a sub-exponential decrease, and that the BN relaxes this sub-exponential decrease into a reciprocal decrease. In other words, the skip-connection in ResNet and the BN preserve the angle between input vectors. Numerical simulation (Fig.3) on the MNIST dataset validated our analysis. Table 2: Dynamics of the angle through layers. DISPLAYFORM0 Figure 3: Dynamics of the angle. We plotted the mean and the standard deviation of the angle over 10 randomly initialized parameters. A desirable ability of the neural network for the classification problem is to separate points from different classes. However, our show that the randomly initialized neural networks contract the angle between input vectors from different classes. Our analysis provide us with an insight how training tackle this problem. We can show that the cosine similarity c l+1 (n, m) is proportional to DISPLAYFORM0 where θ is a parameter we can control by training. Its plot FIG1 implies that training makes small angles smaller and large angles larger by taking the extreme value of θ like 0 or π. In order to validate this insight, we stacked the softmax layer on top of an 1 layer MLP and trained this model by the SGD with 100 labeled examples in the MNIST dataset. Fig.5 shows the change of the angles of feature vectors by training, which validated our insight. The above discussion also shows the relationship between training and the preservation of the angle. The angle of feature vectors at high layer of the initialized MLP is small, which implies that training doesn't take extreme value of θ and doesn't separate points from different classes. On the other hand, the skip-connection and the BN preserve the angle between input vectors even at high layer. Thus, training takes extreme value of θ and separates points from different classes. Numerical simulations (Fig.6), which is the same as the previous one, validated our insight. The ResNet and the BN achieved high performance even when only a few labeled data are available. To clear the reasons for its high performance, we analyzed effects of the skip-connection in ResNet and the BN on the transformation of input vectors through layers. Our show that the skip-connection and the BN preserve the angle between input vectors, which is a desirable ability for the classification problem. Moreover, our analysis shows that the preservation of the angle at initialization encourages trained neural networks to separate points from different classes. These imply that the skip-connection and the BN improve the data separation ability and achieve high performance even when only a few labeled data are available.

Represent the short summary for finding the corresponding scientific article (long)

The Skip-connection in ResNet and the batch-normalization improve the data separation ability and help to train a deep neural network.

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.


Question: Represent the article for finding a single-sentence TLDR!

Question: Learning representations of data is an important issue in machine learning. Though GAN has led to significant improvements in the data representations, it still has several problems such as unstable training, hidden manifold of data, and huge computational overhead. GAN tends to produce the data simply without any information about the manifold of the data, which hinders from controlling desired features to generate. Moreover, most of GAN’s have a large size of manifold, ing in poor scalability. In this paper, we propose a novel GAN to control the latent semantic representation, called LSC-GAN, which allows us to produce desired data to generate and learns a representation of the data efficiently. Unlike the conventional GAN models with hidden distribution of latent space, we define the distributions explicitly in advance that are trained to generate the data based on the corresponding features by inputting the latent variables that follow the distribution. As the larger scale of latent space caused by deploying various distributions in one latent space makes training unstable while maintaining the dimension of latent space, we need to separate the process of defining the distributions explicitly and operation of generation. We prove that a VAE is proper for the former and modify a loss function of VAE to map the data into the pre-defined latent space so as to locate the reconstructed data as close to the input data according to its characteristics. Moreover, we add the KL divergence to the loss function of LSC-GAN to include this process. The decoder of VAE, which generates the data with the corresponding features from the pre-defined latent space, is used as the generator of the LSC-GAN. Several experiments on the CelebA dataset are conducted to verify the usefulness of the proposed method to generate desired data stably and efficiently, achieving a high compression ratio that can hold about 24 pixels of information in each dimension of latent space. Besides, our model learns the reverse of features such as not laughing (rather frowning) only with data of ordinary and smiling facial expression. Developing generative model is a crucial issue in artificial intelligence. Creativity was a human proprietary, but many recent studies have attempted to make machines to mimic it. There has been an extensive research on generating data and one of them, generative adversarial network (GAN), has led to significant achievements, which might be helpful to deep learning model because, in general, lots of data in good performance BID12. Many approaches to creating data as better quality as possible have been studied: for example, variational auto-encoder (VAE) BID9 and GAN BID4. The former constructs an explicit density, ing in an explicit likelihood which can be maximized, and the latter constructs an implicit density BID3. Both can generate data from manifold which is hidden to us so that we cannot control the kind of data that we generate. Because it is costly to structure data manually, we need not only data generation but also automatically structuring data. Generative models produce only data from latent variable without any other information so that we cannot control what we want to generate. To cope with this problem, the previous research generated data first and found distributions of features on latent space by investigating the model with data, since the manifold of data is hidden in generative models. This latent space is deceptive for finding an area which represents a specific feature of our interest; it would Figure 1: Examples of the manifold. Left: a complex manifold which can be seen in general models, Right: a relatively simple manifold in the proposed model. The midpoint M of A and B can be easily calculated in the right manifold, but not in the left one. The midpoint of A and B is computed as N in the left manifold, which is incorrect. take a long time even if we can find that area. Besides, in the most of research, generative models had a large latent space, ing in a low compression rate which leads to poor scalability. To work out these problems, we propose a model which can generate the data whose type is what we want and learn a representation of data with a higher compression rate, as well. Our model is based on VAE and GAN. We pre-define distributions corresponding to each feature and modify the loss function of VAE so as to generate the data from the latent variable which follows the specific distribution according to its features. However, this method makes the latent space to become a more complex multimodal distribution which contains many distributions, ing in an instability in training the LSC-GAN. We prove that this problem can be solved and even made more efficiently by using an auto-encoder model with the theorem in Section 3. Although the proposed model compresses the data into small manifold, it is well-defined with Euclidean distance as shown in Fig. 1, which compares the manifolds in general models and in our model. The distance can be calculated with Euclidean distance in adjacent points but not in far points at the left manifold in Fig. 1. However, in the right manifold, we can calculate the distance between points regardless of the distance of them, where we can recognize the manifold more easily as shown in the left side. Thanks to a relatively simple manifold, it can produce neutral features regardless of their location in latent space, so that all features can be said as independent to each other. Our main contribution is summarized as follows.• We propose a method to improve the stability of a LSC-GAN with LSC-VAE by performing the weight initialization, and prove it theoretically.• We achieve conditional generation without additional parameters by controlling the latent space itself, rather than adding additional inputs like the existing model for condition generation.• We propose a novel model that automatically learns the ability to process data continuously through latent space control.• Finally, we achieve an efficient compression rate with LSC-GAN based on weight initialization of LSC-VAE.The rest of the paper is organized as follows. Section 2 reviews the related works and the proposed LSC-GAN model is illustrated in Section 3. In Section 4, we evaluate the performance of the proposed method with some generated data. The and discussion are presented in Section 5. Many research works have been conducted to generate data such as text, grammar, and images BID24 BID10 BID2. We divide the approaches for data generation into three categories: only generation, conditioned generation, and transforming data to have different features. Several researchers proposed generative models of VAE and GAN BID9 BID4. These are basis of the generative models. Both use maximum likelihood approach, but they have different policies to construct density: explicitly and implicitly. There are lots of variations of these models. BID17 constructed deep convolutional GAN (DCGAN) with convolutional neural networks (CNN) for improving the performance with the fact that CNN had been huge adoption in computer vision applications. BID26 introduced energy-based GAN (EBGAN) using autoencoder in discriminator. BID5 BID6 b) proposed transferred encoder-decoder GAN (TED-GAN) for stabilizing process of training GAN and used it to classify the data. These studies focused on high productivity in generation so that they could not control the type of generated data. Recently, some researchers began to set conditions on the data they generate. BID20 and BID23 inputted data and conditions together into VAE and generated data whose type is what they want, called conditional VAE (CVAE). van den Oord et al. FORMULA0 set discrete embedding space for generating a specific data using vector quantized variational auto-encoder (VQ-VAE), but because of discrete space, they could not control latent space continuously. Larsen et al. FORMULA0 used both VAE and GAN in one generative model. As they just mixed two models and did not analyzed a latent space, so that the manifold of data was hidden to us. To generate image with a specific feature, they extracted a visual attribute vector which is a mean of vector in latent space. BID15 inputted not only data but also conditions into GAN to create data that we want, called conditional GAN (CGAN). BID0 used mutual information for inducing latent codes (InfoGAN) and BID16 added a condition network that tells the generator what to generate (PPGN). These two models needed an additional input to generate the type of data we want. These studies make us to generate data with condition, but we still do not know about latent space and it is hard to find the location of a specific feature in the latent space. Therefore, we propose a model that learns to generate concrete features that we want from the latent space determined when LSC-VAE is trained. Some studies attempted to transfer the given data to others which have different features or even in different domain. BID21 proposed disentangled representation learning GAN (DRGAN) for pose-invariant face recognition. BID18 b) tried matching latent space of text and images and finally they translated text to image. BID25 also translated text to image and generated photo-realistic images conditioned on text by stacking models (StackGAN). BID27 and BID8 discovered cross-domain relations with CycleGAN and DiscoGAN. They can translate art style, face features, and bags to shoes. While other models could only do one conversion task, BID1 proposed StarGAN that could do multiple translation tasks with one model. These studies have been conducted to transform the data into those in other domains. However, they could not generate new data without input data. In addition, the size of latent space of most of them was too large. We aim to generate conditioned data even with a small size of latent space. In this section, we present a method to generate the data with the corresponding characteristics by inputting the latent variable which follows the specific distribution in latent space. As the instability caused by the larger scale of latent space in this process, we use the modified VAE, called LSC-VAE 1. As shown in Fig. 2(a), we train the LSC-VAE with L prior for the data to be projected by the encoder into the desired position in the latent space according to the characteristics of the data. The trained decoder of the LSC-VAE is used as a generator of LSC-GAN so that the LSC-GAN generates the data with the corresponding features by using latent variables sampled from a specific distribution. The proposed model is divided into two phases: initializing latent space (Fig. 2(a) ) and generating data (Fig. 2(b) ). In the first phase, latent semantic controlling VAE (LSC-VAE) is trained to project data into a specific location of latent space according to its features, and it learns to reconstruct data which is compressed. The decoder of LSC-VAE is used in the generator (G) of LSC-GAN in the second phase. G and discriminator (D) are trained simultaneously so that G can produce data similar to real data as much as possible and that D can distinguish the real from the fake. The architecture of the generation process is shown in Fig. 2(b). Auto-encoder has been traditionally used to represent manifold without supervision. In particular, VAE, one type of auto-encoders, is one of the most popular approaches to unsupervised learning of complicated distributions. Since any supervision is not in training process, the manifold constructed is hidden to us. As we mentioned in Section 1, this is usually too complex to generate the conditioned Figure 2: (a) The process of pre-defining a latent space. The LSC-VAE is trained to project the data into the appropriate position on latent space. (b) Generating process of the proposed method. The latent space is pre-defined in the process of (a).data. Therefore, we allow LSC-VAE to learn a representation of data with supervision. It compresses data into a particular place on latent space according to its features. The proposed model consists of two modules that encode a data x i to a latent representation z i and decode it back to the data space, respectively. DISPLAYFORM0 Index i means a feature which is included in data x and latent space z. The encoder is regularized by imposing a prior over the latent distribution P (z). In general, z ∼ N (0, I) is chosen, but we choose z i ∼ N (µ i, I)for controlling latent space. In addition, if we want to produce data which has multiple features i, j, we generate data from z ij ∼ N (µ i + µ j, I) 2. The loss function of LSC-VAE is as follows. DISPLAYFORM1 where D KL is the Kullback-Leibler divergence. The first term of equation 3 is related to reconstruction error and the second term is related to appropriate projection of data to the latent space. For example, when LSC-VAE projects the data with i− and j−features into the latent space, it is trained to map the data into the pre-defined latent space (N (µ i + µ j, I)) with L prior in equation 3 so as to locate the reconstructed data as similar to the input data according to its characteristics using L V AE. Therefore, LSC-VAE can be used in initializing GAN and it is demonstrated that LSC-VAE is valid and efficient for LSC-GAN in the next section. GAN has led to significant improvements in data generation BID4. The basic training process of GAN is to adversely interact and simultaneously train G and D. However, because the original GAN has a critical problem, unstable process of training BID17, the least squares GAN (LS-GAN) is proposed to reduce the gap between the distributions of real data and fake data by BID14. FIG1 shows the objective function of the LS-GAN. p data is the probability distribution of the real data. G(z) is generated data from a probability distribution p z, and it is distinguished from the real by D. DISPLAYFORM0 The main differences of the proposed model with VAE-GAN and LS-GAN is that LSC-GAN is based on LSC-VAE for initializing a latent space to control it. To produce the type of data we want, we just input latent variable z i ∼ N (µ i, I) to G, if the data has i−feature. Besides, we add the encoder of LSC-VAE into LSC-GAN to make sure that the generated data actually have the desired features. The encoder projects back to latent space so as to be trained to minimize the difference between latent space where data is generated and the space where the compressed data is projected. Equation 5 is about loss of D and loss of encoder and G. DISPLAYFORM1 Since the original GAN has disadvantage that the generated data are insensible because of the unstable learning process of the G, we pre-train G with decoder of LSC-VAE. The goal of the learning process of generating data of G is the same as equation 6 from equation 5, and it is equivalent to that of equation 7. However, it is not efficient to pre-train the G, because it depends on the parameters of the D. Therefore, we change this equation to equation 8 again, and it is represented only by the parameters of G. In this paper, to train the G with equation 8, we use the decoder of LSC-VAE, which is trained by using Dec(Enc(x)) ≈ x. The of LSC-VAE is that DISPLAYFORM0 can reach a goal of GAN (p data ≈ p G) stably, which is proved by Theorem 1 and 2. DISPLAYFORM1 where X i is real dataset with i−feature. From the game theory point of view, the GAN converges to the optimal point when G and D reach the Nash equilibrium. In this section, let p G be the probability distribution of data created from G. We show that if G(z) ≈ x, i.e., p data ≈ p G, the GAN reaches the Nash equilibrium. We define DISPLAYFORM0 We train G and D to minimize J(D, G) and K(D, G) for each. Then, we can define the Nash equilibrium of the LSC-GAN as a state that satisfies equation equation 9 and equation equation 10. Fully trained G and D are denoted by G * and D *, respectively. DISPLAYFORM1 Theorem 1. If p data ≈ p G almost everywhere, then the Nash equilibrium of the LSC-GAN is reached. Before proving this theorem, we need to prove the following two lemmas. DISPLAYFORM2 The proof of Lemma 1 and 2 were discussed by Kim et al. BID7. We assume that p data ≈ p G. From Lemma 1 and Lemma 2, if p data ≈ p G, then J(D, G) and K(D, G) both reach minima. Therefore, the proposed GAN reaches the Nash equilibrium and converges to optimal points. By theorem 1, GAN converges when p d ≈ p g, and it is done to some extent by the modified VAE, i.e. | p DISPLAYFORM3 Therefore, the proposed method is useful to initialize the weight of the generative model. However, it shows only validity of using VAE when learning GAN. We prove that it is also efficient by proving theorem 2. Assume that a model f is well-trained if and only if ∇L f ≈ 0, where L f is the loss function of f.Theorem 2. Let en k, de k be k epoch-trained encoder and decoder whose goal is DISPLAYFORM4 Proof. Notice that the derivative is unique, and a derivative of linear function is itself. Since en and de are trained with L V AE and L prior, the following statement is satisfied. DISPLAYFORM5 By differentiating the formula, DISPLAYFORM6 Since the derivative of linear function is itself, it derives to DISPLAYFORM7 With the fact that D(x) = 1, ∀x ∈ X and equation 11, it finally derives to DISPLAYFORM8 By theorem 1 and 2, the proposed learning process is valid and efficient. To verify the performance of the proposed model, we use the celebA dataset BID13. It is a large-scale face attributes dataset. We crop the initial 178×218 size to 138×138, and resize them as 64×64. We use 162,769 images in celebA and 14 attributes: black hair, blond hair, gray hair, male, female, smile, mouth slightly open, young, narrow eyes, bags under eyes, mustache, eyeglasses, pale skin, and chubby. We assign 20 dimensions to each feature and set mean of the i th -20 dimensions as 1. For example, if an image has i-feature, the elements of i * 20 th to (i + 1) * 20 th of the image's latent variable are 1 in average and 0 in the remainder and we denote that latent variable as n i. As shown in FIG0, we generate images from a specific latent space by using LSC-GAN. The images in the first column are generated to have'female' and'blond hair' features. We confirm that the condition works well. The images in the remaining columns are transformed using equation 15 for the features listed below. For example, if we generate an image x i which has i−feature from the latent variable z i, we add n j to add j−feature into the image. DISPLAYFORM0 where x ij is an image which has i− and j−features, and z i is the latent variable for i−feature. To show that the proposed model does not simply memorize data but understand features of data and generate them, we generate images from a series between two random images as in DCGAN. As shown in FIG1, the change between images is natural so that we can say that the latent space of LSC-GAN is a manifold. Besides, the images in the middle column have both features of images in leftmost and rightmost, ing in more simple manifold as shown in Fig. 1.Unlike other GAN models, the LSC-GAN fully understands features of data so as to generate data including inverse-feature. We only train the model about the presence of the'pale skin' and'smile' features, but the model also learned about the reverse of'pale skin' and'smile' automatically as shown in the fourth and the ninth column of FIG2. Besides, if we assign a value of 2 rather than 1 to the average of latent variable which is related to'mustache', we can see that more mustaches are created in the last column in FIG2. Therefore, our model can automatically infer and generate the data with inverse-feature that do not exist in the dataset. This shows that the proposed model has the ability to deduce a negative feature by itself although only positive features are used in trainingTo verify the proposed model, we conduct subjective test about the quality of the generated data. We generate data by using DCGAN, EBGAN, and the proposed GAN. We randomly choose 25 generated data for each model. We perform the subjective test on 30 subjects and ask them to evaluate the quality of the generated data in 5 ways: very low, low, medium, high, and very high. We collect the of 750 questionnaires, which are the evaluated of 25 generated images by 30 subjects, and summarize them in TAB0. We score 1,2,3,4, and 5 points for each evaluation which is shown in the last column in TAB0. Our model not only generates images according to input conditions, but also compress efficiently. We calculate the compression rate with rate= size inputdata /size bottleneck /#classes. As shown in TAB1, our proposed model has the best compression rate compared to others. This proves experimentally that LSC-VAE, theoretically proven with theorems 1 and 2, has been helpful in initializing the weights of the LSC-GAN, and it can achieve good performance even in small latent spaces. In this paper, we address some of significant issues in generative models: unstable training, hidden manifold of data, and extensive hardware resource. To generate a data whose type is what we want, we propose a novel model LSC-GAN which can control a latent space to generate the data that we want. To deal with a larger scale of latent space cause by deploying various distributions in one latent space, we use the LSC-VAE and theoretically prove that it is a proper method. Also, we confirm that the proposed model can generate data which we want by controlling the latent space. Unlike the existing generative model, the proposed model deals with features continuously, not discretely and compresses the data efficiently. Based on the present findings, we hope to extend LSC-GAN to more various datasets such as ImageNet or voice dataset. In future work, we plan to conduct more experiments with various parameters to confirm the stability of model. We will also experiment by reducing the dimension of the latent space to verify that the proposed model is efficient. Besides, since the encoder can project the data to the latent space according to the features inherent in data, it could be used as a classifier.

We propose a generative model that not only produces data with desired features from the pre-defined latent space but also fully understands the features of the data to create characteristics that are not in the dataset.

We use a hypernetwork to predict optimal weights given hyperparameters, and jointly train everything together.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: With the ever increasing demand and the ant reduced quality of services, the focus has shifted towards easing network congestion to enable more efficient flow in systems like traffic, supply chains and electrical grids. A step in this direction is to re-imagine the traditional heuristics based training of systems as this approach is incapable of modelling the involved dynamics. While one can apply Multi-Agent Reinforcement Learning (MARL) to such problems by considering each vertex in the network as an agent, most MARL-based models assume the agents to be independent. In many real-world tasks, agents need to behave as a group, rather than as a collection of individuals. In this paper, we propose a framework that induces cooperation and coordination amongst agents, connected via an underlying network, using emergent communication in a MARL-based setup. We formulate the problem in a general network setting and demonstrate the utility of communication in networks with the help of a case study on traffic systems. Furthermore, we study the emergent communication protocol and show the formation of distinct communities with grounded vocabulary. To the best of our knowledge, this is the only work that studies emergent language in a networked MARL setting. Co-existing intelligent agents affect each other in non-trivial ways. Consider for example, two agents playing a modified variant of archery in two dimensions. Agent A controls the speed at which the arrow is released but it can only shoot along y-axis. Agent B controls wind speed along x-axis. The arrow drifts along the direction of wind with a magnitude proportional to wind speed. A target is specified by (x, y) coordinates and the agents must act cooperatively to shoot the target. In this setup, the optimal action for agent A depends on the current policy of agent B and vice versa. Any change in one agent's policy modifies the other agent's perception about the environment dynamics. Formally, this issue is referred to as non-stationarity of environment in a multi-agent setup. This non-stationarity makes the learning problem hard and approaches that try to independently learn optimal behavior for agents do not perform well in practice . Thus, it is important to develop models that have been tailored towards training multiple agents simultaneously. In this paper, we focus on a specific multi-agent setup where the agents are connected to each other via an underlying fixed network topology. We cast the problem in the multi-agent reinforcement learning (MARL) framework and assume that agents are rewarded by the environment based on their actions and their goal is to cooperatively maximize their rewards. We further assume that the agents have been endowed with the ability to communicate with one another along the network edges to achieve cooperation. However, the communication protocol is not fixed and the agents must learn a protocol to communicate with each other in order to maximize their rewards. Communication is essential in a multi-agent setup. In many practical scenarios, agents may only observe a small portion of the global environment state and they must take actions based on their local observations. As discussed above, agents affect each other in non-trivial ways through their actions. Thus, for achieving long term cooperation, it is essential for agents to be able to share their intents to complement the information provided by the local observation of each agent. Communication provides the ability to do so to the agents. Many real world problems can be cast in this framework and we provide a number of concrete examples after formally defining the problem setup in Section 2. For clarity of exposition and to be more concrete, in this paper, we focus on a particular real world problem as a case study, the problem of intelligently managing traffic. We present the traffic management problem as a particular instantiation of the abstract multi-agent reinforcement learning problem that we have informally defined above (see Section 2 for a formal definition). In this context, the agents correspond to traffic lights and the underlying network is the network of roads. Agents receive rewards from the environment based on factors like queue length at the traffic junction and must communicate with each other to cooperatively maximize their rewards, thereby ensuring a smooth flow of traffic. We propose a MARL-based traffic system that allows coordination between traffic signals (agents) via: (i) inter-agent communication; and (ii) a cooperative reward structure. At each time-step, the agents communicate with their immediate neighbours in the underlying network by broadcasting a message in the form of a discrete symbol (each message corresponds to a word, represented by a binary vector in our experiments). Over time, the agents are trained to exploit this broadcasted message to coordinate with each other and maximize their rewards. As the agents are trained, a language emerges between pairs of agents. Since the agents learn the communication protocol themselves, our approach is different from methods that use a fixed protocol for communication, like smart-grids. We empirically demonstrate the utility of communication in this setup and also investigate a cooperative reward structure over the network of traffic junctions. Our model uses a query-based soft attention mechanism to help the agents come up with more complex cooperative strategies. We perform extensive experimental evaluation to demonstrate that (i) our method outperforms baseline approaches; (ii) communication is useful, (iii) communication is grounded in actions taken by the agents, and (iv) the cooperative reward structure promotes communication and hence coordination. Markov games generalize the notion of a Markov decision process (MDP). A Markov game is used to model an environment where N intelligent agents co-exist. Let S ⊆ R ds be the set of all possible environment states (i.e., state-space) and denote by O i ⊆ R do i the observation space of agent i. At each step, agent i chooses an action from its action space A i and in response to the actions chosen by all agents, the environment state is updated using the transition function T: S × A 1 × · · · × A N → ∆(S) where ∆(S) is the set of all probability distributions defined over set S. The environment provides a reward to each agent at each time-step using a reward function r i: S × A i → R. For i = 1, 2,..., N, the goal of i th agent is to find a policy π θi: O i → ∆(A i) that chooses optimal actions so as to maximize the long term reward, R i = t γ t r t i where r t i is the reward received by agent i at time-step t and γ ∈ is the discount factor. We model the problem as a Markov game with two additional assumptions: (i) let V = {1, 2, . . ., N} be the set of all agents, we assume that the agents are connected to each other via an underlying network whose edge set is given by E; and (ii) agents can communicate with their immediate neighbors in the underlying network. To communicate, at each step, agents broadcast a message. This message is received by the neighbors at the next time-step. The observation space of each agent is augmented to consider the messages received from all of its neighbors in addition to considering the local observation made by that agent. The agents act in a cooperative setting as follows: first, the environment provides a reward r t i to all agents. Then, each agent i, augments this reward to include the information about rewards received by its neighbors, i.e.,r Here, β assigns relative importance to agent's own rewards and the rewards of its neighbors (β ∈) and U(i) is the set of neighbors of agent i. The agents are trained to maximize long term rewardsR i = t γ trt i. As agents also benefit from the rewards obtained by their neighbors, they act cooperatively. Such a setup can for instance be used to model an intelligent electrical distribution network. In this application, agents represent power stations and the underlying network corresponds to the electrical grid. Each agent has a production capacity and it may choose to share the power generated by it with a neighboring agent (action space). All agents have to meet the local demand which changes stochastically and they observe various attributes like power requirements, consumer demand and so on (observation space). Rewards r i are provided based on how successful an agent is in meeting the demand. The agents must learn to communicate with each other to effectively share the power generated by them to maximize their cooperative long term rewardsR i. As another application, consider a supply chain where interconnected warehouses (agents) have to manage their inventory to meet the local demand. As before, agents can choose to ship goods that they have in their inventory, to their neighbors (action space). The warehouses have to meet stochastically changing consumer demands and must learn to communicate effectively in order to share goods so that an appropriate level of inventory is maintained at each warehouse. As in the case of electrical distribution network, rewards received by agents depend on their ability to effectively meet the local demand. While many applications are possible, in this paper, we focus on the problem of traffic management as a concrete instantiation of the proposed abstract problem. Each traffic junction corresponds to an agent which are connected to each other via the underlying network of roads. Agents can control the flow of traffic in the system by modulating the traffic lights (action space). They take actions based on their local observations, which in our case, is a image containing snapshot of the immediate surroundings of the agent. Agents are rewarded based on predefined attributes like queue length at the traffic junction, average waiting time of vehicles at the junction and so on. We again train the agents to maximize the long term cooperative rewards. Clearly, in order to ensure a smooth flow of traffic, the agents must communicate with each other to share their intent. We use the SUMO traffic simulator to simulate the traffic for the experiments. In our setup, we use binary messages with d = 8. As such, agents are limited to a vocabulary of 256 words. Communication is one-hop, i.e. we only allow agents to communicate with its immediate neighbours. At each time-step, the agents broadcast 8-bit messages to its respective neighbours. Agents rely on their local observations and received messages to modulate the traffic signals and to decide the next message to be broadcasted. We opted for discrete communication rather than continuous since it is more interpretable. For the real world applications that we have mentioned above, transparency in decision making is important and using discrete communication is a step in this direction. Towards this end, we demonstrate that the emergent language is grounded, i.e. messages correspond to actions in Section 5. By forcing agents to communicate with the help of discrete symbols, we ensure that humans can inspect and interpret the conversation logs. The simplest and one of the most widely used approaches to tackle the traffic management problem is the Fixed-time Control , which uses a predefined cycle for planning. Network level control like the Max-Pressure Controller differ from the conventional approaches in respect to greedily switching phases based on demand (queue length in adjoining lanes). Similarly, Self-Organizing Traffic Light Control (SOTL) method switches the traffic lights based on when the number of vehicles in the lanes crosses a predefined threshold. These methods are locally adapted to the traffic conditions which in turn is used to achieve global synchronization. Optimization methods like TUC assume uniform traffic flow in a certain time period to minimize the vehicle travel time. We argue that conventional traffic control methods base their problem setup on assumptions which do not hold while modelling the dynamics of traffic. In the recent past, people have resorted to the use of Reinforcement Learning methods to dynamically adapt to the traffic conditions (Prabuchandran K. J. et al., 2014; ;), wherein, each junction is treated as an agent and the changes in traffic lights are actions. The agents try to maximize their long term reward by mapping a predefined observation space (representation of the traffic conditions at its junction) to the action space. However, these Deep-Q Learning (DQN) based approaches often consider the agents to be independent of one another, thus rendering the environment non-stationary, and raising convergence and stability issues. A straightforward way to address the above problems is to make the agents coordinate amongst themselves by including the information from the neighbouring agents in its state space (; ;), assuming total observability. Another way would be to use centralized training to address stationary related issues. However, centralized training poses scalabilty problems (especially in a traffic-management system where the complexity increases linearly with the junctions) and cannot be used to learn policies in real time. Scalable multi-agent approaches like Van der propose to train a smaller source problem involving fewer agents (2 agents) and transfer it to a larger target problem using transfer-learning. Prior work on extending MARL setup to networks using a fully decentralized training has been proposed by , wherein the authors propose to share the local parameters of the agents through communication. Drawing motivation from the recent advancements in emergent communication , we try to induce coordination amongst agents with partial observability. We argue that with a cooperative reward structure, agents can be made to pass on relevant information to their neighbours, which the independent agents were previously incapable of modelling using their own observation space. Concurrent to our work, proposed a similar idea of using communication in traffic networks. Their approach differs from ours in the following ways: (i) They consider cars as agents, whereas we formulate this as a network problems with nodes, which are fixed, as agents; (ii) They use continuous communication, whereas we use discrete communication in our setup for better interpretability; (iii) In their setup, every agent can communicate with the other agents, whereas we restrict the communication to its immediate neighbours with a priority assigned to each message. Image Encoder: As shown in Fig. 1, we use a 10-agent traffic network (Network 1) which comprises both, 3-way as well as 4-way junctions. The observation space of the agents in the traffic network comprises the image representation of the junction. This amorphous representation of the observation space propels the agent to extract necessary information from the image, like length of the queue in the adjoining lanes or the current phase at the junction. One can always add these information to the state directly and, in general, adding more information to the state space often leads to better , nonetheless, such systems carry redundant information and are impractical to deploy. For instance, consider lining the lanes with sensors to calculate the number of vehicles. A Convolution Neural Network (CNN) is used to process the input image and extract specific features required for optimizing traffic. The agents, then, use this information to take actions. Since, it is unreasonable to change the traffic lights every second, we constrain the agents to take an action once in every t timesteps (t = 5s in our setup). The Accumulator is a Recurrent Neural Network (RNN) which keeps track of the observations for t time-steps before an action is taken. Let the encoded image information of agent i at time-step k be o is the output dimension of the encoder CNN). The Accumulator uses this encoded vector to update its memory by h Acc Communicator: In order to establish complex cooperative strategies, the agents must learn to prioritize messages received from its neighbours. In that context, we incorporate a query-based softattention mechanism in our communication setup. At each time-step, an agent generates a query (q) which is used to assign weights to the received messages in the previous time-step. Intuitively, an agent enquires about the unaccounted information (which it cannot model) in O i and the weights (α) are a way to understand the contributions of its neighbours in the state information. Let us assume that the messages broadcasted are denoted by m, where m ∈ M (M is set of all messages). For agent i at k th time-step with neighbours belonging to the set U i, Here W ∈ R d×d h, q ∈ R d and α ∈ R d are the attention parameters. At each time-step, the pooled message is fed to the Communicator RNN and is processed as h com It should be noted that we reinitialize the hidden state of the RNN as soon as the action is taken (i.e. after every t time-steps). A weighted mixture of the output of the RNN along with the output of the image encoder (o k i) is passed through a setup which emits discrete symbols with the help of StraightThrough Gumbel-Softmax (G) . This renders the model differentiable even with discrete symbols. The output message is then broadcasted to all its neighbours (U i). Refer Appendix A.2 for more details. Action Selector: A 4-way junction can have 2 16 combinations of actions (2 9 for a 3-way junction), however, most of them are either highly dangerous for traffic or don't contribute to a smoother flow. Adopting conventional approaches , we fix the action space to 4 actions for a 4-way junction and 3 for a 3-way junction (Fig. 3). The actions are determined such that the vehicles can move smoothly without conflicts. The action values are a function of Accumulator-RNN and Communicator-RNN hidden states i.e. f actioni (h Acc k i, h com k i). At every t th time-step, the action values are generated and the traffic lights are switched to the action with the maximum value. Reward Structure: The reward at a junction i is a combination of the following factors: (i) Queue Length: Total number of waiting vehicles (with speed < 0.1m/s) on the adjoining lanes; (ii) Waiting Time: Sum of waiting time of all vehicles in the adjoining lanes, wherein, a vehicle is considered to be waiting if it travels at a speed < 0.1m/s; (iii) Delay: For lane l, delay is calculated as D Communication What gives the agents the incentive to communicate? An agent might turn out to be selfish and focus on solving its own problems even while it broadcasts unstructured messages to its neighbours. Even if the agents are forced to communicate, what makes us believe that the agents will pay any heed to the messages that they receive, considering the individual reward structure? In view of this, we highlight two important aspects of communication: (i) Cooperation: Meaningful information exchange will arise only if there is a dearth of knowledge in the local observations of the agents. In other words, there are factors that the agent is uncertain of and cannot model in the current setup. To see why, let us consider a scenario where there are two interconnected independent agents (say A and B) trying to model the traffic. The arrival rate of agent A is modelled using Poisson, with rate λ, which is a function of the environment. As agent B takes an action, the environment is no longer stationary and λ varies in a way that agent A is incapable of modelling (let us assume λ now becomes (λ − δλ)). Hence, the agent learns to ignore these variables altogether. In order to make sense of the unaccounted changes in its observation space, agent B has to communicate the difference in λ to agent A which can be assumed to be Poisson distributed with rate δλ (superposition of independent Poisson processes in a new Poisson process whose rate is the sum of the rates of the independent Poisson processes); (ii) Coordination: Even while groups of agents implicitly come up with a common communication protocol, there should be a higher level harmony amongst all agents, i.e. the groups of agents in a network must systematically synchronize or coordinate to realize the same goal. In order to drive the agents to work for the greater good, they must arrive at a common protocol which can be used universally. This leads to the development of a common language with overlapping vocabularies from different communities of agents. Additionally, if an agent in the sub-network fails, the other agents in that group can still communicate with other groups. Comparison with baselines: We compare our model with the following baselines: We use our action space and periodically switch from one action to the other in a round-robin fashion after fixed time intervals of duration t = 5 steps. (ii) Self Organizing Traffic Light control (SOTL): SOTL switches from one phase to other when the queue length at any of the adjoining lanes exceed a predefined threshold. In our implementation, we fix this threshold to 5 while retaining our action space. (iii) Deep-Q Learning (DQN): We also implement the standard DQN framework with the same observation and action space as ours where each agent is trained independently. (iv) IntelliLight: IntelliLight uses a more refined state representation which includes queue length, number of vehicles and updated waiting time at the adjoining lanes of a junction, in addition to its image representation. We replicate their architecture with the same state representation but with our action space, since their action space is fraught with danger in a real life traffic management system. The (Fig. 4) reveal that our method outperforms all baselines in terms of the end (final reward at convergence). While DQN training commences at a high reward (because of high initial value of in the -greedy strategy), it converges quickly without much improvement. We also wish to point out that parameter updates in DQN happen every timestep once the buffer-size exceeds the batch-size, as opposed to once every episode in our setup. On an average, the difference between the final rewards at convergence of the DQN and the our setup is ≈ 55. An important question that naturally follows is whether communication is required at all. It may indeed be possible for the agents to adapt even while they broadcast random set of symbols which do not bear any significance. To test this possibility, we perform two sets of experiments: (i) We made the agents broadcast 8-bit messages with all bits set to zero. We found that the training not only converged slowly, it also stabilized at a lower reward compared to the original setting. Post convergence the difference in rewards was ≈ 40. (ii) We visually impaired one of the agents (we call it a blind-agent) such that it no longer receives its local observation while taking an action, although it can still receive messages from its neighbours and transmit its state information to them i.e. the agent processes its input observation at each time-step but the weight W CNN is set to zero. Hence, it cannot backpropagate valuable gradients from its actions to the image encoder. We noticed that the overall setup still converged to the same as the original setup. We attribute it to the fact that the blind-agent receives necessary information from its neighbours through communication. On repeating the experiment for two neighbouring blindagents, the convergence worsens (Fig. 5). Thus, we can safely conclude that communication not only works but is also necessary. We performed the above experiments on agents 4 & 5 (Network 1 of Fig. 1). In order to verify grounding, we performed experiments using Pointwise Mutual Information (PMI) . We constructed a matrix for each pair of agents where the rows correspond to the actions of one agent (say i) and the columns correspond to the message sent by the other agent in the pair. We computed the Singular Value Decomposition (SVD) of the PMI matrix (pmi ∈ R |Ai|×256) given as pmi = USV T, where U ∈ R |Ai|×k, S ∈ R k×k and V ∈ R k×256 (k ≤ |A i |). Figure 6 shows the t-SNE (van der) plot of the rows of V matrix. The rows of V can be interpreted as word embeddings. As shown in Fig. 6, we see distinct clusters forming for different actions. In other words, we can say that neighbouring agents use specific set of words to indicate actions. The words which are haphazardly placed can be considered as those that haven't been assigned a specific meaning or the ones which can be used in many different contexts. Additionally, we plot the the rows of U matrix (with k = 2) i.e. the action embeddings corresponding to the broadcasted messages of all agents. For instance, as shown in Fig. 8, we pair the actions of agent 0 with the broadcasted messages from the rest of the agents. For each such pair, we get a U matrix which we plot using different colors for different agents. We notice that the points corresponding to the U matrix of the neighbours tend to overlap as highlighted by the red circles (for agent 0, agents 1 and 3 overlaps). This implies that neighbours U 0 are consistent in the use of messages to agent 0, which in turn bases its actions on the received messages. See Appendix A.3 for the remaining plots on grounding. Fig.(a,b) ]. The numbers {0, . . ., 1} denote the agents in the network. Fig.(c) on the right represents the clustering in the 28-agent network (Network 2) with A & B denoting two 14-agent sub-networks. The position of the numbers denote the mean of the clusters. We obtain a tf-idf matrix where rows correspond to agents and columns correspond to the words spoken by these agents. On plotting a t-SNE plot of the rows of this matrix for Network 1 (Fig. 1), we noticed that the agents that are broadcasting to the same neighbour tend to be clustered together. For instance, from Experiments on larger networks: Due to communication, we expect well-defined communities to emerge (in terms of the vocabulary usage) if huge networks are sparsely connected to one another. To test this hypothesis, we take a pair of networks, each having 14 agents and connect them by a single one-way lane (Network 2 in Fig. 1). We plot the t-SNE embeddings of the rows of tf-idf matrix (Fig. 7 (c) ). We notice two distinct clusters (denoted by A and B) with a overlapping region, which we argue, comprises the common vocabulary used by agents from both networks. To sum it up, we not only find evidence of cooperation among agents but also coordination among groups of agents across networks. On removing the cooperative reward structure, the model takes twice as much time to converge, presumably because the agents have little incentive to communicate. In this paper, we proposed an approach to mitigate network congestion with the help of traffic networks. Though extensive experiments, we demonstrated the benefit of emergent communication to optimize traffic flow over existing MARL approaches. Additionally, we performed qualitative studies on the emergent language and showed that it is grounded in actions. Human communication is discrete in nature and can, in general, be represented by categorical variables. Additionally, discrete variables are more interpretable which makes it well suited for real life problems like traffic management, where one needs transparency. However, the use of discrete latent variables render the neural network non-differentiable. The Gumbel Softmax gives a differentiable sample from a discrete distribution by approximating the hard one-hot vector into a soft version. The Gumbel distribution has two parameters: µ and β. The standard Gumbel distribution where µ and β are 0,1 respectively has probability density function: G = e −(z+e −z). Suppose, for a given agent, our model (Communicator) outputs a Multinomial distribution of message bits with logits: p = (p 1, . . ., p d) where d = 8. These logits are functions of inputs and weights which need to be trained. A simple way to draw samples from a discrete distribution would be to use the Gumbel-max trick as shown, Here, the noise in form of z i is independently sampled from standard Gumbel distribution and are obtained as z i = − log(− log(u i)), u i being i. β is the temperature parameter, which, in our experiments, is set to 0.5. As β > 0, we obtain welldefined gradients ∂m i /∂p i with respect to parameters p i. Gumbel-Softmax can produce a 0.9999-hot vector instead of a hard one-hot vector depending on β. Since we want the communication to be discrete, we employ the Straight-Through version of the Gumbel-Softmax estimator with a simple reformulation of the form, Here,m i is the one-hot vector ofm i i.e. such thatm i = I{arg max im i = k, k ≤ k}. We use binary 8-bit messages, therefore k = 1 and k is fixed during the training process. Now, detach prevents the gradients from flowing through that node, hence, ∇ p i m i = ∇ p imi. This makes the communication discrete in the forward pass even as the gradients flow is smooth during the backward pass. The final output message of agent is given as m = (m 1, . . ., m d). A.3 COMMUNICATION. Action embeddings from matrix U (orthonormal matrix) of agent 0 corresponding to messages from all agents. As highlighted by the red circles, the action embeddings of agent i in response to messages from neighbours U i (as highlighted in gray at the centres of the plots) are overlapped. The color bar represents different agents. Here, we plots are for agents 0, 1, 8, 9. Similar trends are observed for rest of the agents as well.

represent the short summary for finding the corresponding scientific article (long)

A framework for studying emergent communication in a networked multi-agent reinforcement learning setup.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.