Question: Represent the article for finding a single-sentence TLDR

Question: Weight pruning has been introduced as an efficient model compression technique. Even though pruning removes significant amount of weights in a network, memory requirement reduction was limited since conventional sparse matrix formats require significant amount of memory to store index-related information. Moreover, computations associated with such sparse matrix formats are slow because sequential sparse matrix decoding process does not utilize highly parallel computing systems efficiently. As an attempt to compress index information while keeping the decoding process parallelizable, Viterbi-based pruning was suggested. Decoding non-zero weights, however, is still sequential in Viterbi-based pruning. In this paper, we propose a new sparse matrix format in order to enable a highly parallel decoding process of the entire sparse matrix. The proposed sparse matrix is constructed by combining pruning and weight quantization. For the latest RNN models on PTB and WikiText-2 corpus, LSTM parameter storage requirement is compressed 19x using the proposed sparse matrix format compared to the baseline model. Compressed weight and indices can be reconstructed into a dense matrix fast using Viterbi encoders. Simulation show that the proposed scheme can feed parameters to processing elements 20 % to 106 % faster than the case where the dense matrix values directly come from DRAM. Deep neural networks (DNNs) require significant amounts of memory and computation as the number of training data and the complexity of task increases BID0. To reduce the memory burden, pruning and quantization have been actively studied. Pruning removes redundant connections of DNNs without accuracy degradation BID6. The pruned are usually stored in a sparse matrix format such as compressed sparse row (CSR) format or compressed sparse column (CSC) format, which consists of non-zero values and indices that represent the location of non-zeros. In the sparse matrix formats, the memory requirement for the indices is not negligible. Viterbi-based pruning BID14 significantly reduces the memory footprint of sparse matrix format by compressing the indices of sparse matrices using the Viterbi algorithm BID3. Although Viterbi-based pruning compresses the index component considerably, weight compression can be further improved in two directions. First, the non-zero values in the sparse matrix can be compressed with quantization. Second, sparse-to-dense matrix conversion in Viterbi-based pruning is relatively slow because assigning non-zero values to the corresponding indices requires sequential processes while indices can be reconstructed in parallel using a Viterbi Decompressor (VD).Various quantization techniques can be applied to compress the non-zero values, but they still cannot reconstruct the dense weight matrix quickly because it takes time to locate non-zero values to the corresponding locations in the dense matrix. These open questions motivate us to find a non-zero value compression method, which also allows parallel sparse-to-dense matrix construction. The contribution of this paper is as follows.(a) To reduce the memory footprint of neural networks further, we propose to combine the Viterbibased pruning BID14 ) with a novel weight-encoding scheme, which also uses the Viterbi-based approach to encode the quantized non-zero values. (b) We suggest two main properties of the weight matrix that increase the probability of finding "good" Viterbi encoded weights. First, the weight matrix with equal composition ratio of'0' and'1' for each bit is desired. Second, using the pruned parameters as "Don't Care" terms increases the probability of finding desired Viterbi weight encoding. (c) We demonstrate that the proposed method can be applied to Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) with various sizes and depths. (d) We show that using the same Viterbi-based approach to compress both indices and non-zero values allows us to build a highly parallel sparse-to-dense reconstruction architecture. Using a custom cycle-simulator, we demonstrate that the reconstruction can be done fast. DNNs have been growing bigger and deeper to solve complex nonlinear tasks. However, BID2 showed that most of the parameters in neural networks are redundant. To reduce the redundancy and minimize memory and computation overhead, several weight reduction methods have been suggested. Recently, magnitude-based pruning methods became popular due to its computational efficiency BID6. Magnitude-based pruning methods remove weights according to weight magnitude only and retrain the pruned network to recover from accuracy loss. The method is scalable to large and deep neural networks because of its low computation overhead. BID6 showed 9×-13× pruning rate on AlexNet and VGG-16 networks without accuracy loss on ImageNet dataset. Although the compression rate was high, reduction of actual memory requirement was not as high as the compression rate because conventional sparse matrix formats, such as CSR and CSC, must use large portion of memory to store the indices of surviving weights. BID14 succeeded in reducing the amount of index-related information using a Viterbi-algorithm based pruning method and corresponding custom sparse matrix format. BID14 demonstrated 38.1% memory reduction compared to BID6 with no accuracy loss. The memory reduction was limited, however, due to uncompressed non-zero values. Several weight quantization methods were also suggested to compress the parameters of neural networks. BID1 BID15; BID20 demonstrated that reducing the weights to binary or ternary was possible, but the accuracy loss of the binary neural networks was significant. BID25 reduced the bit resolution of weights to binary, activations to 2 bits and gradients to 6 bits with 9.8 % top-1 accuracy loss on AlexNet for ImageNet task. BID5 demonstrated a binary-weight AlexNet with 2.0% top-1 accuracy loss, achieving ∼10× compression rate. BID22 showed that RNNs can also be quantized to reduce the memory footprint. By quantizing the weight values to 3 bits with proposed method, the memory footprint of RNN models were reduced ∼10.5× with negligible performance degradation. BID8 suggested to combine pruning with weight quantization to achieve higher compression rate. The showed 35× increase in compression rate on AlexNet. However, the reduction was limited since the memory requirement of index-related information was only slightly improved with Huffman coding. Although several magnitude-based pruning methods showed high compression rate, computation time did not improve much, because it takes time to decode the sparse matrix formats that describe irregular weight indices of pruned networks. BID7 suggested to use dedicated hardware, custom sparse matrix formats, and dedicated pruning methods to accelerate the computation even after pruning. BID10; tried to accelerate the computation by limiting the irregularity of weight indices. By pruning neurons or feature maps, pruned weight matrices could maintain the dense format. These approaches successfully reduced the number of computation of neural networks, but the compression rate was limited due to additional pruning conditions. Although BID14 could use the Viterbi encoder to construct the index matrix fast, the process of pairing the non-zero weight values with the corresponding indices is still sequential, and thus relatively slow.3 WEIGHT PRUNING AND QUANTIZATION USING DOUBLE-VITERBI APPROACH Figure 1 illustrates the flowchart of the proposed compression method. Viterbi-based pruning BID14 ) is applied first, and the pruned matrix is quantized using alternating multi-bit quantization method BID22. Quantized binary code matrices are then encoded using the Viterbi-based approach, which is similar to the one used in pruning. Figure 1: Flowchart of Double Viterbi compression. W is the weight of an original network and W P, W P Q, andŴ P Q represent the compressed weights after each process. M ∈ {0, 1} is an index matrix which indicates whether each weight is pruned or not, and means element-wise multiplication. DISPLAYFORM0 ∈ {−1, +1} are constants and binary weights generated by quantization. DISPLAYFORM1 ∈ {−1, +1} is binary weights encoded by the Viterbi algorithm. As the first step of the proposed weight encoding scheme, we compress the indices of the non-zero values in sparse weight matrix using the Viterbi-based pruning (Figure 1) BID14. In this scheme, Viterbi algorithm is used to select a pruned index matrix which minimizes the accuracy degradation among many candidates which a Viterbi decompressor can generate. While the memory footprint of the index portion is significantly reduced by the Viterbi-based pruning, the remaining non-zero values after pruning still require non-negligible memory when high-precision bits are used. Hence, quantization of the non-zero values is required for further reduction of the memory requirement. Appendix A.1 explains the Viterbi-based pruning in detail. After Viterbi-based pruning is finished, the alternating multi-bit quantization BID22 ) is applied to the sparse matrix (Figure 1). As suggested in BID22, real-valued non-zero weights are quantized into multiple binary codes DISPLAYFORM0 In addition to the high compression capabilities, another important reason we chose the alternating quantization is that the output distribution of the method is well suited to the Viterbi algorithm, which is used to encode the quantized non-zero values. Detailed explanation is given in Section 3.3. A sparse matrix that is generated by Viterbi-based pruning and quantization can be represented using the Viterbi Compression Matrix (VCM) format BID14. A sparse matrix stored in VCM format requires much smaller amount of memory than the original dense weight matrix does. However, it is difficult to parallelize the process of reconstructing sparse matrix from the representation in VCM format, because assigning each non-zero value to its corresponding index requires a sequential process of counting ones in indices generated by the Viterbi encoder. To address this issue, we encode binary weight codes DISPLAYFORM0 in addition to the indices, based on the same Viterbi algorithm BID3. By using similar VD structures (While using VD structures to generate binary weight codes allows parallel sparse-to-dense matrix conversion, it requires the quantization method to satisfy a specific condition to minimize accuracy DISPLAYFORM1 Proposed process of sparse-to-dense matrix conversion for the Viterbi-based compressed matrix. This figure shows an example such that weight values and weight index values are generated by independent Viterbi decompressors simultaneously.loss after Viterbi-based encoding. It is known that the VD structure acts as a random number generator BID13, which produces '0' and '1' with 50 % probability each. Thus, generated binary weight codes will be closer to the target binary weight codes if the target binary weight code matrix also consists of equal number of '0' and '1'. Interestingly, the composition ratio of '-1' and '+1' in each b i, which was generated by the alternating quantization method, is 50 % each. It is because the weights in DNNs are generally initialized symmetrically with respect to '0' BID4 BID11 and the distribution is maintained even after training BID16 . The preferable output distribution of the alternating quantization implies that the probability of finding an output matrixb i close to b i with the Viterbi algorithm is high. For comparison, we measured the accuracy differences before and after Viterbi encoding for several quantization methods such as linear quantization BID16, logarithmic quantization BID19, and alternating quantization BID22 . When the Viterbi encoding is applied to the weight quantized by alternating quantization BID22, the validation accuracy degrades by only 2 %. However, accuracy degrades by 71 % when the Viterbi encoding is applied to the weight quantized using other methods BID16 BID19 . The accuracy difference mainly comes from the uneven weight distribution. Because weights of neural networks usually have normal distribution, the composition ratio of '0' and '1' is not equal when the linear or logarithmic quantization is applied to the weights unlike alternating quantization. Another important idea to increase the probability of finding "good" Viterbi encoded weight is to consider the pruned parameters in b i as "Don't Care" terms ( FIG2 . The "Don't Care" elements can have any values when findingb i, because they will be masked by the zero values in the index matrix generated by the Viterbi pruning. Next, let us describe how we use the Viterbi algorithm for weight encoding. We select theb i that best matches with b i among all possibleb i cases that the VD can generate, as follows. We first construct a trellis diagram as shown in FIG3 . The trellis diagram is a state diagram represented A cost function for each transition using path and branch metrics is set and computed in the next step. The branch metric λ i,j t is the cost of traveling along a transition from a state i to the successor state j at the time index t. The path metric is expressed as DISPLAYFORM2 DISPLAYFORM3 where i1 and i2 are two predecessor states of j. Equation 1 denotes that one of the two possible transitions is selected to maximize the accumulated value of branch metrics. 2 The branch metric is defined as DISPLAYFORM4 To maintain the accuracy, the number of incorrect bits in the encoded binary codeb i compared to the original binary code b i needs to be minimized. Thus, we retrain the network withŴ P Q = k i=1 α ibi M (M is the index matrix of non-zeros in W), apply the alternating quantization, and then perform the Viterbi encoding repeatedly (Figure 1). By repeating the retraining, quantization, and Viterbi encoding, the number of incorrect bits betweenb i and b i can be reduced because the parameters in the network are fine-tuned close to parameters inŴ P Q. During the retraining period, we apply the straight-through estimate BID20, i.e. ∂C ∂Ŵ P Q = ∂C ∂W as adopted in BID22. After the last Viterbi encoding is finished, small amount of components inb i can be still different from the corresponding values in b i. To maintain the accuracy, location data for the incorrect components are stored separately and are used to flip the corresponding VD encoded bits during on-chip weight reconstruction period. In our experiments, the memory requirement for the correction data was negligible. After the retraining is finished, we can obtain a compressed parameter in Viterbi Weight Matrix (VWM) format, which includes DISPLAYFORM0, compressed index in VCM format, and indices where DISPLAYFORM1. Note that entire training process used the training dataset and the validation dataset only to decide the best compressed weight data. The accuracy measurement for the test dataset was done only after training is finished so that any hyperparameter was not tuned on the test dataset. All the experiments in this paper followed the above training principle. We first conduct experiments on Penn Tree Bank (PTB) corpus BID17. We use the standard split version of PTB corpus with 10K vocabulary BID18, and evaluate the performance using perplexity per word (PPW). We pretrain the RNN model 3 which contains 1 layer of LSTM with 600 memory units, then prune the parameters of LSTMs with 80 % pruning rate using the Viterbi-based pruning technique 4 and retrain the model. Then, we quantize the parameters of LSTMs using alternating quantization technique, encode the binary weight codes by using the Viterbi algorithm, and retrain the model. We repeat the quantization, binary code encoding, and retraining process 5 times. We quantize the LSTM model with different numbers of quantization bits k with the fixed N o = 5. As k increases, PPW is improved, but the memory requirement for parameters is also increased (TAB0). Note that k = 3 is the minimum number of bits that minimizes the model size without PPW degradation. Compared to BID14, further quantization and Viterbi-based compression reduce the parameter size by 78 % to 90 % TAB0. We compress the binary weight codes with different number of VD outputs N o in case of k = 3. As N o increases, PPW degrades while the memory requirement for parameters is increased TAB0. Large N o implies that the binary weight codes are compressed with high compression ratio 1/N o, but the similarity betweenb i and b i decreases. The optimal N o is 100/(100-pruning rate (%)), where the average number of survived parameters per N o serial parameters is 1 statistically, which in no model performance degradation. Effectiveness of "Don't Care": To verify the effectiveness of using the "Don't Care" elements, we apply our proposed method on the original network and pruned one. While the pruned network maintains the original PPW after applying our proposed compression method, applying our method to the dense network degrades PPW to 102.6. This is because the ratio of incorrect bits betweenb i and b i decreases from 28.3 % to 1.7 % when we use the sparse b i. Therefore, combination of the Viterbi pruning and alternating quantization increases the probability of findingb i close to b i using the VD for weight encoding, which in no PPW degradation. The latest RNN for language modeling: We further test our proposed method on the latest RNN model BID23, which shows the best perplexity on both PTB and WikiText-2 (WT2) corpus. We prune 75 % of the parameters in three LSTMs with the same condition as we prune the above 1-layer LSTM model, and quantize them to 3 bits (k = 3). Note that we do not apply fine-tuning and dynamic evaluation BID12 in this experiment. The compression in TAB1 shows that the memory requirements for the models are reduced by 94.7 % with our VWM format on both PTB and WT2 corpus without PPW degradation. This implies that our proposed compression method can be applied regardless of the depth and size of the network. Detailed experiment settings and compression are described in Appendix A.3. In addition, we extend our proposed method to the RNN models for machine translation, and its experimental are presented in Appendix A.4. We also apply our proposed method to a CNN, VGG-9 (2×128C3 -2×256C3 -2×512C3 -2×1024FC -10SM 5) on CIFAR-10 dataset to verify the proposed technique is valid for other types of DNNs. We randomly select 5 K validation images among 50 K training images in order to observe validation error during retraining process and measure the test error after retraining. We use k = 3 for all layers. Optimal N o for each layer is chosen based on the pruning rate of the parameters; N o = 4 for convolutional layers, N o = 25 for the first two fully-connected layers, and N o = 5 for the last fully-connected layer. We also compute the memory requirement for other compression methods. Experimental on VGG-9 is found in Table 3. Compared to BID8, the VWM format generated by the proposed scheme has 39 % smaller memory footprint due to the compressed indices, smaller number of bits for quantization, and encoded binary weight codes. This experiment on CIFAR-10 shows that our proposed method can be applied to DNNs with various types and sizes. Meanwhile, it can be seen that the combination of the Viterbi-pruning BID14 ) with the alternating quantization BID22 ) requires 10% smaller memory requirement than the VWM format because the VWM format requires additional memory for indices where DISPLAYFORM0 However, additional "Viterbi-based binary code encoding" process for the VWM format allows parallel sparse-to-dense matrix conversion, which increases the parameter feeding rate up to 40.5 % compared to BID14. In Section 4.3, we analyze the speed of sparse-to-dense matrix conversion in detail. a) Non-zero values are represented as 32-bit floating point numbers. b) Convolution filters are quantized to 8-bit, and weights of fully-connected layers and indices of sparse matrices are quantized to 5-bit, which is the same quantization condition as the condition used in BID8. c) For the Conv1 layer, pruning is not applied and only the alternating quantization is applied. We built a cycle-level simulator for the weight matrix reconstruction process of the proposed format to show that the sparse matrix-matrix multiplications with the proposed method can be done fast with parallel reconstruction of dense matrix. In the simulator, baseline structure feeds two dense input matrices to processing elements (PEs) using raw data fed by DRAM FIG5 ), while the proposed structure reconstructs both index masks and binary codes using the highly compressed data fed by DRAM and sends the reconstructed values to PEs FIG5 ). Both index masks and binary codes are reconstructed by several Viterbi encoders in parallel, and bit errors in binary codes are corrected in a serial manner using the small number of flip-bit related data, which are received from DRAM. Simulation show that the feeding rate of the proposed scheme is 20.0-106.4 % higher than the baseline case and 10.3-40.5 % higher than BID14, depending on the pruning rate (FIG5). The gain mainly comes from the high compression rate and parallel reconstruction process of the proposed method. As shown in FIG5, higher sparsity leads to higher feeding rate. Higher sparsity allows using many VD outputs for the index N ind, and increasing N ind leads to faster reconstruction. Also, the reconstruction rate of binary codes becomes higher with reduced number of non-zero values and corresponding bit corrections. (c) Rate of parameter feeding into PEs for the proposed scheme compared to those for the baseline structure, which receives the dense matrix data directly from DRAM, and BID14. We assumed the number of VD outputs for the index N ind = 3, 4, 5, 6, 10, 10 respectively as the reciprocal of each sparsity value. We used N ind = 10 for 95 % sparsity since we compressed matrices with over 90 % sparsity with N ind = 10. We also assumed k = 3, and 1 % bit-wise difference betweenb i and b i during simulation. We also assumed that 16 non-zero parameters can be fed into the PE array in parallel and DRAM requires 10 cycles to handle a 256 bit READ operation. We proposed a DNN model compression technique with high compression rate and fast dense matrix reconstruction process. We adopted the Viterbi-based pruning and alternating multi-bit quantization technique to reduce the memory requirement for both non-zeros and indices of sparse matrices. Then, we encoded the quantized binary weight codes using Viterbi algorithm once more. As the non-zero values and the corresponding indices are generated in parallel by multiple Viterbi encoders, the sparse-to-dense matrix conversion can be done very fast. We also demonstrated that the proposed scheme significantly reduces the memory requirements of the parameters for both RNN and CNN. A APPENDIX A.1 PRUNING USING THE VITERBI ALGORITHM In Viterbi-based pruning scheme, the binary outputs generated by a Viterbi Decompressor (VD) are used as the index matrix that indicates whether a weight element is pruned ('0') or not ('1'). Suppose the number of elements in a target weight matrix is q, and the number of outputs generated by a VD at each time step is N ind, then only 2 q/N ind binary matrices can be generated by the VD among all 2 q binary matrices. The index matrix which minimizes the accuracy loss should be selected among binary matrix candidates which VD can generate in this pruning scheme, and the Viterbi algorithm is used for this purpose. The overall pruning process is similar to the binary weight encoding process using the Viterbi algorithm in Section 3.3. First, Trellis diagram (FIG3) of the VD which is used for pruning is constructed, and then the cost function is computed by using the path metric and the branch metric. The same path metric shown in Equation 1 in Section 3.3 is used to select the branch which maximizes the path metric between two connected branches from the previous states. On the other hand, a different branch metric λ i,j t is used for pruning, which is expressed as: DISPLAYFORM0 where W i,j,m t is the magnitude of a parameter at the m th VD output and time index t, normalized by the maximum absolute value of all elements in target weight matrix, and TH p is the pruning threshold value determined heuristically. As β i,j,m t gives additional points (penalties) to the parameters with large magnitude to survive (be pruned), the possibility to prune small-magnitude parameters is maximized. S 1 and S 2 are the scaling factors which is empirically determined. BID14 uses 5.0 and 10 4 each). After computing the cost function through the whole time steps, the state with the maximum path metric is chosen, and we trace the previous state by selecting the surviving branch and corresponding indices backward until the first state is reached. The ideal pruning rate of the Viterbi-based pruning is 50 %, because the VD structures act like random number generator and the probability to generate'0' or'1' is 50 % each. For various pruning rates, comparators and comparator threshold value, TH c, are used. A N C -bit comparator receives N c VD outputs and generates 1-bit whether the value made by the combination of the received VD outputs (e.g. {out 1, out 2, · · ·, out N ind} where out i indicates the i th VD output) is greater than TH c or not. For example, suppose a 4-bit comparator is used to the VD in Figure 1 and TH c = 3, then the probability for the comparator to generate'1' is 25%(= (3 + 1)/2 4 ) and this percentage is the target pruning rate. Comparators and TH c control the value of pruning rates and the index compression ratio decreases by 1/N c times. It is reported that a low N ind is desired to prune weights of convolutional layers while high N ind can be used to prune the weights of fully-connected layers because of the trade-off between the index compression ratio and the accuracy BID14. Thus, in our paper, we use N ind = 50 and N c = 5 to prune weights of LSTMs and fully-connected layers in VGG-6 on CIFAR-10. On the other hand, we use N ind = 10 and N c = 5 to prune weights of convolutional layers in VGG-6 on CIFAR-10. The RNN model in BID23 is composed of three LSTM layers, and use various learning techniques such as mixture-of-softmaxes (MoS) to achieve better perplexity. As shown in TAB0, the parameters in the first layer have high sparsity, so we use N o = 6. In the remaining layers, however, we use N o = 3 because the parameters are pruned with only about 70 % pruning rate. We repeat the process of quantization, binary code encoding, and retraining only once. We also extend our experiments on the RNN models for machine translation 6. We use the model which consists of an encoder, a decoder and an attention layer. 4-layer LSTMs with 1024 units compose each encoder and decoder. A bidirectional LSTM (BiLSTM) is used for the first layer of the encoder. The weights of LSTM models are pruned with 75 % pruning rate by the Viterbi-based pruning techinque, then k = 4 is used for quantization. Optimal N o values are used according to the sparsity of each LSTM layer (i.e. 3 ≤ N o ≤ 6 is enough to encode binary weight codes with 70 -83% of sparsity). The process of quantization, binary code encoding, and retraining is repeated only once in this case, too. As shown in TAB5, we reduce the memory requirement of each baseline model by 93.5 % using our proposed technique. This experiment show that our proposed scheme can be extended to RNNs for other complex tasks.

Represent the TLDR for finding the corresponding scientific article

We present a new weight encoding scheme which enables high compression ratio and fast sparse-to-dense matrix conversion.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective, and we show that, given a desired FLOPs requirement, different neural networks are successfully trained.


Question: represent the scientific article to find a short tldr

Question: The use of deep learning models as priors for compressive sensing tasks presents new potential for inexpensive seismic data acquisition. An appropriately designed Wasserstein generative adversarial network is designed based on a generative adversarial network architecture trained on several historical surveys, capable of learning the statistical properties of the seismic wavelets. The usage of validating and performance testing of compressive sensing are three steps. First, the existence of a sparse representation with different compression rates for seismic surveys is studied. Then, non-uniform samplings are studied, using the proposed methodology. Finally, recommendations for non-uniform seismic survey grid, based on the evaluation of reconstructed seismic images and metrics, is proposed. The primary goal of the proposed deep learning model is to provide the foundations of an optimal design for seismic acquisition, with less loss in imaging quality. Along these lines, a compressive sensing design of a non-uniform grid over an asset in Gulf of Mexico, versus a traditional seismic survey grid which collects data uniformly at every few feet, is suggested, leveraging the proposed method. Conventional computational recovery is suffered from undesired artifacts such as over-smoothing, image size limitations and high computational cost. The use of deep generative network (GAN) models offers a very promising alternative approach for inexpensive seismic data acquisition, which improved quality and revealing finer details when compared to conventional approaches or pixel-wise deep learning models. As one of the pioneers to apply a pixel inpainting GAN on large, real seismic compressed image recovery, we contributes the following points: 1) Introduction of a GAN based inpainting model for compressed image recovery, under uniform or nonuniform sampling, capable to recover the heavily sampled data efficiently and reliably. 2) Superior model for compressive sensing on uniform sampling, that performs better than the originial network and the state-of-the-art interpolation method for uniform sampling. 3) Introduction of an effective, non-uniform, sampling survey recommendation, leveraging the GIN uniform sampling reconstructions and a hierarchical selection scheme. Compressed image recovery can be stated as a missing pixel inpainting problem: given an incomplete image, filling the missing trace values. Using historical images of the uncompressed dataset, we train a data-driven deep learning model, utilizing the raw image as ground truth and the binary mask to indicate the locations of the missing pixels. Once trained, one can test the model's performance on any incomplete image from a different dataset. We used Compression Rate (CR) to define the proportion between uncompressed data size and compressed data size. The main challenges in using an inpainting model to solve the seismic image sampling problems are: 1) seismic images have significantly different statistical characteristics, such as texture-based patterns and wide range of frequencies, compared to natural images; 2) the largest number of unknown pixels is only ¼ of the full image in the original network, whereas in our task covers at least ½ of the image (i.e. CR=2); 3) the known regions in the compressed image are sparsely distributed, contrary to the compact ones in the general inpainting problems. To address these problems, we will modify the original network and employ it on different experiments. The Generative Inpainting Network with Contextual Attention (GIN) is a feed-forward GAN, combining and outperforming several state-of-the-art approaches including context encoders, dilated convolutions of inpainting Error! Reference source not found., Wasserstein GAN and its improvement WGAN-GP. The architecture of GIN is composed by a coarse network recovering the general features for the missing traces, and a refinement network which further reconstructs the detail. Especially, the Contextual Attention in the latter, does not only learn from the known pixels surrounding the masked image area, but it also looks for useful patches from other known image locations. In our Generative Inpainting Network for Compressive Sensing (named as GIN-CS), we replace the single bounding boxes, used in the incomplete image generation, by predefined binary masks. On the one hand, a binary mask could be regarded as the combination of non-adjacent bounding boxes, so that the multiple edges lose the continuity of original spatial relations. On the other hand, the maximum width of connected missing traces is generally small, so the edge effect would be ignored over the global image size. We used a small portion of an internal offshore dataset to train the network, where 5000 of the processed offshore seismic images are cropped into 256×256 and mix the in-line and cross-line cases. There are two ways to arrange the training masks: random single bounding boxes, as in the original GIN or predefined binary sampling masks, as in our modified GIN-CS. For testing, both methods use binary masks. The testing seismic dataset was collected from the Gulf of Mexico (GoM), on the courtesy of TGS. By comparing the performance of our modified GIN-CS with the original GIN and the conventional biharmonic method in the same CR, our model demonstrates the overall superior performance in terms of Mea Square Error (MSE) and Structural Similarity (SSIM) index in all CR cases (Table 1). Also, the GIN related methods are much faster than the traditional method and hardly influenced by the value of CR. Focusing on one trace from the testing image, our model's prediction aligns better with the ground truth relatively to the GIN (Figure 1.1). Although for CR=8,16, our method does not get better performance in terms of PSNR (Table 1. 3), it still generates closer-to-real seismic images without adding additional artificial noise(GIN) or creating blurry fillings (biharmonic), as seen in Figure 1. In order to construct a non-uniform optimal sampling survey set-up, we propose a sampling recommendation approach that leverages the fast implementation of image reconstruction with GIN. This is an efficient non-uniform sampling recommendation method based on hierarchical uniform sampling, which requires only a small number of sampling test cases. Noted that our recommended sampling method does not consider the connected sampling crossing section width, and the effectiveness highly relies on the performance of GIN. 1) Mask Generation. For a given uncompressed seismic image, we designed a set of binary masks that complementary to the whole masks with equal bin width b∈{1,2,4,8} of each groups 2) Difference map generation. The image is tested with all the designed compression cases. Then, we create the corresponding error matrix by calculating the pixel-wise square error of the reconstructions compared with ground truth. Then, summing them up to form a complete image difference map and calculate its trace-wise mean vector. We further split the vector into individual difference values for each trace. Smaller value indicates better reconstruction at trace. 3) Initial candidate traces generation. In order to compare for each trace without breaking the unknown connected traces, we introduce the observed interval ∈ {2,4,8} to distinguish from bin width. The step mean difference is then defined by simply replacing the actual difference value as its mean value over every interval For an observed interval, the initial recommend candidate traces ′ are the first trace indexes that reach the smallest step mean difference, when =, i.e. All candidate traces over the interval form = {′ } as a subset of all traces. One trace might be a candidate for selection from several intervals, or might never become a candidate for selection. 4) Top-to-bottom hierarchical sorting. To avoid repetitive trace selection, a two-order sorting on all the candidate traces is implemented with the descending order of interval followed by ascending order of step mean. The traces in the higher orders are selected firstly and all the adjacent traces within all the intervals has been removed corresponding until the total missing trace reaching the limitation of CR or empty traces remains. The elements in the final set of are the prospective missing traces, which are easy to be recovered by GIN. We have compared the reconstruction performance of our recommended sampling survey with an average of 100 random samplings, and report the improvement in Table 2. Moreover, we compared the uniform sampling and sampling recommendation separately on the same GoM dataset as the 2D depth view in Figure 1Error! Reference source not found. shown. The of cross-line and in-line is combined by taking the average of the overlapped regions, in order to mimic the actual sampling in both dimensions at the same time. All the cross-line and in-line sampled images are stacked together to form a 3D reconstruction of the whole block, showing its 2D depth view in the figure. The recommend sampling points are densely distributed in regions with lithologic features and sparsely distributed in channelized regions. This successfully captures the heterogenetic of the seismic image. We designed and implemented a modification of the GIN model, the GIN-CS, and successfully tested its performance on uniform samplings with compression rates ×2, ×4, ×8, ×16. GIN-CS demonstrates superior reconstruction performance relatively to both the original GIN and the conventional biharmonic method. More precisely, we show that seismic imaging can be successfully recovered by filling the missing traces, revealing finer details, even in high compression rate cases. In addition, the proposed method runs approximately 300 times faster than the conventional method. Finally, a strategy for constructing a recommendation of non-uniform survey is proposed for a field dataset from Gulf of Mexico, based on our from a combination of limited amount of uniform sampling experiments.

represent the tldr for finding the corresponding scientific article

Improved a GAN based pixel inpainting network for compressed seismic image recovery andproposed a non-uniform sampling survey recommendatio, which can be easily applied to medical and other domains for compressive sensing technique.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We suggest the sufficient number of bits for representing weights of DNNs and the optimum bits are conservative when solving real problems.


Question: Represent the scientific article to find a short TLDR!

Question: In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e. the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with an auxiliary task for state prediction and best practices for multi-agent learning, SAD establishes a new state of the art for 2-5 players on the self-play part of the Hanabi challenge. Humans are highly social creatures and spend vast amounts of time coordinating, collaborating and communicating with others. In contrast to these, at least partially, most progress on AI in games has been in zero-sum games where agents compete against each other, typically rendering communication futile. This includes examples such as Go 2018), poker (; Moravčík et al., 2017;) and chess . This narrow focus is unfortunate, since communication and coordination require unique abilities. In order to enable smooth and efficient social interactions of groups of people, it is commonly required to reason over the intents, points of views and beliefs of other agents from observing their actions. For example, a driver can reasonably infer that if a truck in front of them is slowing down when approaching an intersection, then there is likely an obstacle ahead. Furthermore, humans are both able to interpret the actions of others and can act in a way that is informative when their actions are being observed by others, capabilities that are commonly called theory of Mind (ToM),. Importantly, in order to carry out this kind of reasoning, an agent needs to consider why a given action is taken and what this decision indicates about the state of the world. Simply observing what other agents are doing is not sufficient. While these abilities are particularly relevant in partially observable fully cooperative multi-agent settings, ToM reasoning clearly matters in a variety of real world scenarios. For example, autonomous cars will likely need to understand the point of view, intents and beliefs of other traffic participants in order to deal with highly interactive settings such as 4-way crossing or dense traffic in cities. Hanabi is a fully cooperative partially-observable card game that has recently been proposed as a new benchmark challenge problem for AI research to fill the gap around ToM. In Hanabi, players need to find conventions that allow them to effectively exchange information from their local observations through their actions, taking advantage of the fact that actions are observed by all team mates. Most prior state-of-the-art agents for Hanabi were developed using handcrafted algorithms, which beat off-the-shelf deep multi-agent RL methods by a large margin. This makes intuitive sense: Beyond the "standard" multi-agent challenges of credit assignment, non-stationarity and joint exploration, learning an informative policy presents an additional fundamentally new conflict. On the one hand, an RL agent needs to explore in order to discover good policies through trial and error. On the other hand, when carried out naively, this exploration will add noise to the policy of the agent during the training process, making their actions strictly less informative to their team mates. One possible solution to this is to explore in the space of deterministic partial policies, rather than actions, and sample these policies from a distribution that conditions on a common knowledge Bayesian belief. This is successfully carried out in the Bayesian Action Decoder (BAD), the only previous Deep RL method to achieve a state of the art in Hanabi. While this is a notable accomplishment, it comes at the cost of simplicity and generality. For a start, BAD requires an explicit common knowledge Bayesian belief to be tracked, which not only adds computational burden due to the required sampling steps, but also uses expert knowledge regarding the game dynamics. Furthermore, BAD, as presented, is trained using actor-critic methods which are sample inefficient and suffer from local optima. In order to get around this, BAD uses population based training, further increasing the number of samples required. Lastly, BAD's explicit reliance on common knowledge limits the generality of the method. In this paper we propose the Simplified Action Decoder (SAD), a method that achieves a similar goal to BAD, but addresses all of the issues mentioned above. At the core of SAD is a different approach towards resolving the conflict between exploration and being interpretable, which, like BAD, relies on the centralized training with decentralized control (CT/DC) regime. Under CT/DC information can be exchanged freely amongst all agents during centralized training, as long as the final policies are compatible with decentralized execution. The key insight is that, during training we do not have to chose between being informative, by taking greedy actions, and exploring, by taking random actions. To be informative, the greedy actions do not need to be executed by the environment, but only need to be observed by the team mates. Thus in SAD each agent takes two different actions at each time step: One greedy action, which is not presented to the environment but observed by the team mates at the next time step as an additional input, and the "standard" (exploratory) action that gets executed by the environment and is observed by the team mates as part of the environment dynamics. Importantly, during greedy execution the observed environment action can be used instead of centralized information for the additional input, since now the agent has stopped exploring. Furthermore, to ensure that these greedy actions and observations get decoded into a meaningful representation, we can optionally train an auxiliary task that predicts key hidden game properties from the action-observation trajectories. While we note that this idea is in principle compatible with any kind of model-free deep RL method with minimal modifications to the core algorithm, we use a distributed version of recurrent DQN in order to improve sample efficiency, account for partial observability and reduce the risk of local optima. We also train a joint-action Q-function that consists of the sum of per-agent Q-values to allow for off-policy learning in this multi-agent setting using Value Decomposition Networks (VDN) . Using SAD we establish a new SOTA for 2-5 players in Hanabi, with a method that not only requires less expert knowledge and compute, but is also more general than previous approaches. In order to ensure that our can be easily verified and extended, we also evaluate our method on a proof-of-principle matrix game and plan to open-source our training code and agents. Beyond enabling more research into the self-play aspect of Hanabi, we believe these resources will provide a much needed starting point for the ad-hoc teamwork part of the Hanabi challenge. Our work relates closely to research on emergent communication protocols using deep multi-agent RL, as first undertaken by and. There has been a large number of follow-up papers in this area, so listing all relevant work is beyond the scope and we refer the reader to , a recent survey on deep multi-agent RL. One major difference to our work is that the environments considered typically contain a cheap-talk channel, which can be modeled as a continuous variable during the course of training. This allows agents to, for example, use differentiation across the communication channel in order to learn protocols. In contrast, in our setting agents have to communicate through the observable environment actions themselves, requiring fundamentally different methods. Furthermore, our work is an example of cooperative multi-agent learning in partially observable settings under centralized training and decentralized control. There have been a large number of papers in this space, with seminal work including MADDPG and COMA (a), both of which are actor-critic methods that employ a centralized critic with decentralized actors. Again, we refer the reader to for a more comprehensive survey. Until 2018, work on Hanabi had been focused on hand-coded methods and heuristics. Some relevant examples include SmartBot and the so-called "hat-coding" strategies, as implemented by WTFWThat . These strategies use the information theoretic ideas that allow each hint to reveal information to all other agents at the same time. While they do not perform well for 2-player Hanabi due to the smaller action space, they get near perfect scores for 3-5 players. In contrast, so far learning methods have seen limited success on Hanabi. undertake a systematic evaluation of current Deep RL methods for 2-5 players in two different regimes and open-source the Hanabi-Learning-Environment (HLE) to foster research on the game. They evaluate a feed-forward version of DQN trained on 100 million samples and a recurrent actor-critic agent with population based training using 20 billion samples. Notably both agents achieve near 0% win rate for 3-5 players in Hanabi. At a high level their DQN agent is a good starting point for our work. However, since the authors did not propose any specific method of accounting for the issues introduced by -greedy exploration in a ToM task, they resorted to setting to zero after a short burn-in phase. The only state-of-the-art in Hanabi established by an RL agent is from which we refer to in more detail in Section 1 and Section 4. Recently there have also been attempts to train agents that are robust to different team-mates and even to extend to human-AI collaboration . Poker is another partially observable multi-agent setting, although it is fundamentally different due to the game being zero-sum. Recent success in Poker has extensively benefited from search (Brown et al.). Examples of using search in Hanabi include. For a more comprehensive review on previous on Hanabi we refer the reader to. In this paper we assume a Dec-POMDP , in which N agents interact in a partially observable environment. At each time step agent a ∈ 1..N obtains an observation, o a t = O(s t, a), where s t ∈ S is the Markov state of the system and O(s t, a) is the deterministic observation function. Since we are interested in ToM, in our setting the observation function includes the last action of the acting agent, which is observed by all other agents at the next time step. We note that actions are commonly observable not only in board games but also in some real world multi-agent settings, such as autonomous driving. For simplicity, we restrict ourselves to turn based settings, in which at each time step only the acting agents takes an action, u a t, which is sampled from their policy, u a ∼ π a θ (u a |τ a), while all other agents take a no-op action. Here τ a is the action-observation history of agent a, τ a = {o T is the length of the episode and θ are the weights of a function approximator that represents the policy, in our case recurrent neural networks, such as LSTMs . We further use τ t to describe the state-action sequence, τ = {s 0, u 0, r 1, ..r T, s T}, where u t is the joint action of all agents. As is typical in cooperative multi-agent RL, the goal of the agents is to maximize the total expected return, J θ = E τ ∼P (τ |θ) R 0 (τ), where R 0 (τ) is the return of the trajectory (in general R t (τ) = t ≥t γ t −t r t ) and γ is an optional discount factor. We have also assumed that agents are sharing parameters, θ, as is common in cooperative MARL. In Q-learning the agent approximates the expected return for a given state action-pair, s, u, assuming that the agent acts greedily with respect to the Q-function for all future time steps, Q(s, u) = E τ ∼P (τ |s,u) R t (τ), where τ = (s t, u t, r t+1, . . ., s T), u t = u and u t = arg max u Q(s t, u), ∀t > t. A common exploration scheme is -greedy, in which the agent takes a random action with probability and acts greedily otherwise. Importantly, the Q-function can be trained efficiently using the Bellman equation:, where for simplicity we have assumed a deterministic reward. In Deep Q-Learning (DQN) the Q-function is parameterized by a deep neural network and trained with transitions sampled from experience replay. In our work we also incorporate other best practice components of the last few years, including double-DQN (van), dueling network architecture and prioritized replay . We also employ a distributed training architecture similar to the one proposed by where a number of different actors with their own exploration rates collect experiences in parallel and feed them into a central replay buffer. Since our setting is partially observable the natural choice for the function approximator is a recurrent neural network. A combination of these techniques was first explored by in single agent environments such as Atari and DMLab-30. Another common best-practice in RL are auxiliary tasks; , in which the agent produces extra output-heads that are trained on supervised task and optimized alongside the RL loss. The most straight forward application of Q-learning to multi-agent settings is Independent Q-Learning (IQL) in which each agent keeps an independent estimate of the expected return, treating all other agents as part of the environment. One challenge with IQL is that the exploratory behavior of other agents is not corrected for via the max operator in the bootstrap. Notably, IQL does typically not take any advantage of centralized training with decentralised control (CT/DC), a paradigm under which information can be exchanged freely amongst agents during the training phase as long as the policies rely only on local observations during execution. There are various approaches for learning joint-Q-functions in the CT/DC regime. For example, Value-Decomposition-Networks (VDN) represent the joint-Q-function as a sum of per-agent contributions and QMIX learns a non-linear but monotonic combination of these contributions. At the very core of interpreting the actions of another agent, and ToM in general, is Bayesian reasoning. Fundamentally, asking what a given action by another agent implies about the state of the world requires understanding of why this action was taken. To illustrate this, we start out with an agent that has a given belief about the state-action history of the world, τ t, given her own action-observation history τ a t: B(τ t) = P (τ t |τ a t). Next the agent observes the action u a t of her team mate, a, and carries out a Bayesian update: where, with a slight abuse of notation, we have used (and will keep using) O(a, τ t) for the actionobservation history, τ a t, that from applying the observation function for agent a to τ t at each time step. Note that for non-deterministic observation functions we would have to marginalize over P (τ a t |τ t). Clearly, since agents have access to the policy of their teammate during centralised training, we could in principle evaluate this explicit Bayesian belief. However, beyond the practical difficulty of computing this explicit belief, when it is used as an input to the policy it will lead to prohibitively costly higher order beliefs. The typical workout for this is a public belief over private features which only conditions on common knowledge and can therefore be calculated by all agents individually, we refer to Moravčík et al.;; Foerster et al. (2018b) for more details. Instead, in this work we rely on RNNs to learn implicit representations of the sufficient statistics over the distribution of the Markov state given the action-observation histories, noting that they are unlikely to recover exact beliefs due to the issues mentioned above. Next we illustrate the impact of exploration on the beliefs, which we will do in the explicit (exact) case, since it serves as an upper bound on the accuracy of the implicit beliefs. Since we are looking at fully-cooperative settings we assume that the optimal policy of the agent is deterministic and any randomness is due to exploration. Given that we are focused on value based methods we furthermore assume an -greedy exploration scheme, noting that the same analysis can be extended to other methods. Under this exploration scheme π a (u a t |O(a, τ t)) becomes: where we have used u * (τ t) to indicate the greedy action of the agent a, u * (τ t) = arg max u Q a (u, O(a, τ t)) and I is the indicator function. While the first part corresponds to a filtering operator, in which the indicator function only attributes finite probability to those histories that are consistent with the action taken under greedy execution, the exploration term adds a fixed (history independent) probability, which effectively'blurs' the posterior: We find that the posterior includes an additional term of the form B(τ t) which carries over an unfiltered density over the trajectories from the prior. We further confirm that in the limit of = 1, the posterior collapses to the prior, P (τ t |τ a t, u a t) = B(τ t). This can be particularly worrisome in the context of our training setup, whereby different agents run different, and potentially high, throughout the course of training. It fundamentally makes the beliefs obtained less informative. While not making the above argument explicitly, the Bayesian Action Decoder (BAD), resolves this issue by shifting exploration to the level of deterministic partial policies, rather than action-level, and tracking an approximate Bayesian belief. As outlined in Section 1 this comes at a huge cost in the complexity of the method, the computation requirements and in the loss of generality of the method. In this paper we take a drastically simpler and different approach towards the issue. We note that the'blurring', which makes decoding of an action challenging, is entirely due to the -greedy exploration term. Furthermore, in order for another agent to do an implicit Bayesian update over an action taken, it is not required that this action is executed by the environment. Indeed, if we assume that other agents can observe the greedy action, u *, at every time step and condition their belief updated on this, the terms depending on disappear from the Bayesian update: Therefore, to have our cake and eat it, in the Simplified Action Decoder (SAD) the acting agent is allowed to'take' two actions at any given time step. The first action, u a, is the standard environment action, which gets executed as usual and is observed by all agents through the observation function at the next time step, as mentioned in Section 3. The second action, u *, is the greedy action of the active agent. This action does not get executed by the environment but instead is presented as an additional input to the other agents at the next time step during training, taking advantage of the centralized training regime during which information can be exchanged freely. Clearly we are not allowed to pass around extra information during decentralized control, but luckily this is not needed. Since we set to 0 at test time we can simply use the, now greedy, environment action obtained from the observation function as our greedy-action input. While this is most straight forward in settings where the last action is observed by other agents directly, in principle SAD can also be extended to settings where it is indirectly observed by all agents through the environment dynamics. In these cases we can replace the greedy-action side-channel with a learned inverse model that recovers the action from the observation history during execution. Furthermore, to encourage the agent to meaningfully decode the information contained in the greedy action, we can also add an auxiliary task to the training process. While this idea is compatible with any deep RL algorithm with minimal modifications, we use a recurrent version of DQN with distributed training, dueling networks and prioritized replay. We also learn a joint Q-function using VDN in order to address the challenges of multi-agent off-policy learning, please see Section 3 for details on all of these standard methods. • Each player knows their card but can't see the other player's card. • Player 1 acts first. • Player 2 observes Player 1's action and acts second. We first verify the effectiveness of SAD in the two step, two player matrix game from, which replicates the communication through action challenge of Hanabi in a highly simplified setting. In this fully cooperative game each player obtains a privately observed'card', which is drawn iid from two options. After observing her card, the first player takes one of three possible discrete actions. Crucially, the second player observes both her own private card and the team mate's action before acting herself, which establishes the opportunity to communicate. The payout is a function of both the two private cards and the two actions taken by both agents, as shown in Figure 1. Importantly, there are some obvious strategies that do not require any communication. For example, if both player learn to play the 2nd action, the payout is always 8 points, independent of the cards dealt. However, if the players do learn to communicate it is possible to achieve 10 points for every pair of cards dealt. Hanabi is a fully cooperative card game in which all players work together to complete piles of cards referred to as fireworks. Each card has a rank, 1 to 5, and a color, G / B / W / Y / R. Each firework (one per color) starts with a 1 and is finished once the 5 has been added. There are three 1s, one 5 and two of all other ranks for each of the colors, adding up to a total of 50 cards in the deck. The twist in Hanabi is that while players can observe the cards held by their team mates, they cannot observe their own cards and thus need to exchange information with each other in order to understand what cards can be played. There are two main means for doing so: First of all, players can take grounded hint actions, in which they reveal the subset of a team mate's hand that matches a specific rank or color. An example hint is "Your third and fifth card are 1s". These hint actions cost scarce information tokens, which can be replenished by discarding a card, an action that both removes the card from the game and makes it visible to all players. Finally players can also choose to play a card. If this card is the next card for the firework of the corresponding color, it is added to the firework and the team scores one point. Otherwise the card is removed from the game, the identity is made public, and the team loses one of the 3 life tokens. If the team runs out of life tokens before the end of the game, all points collected so far are lost and the game finishes immediately. These rules in a maximum score of 5 × 5 = 25 points in any game, which corresponds to all five fireworks being completed with five cards per firework. To ensure reproducibility and comparability of our we use the Hanabi Learning Environment (HLE) for all experimentation. For further details regarding Hanabi and the self-play part of the Hanabi challenge please see. We borrow some ideas and insights from prior distributed Q-learning methods while bring extensions to MARL as well as innovations to improve throughput and efficiency. and , we use a distributed prioritized replay buffer shared by N asynchronous actors and a centralized trainer that samples mini-batches from the replay buffer to update the model. In each actor thread, we run K environment sequentially and batch their observations together. The observation batch is then fed into an actor that utilize a GPU to compute a batch of actions. All asynchronous actors share one GPU and the trainer uses another GPU for gradient computation and model updates. This is different from prior works which run single actor and single environment in each thread on a CPU. Our method enables us to run a very large number of simulations with moderate computation resources. In all Hanabi experiments, we run N = 80 actor threads with K = 80 environments in each thread on single machine with 40 CPU cores and 2 GPUs. Without this architectural improvement, it may require at least a few hundred CPU cores to run 6400 Hanabi environments with neural network agents and simulations have to be distributed across multiple machines, which will greatly hinder the reproducibility and accessibility of such research. Please refer to Appendix A for implementation details and hyper-parameters. 6.1 MATRIX GAME 0 2000 4000 6000 8000 10000 Epoch As we can see in Figure 2, even in our simple matrix game the greedy action input makes a drastic difference. With an average reward of around 9.5 points, tabular IQL does well in this task, matching the BAD from. However, just by adding the greedy action as an additional input, we obtain an average performance of 9.97 ± 0.02. Results are averaged over 100 seeds, and shading is s.e.m. The code is available here: www.bit.ly/2mBJLyk. As shown in Table 1, our findings from the matrix game are for the most part confirmed on the challenging Hanabi benchmark. To illustrate the contributions of the different components, we compare average scores and win rates across 13 independent training runs of SAD and three different options: IQL is simply the recurrent DQN agent with parameter sharing, VDN is the same agent but also learns a joint Q-function and finally SAD & AuxTask is the SAD agent with the auxiliary task. While we find that SAD significantly outperforms our baselines (IQL and VDN) for 2, 4 and 5 players in terms of average score and/or win rate, there is no significant difference for 3 players, where VDN matches the performance of SAD. Interestingly, the auxiliary task only significantly helps the 2-player performance, where it substantially boosts the average score and win rate. In contrast, it drastically hurts performance for 3-5 players, which opens an interesting avenue for future work. For completeness we have included training curves showing average scores and s.e.m. across all training runs for all numbers of players for our methods and ablations in Appendix B. We find that for 5 players the auxiliary task drastically reduces the variance of SAD and intermittently leads to higher performance during training but ultimately in lower final performance. We can also clearly see that despite 72 hours of training and billions of samples consumed, the performance has not plateaued for 3-5 players, pointing to an obvious avenue for further improvements. The original numbers in the Hanabi challenge and BAD used population based training , effectively reporting maximum performance across a large number of different runs. Therefore, for reproducibility purposes, we report evaluations of the best model from our various training runs for each method in Table 2. As shown, under this reporting we establish a new SOTA for learning methods on the self-play part of the Hanabi challenge for 2-5 players, with the most drastic improvements being achieved for 3-5 players. In particular, we beat both the ACHA agent from and the BAD agent on average score, even though both of them used population based training and require more compute. We note that while we follow the counting convention proposed by the challenge paper, BAD was optimized for a different counting scheme, in which agents keep their scores when they run out of lives. This may explain the higher win rate (58.6%) of BAD combined with a relatively low mean score, which is exceeded even by our baseline methods. Once again, only the performance for 2-player is significantly improved by the auxiliary task and the 3-player setting is an outlier in the sense that SAD does not improve the best performance compared to VDN. In this paper we presented the Simplified Action Decoder (SAD), a novel deep multi-agent RL algorithm that allows agents to learn communication protocols in settings where no cheap-talk channel is available. On the challenging benchmark Hanabi our work substantially improves the SOTA for an RL method for all numbers of players. For two players SAD establishes a new high-score across any method. Furthermore we accomplish all of this with a method that is both simpler and requires less compute than previous advances. While these are encouraging steps, there is clearly more work to do. In particular, there remains a large performance gap between the numbers achieved by SAD and the known performance of hat-coding strategies for 3-5 players. One possible reason is that SAD does not undertake any explicit exploration in the space of possible conventions. Another promising route for future work is to integrate search with RL, since this has produced SOTA in a number of different domains including Poker, Go and backgammon. A NETWORK ARCHITECTURE AND HYPER-PAMAMETERS FOR HANABI Our Hanabi agent uses dueling network architecture . The main body of the network consists of 1 fully connected layer of 512 units and 2 LSTM layers of 512 units, followed by two output heads for value and advantages respectively. The same network configuration is used across all Hanabi experiments. We take the default featurization of HLE and replace the card knowledge section with the V0-Belief proposed by. The maximum length of an episode is capped at 80 steps and the entire episode is stored in the replay buffer as one training sample. This avoids the "slate hidden states" problem as described in so that we can simply initialize the hidden states of LSTM as zero during training. For exploration and experience prioritization, we follow the simple strategy as in and. Each actor executes an i -greedy policy where.., N − 1} but with a smaller = 0.1 and α = 7. For simplicity, all players of a game use the same epsilon. The per time-step priority δ t is the TD error and per episode priority is computed following δ e = η max t δ i + (1 − η)δ where η = 0.9. Priority exponent is set to 0.9 and importance sampling exponent is set to 0.6. We use n-step return and double Q-learning (van) for target computation during training. The discount factor γ is set to 0.999. The network is updated using Adam optimizer with learning rate = 6.25 × 10 −5 and = 1.5 × 10 −5. Trainer sends its network weights to all actors every 10 updates and target network is synchronized with online network every 2500 updates. These hyper-parameters are fixed across all experiments. In the baseline, we use Independent Q-Learning where each player estimates the Q value and selects action independently at each time-step. Note that all players need to operate on the observations in order to update their recurrent hidden states while only the current player has non-trivial legal moves and other players can only select'pass'. Each player then writes its own version of the episode into the prioritized replay buffer and they are sampled independently during training. The prioritized replay buffer contains 2 17 episodes. We warm up the replay buffer with 10,000 episodes before training starts. Batch size during training is 128 for games of different numbers of players. As mentioned in Section 4, the SAD agent is built on top of joint Q-function where the Q value is the sum of the individual Q value of all players given their own actions. One episode produces only one training sample with an extra dimension for the number of players. The replay buffer size is reduced to 2 16 for 2-player and 3-player games and 2 15 for 4-player and 5-player games. The batch sizes for 2-, 3-, 4-, 5-players are 64, 43, 32, 26 respectively to account for the fact that each sample contains more data. Auxiliary task can be added to the agent to help it decode the greedy action more effectively. In Hanabi, the natural choice is the predict the card of player's own hand. In our experiments, the auxiliary task is to predict the status of a card, which can be playable, discardable, or unknown. The loss is the average cross entropy loss per card and is simply added to the TD-error of reinforcement learning during training. Figure 3 shows learning curves of different algorithms averaged over 13 seeds per algorithm per player setting. Shading is error of the mean.

Represent the TLDR for finding the corresponding scientific article.

We develop Simplified Action Decoder, a simple MARL algorithm that beats previous SOTA on Hanabi by a big margin across 2- to 5-player games.

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.