Question: Represent the article for finding a single-sentence TLDR

Question: Standard image captioning tasks such as COCO and Flickr30k are factual, neutral in tone and (to a human) state the obvious (e.g., “a man playing a guitar”). While such tasks are useful to verify that a machine understands the content of an image, they are not engaging to humans as captions. With this in mind we define a new task, Personality-Captions, where the goal is to be as engaging to humans as possible by incorporating controllable style and personality traits. We collect and release a large dataset of 201,858 of such captions conditioned over 215 possible traits. We build models that combine existing work from (i) sentence representations (Mazaré et al., 2018) with Transformers trained on 1.7 billion dialogue examples; and (ii) image representations with ResNets trained on 3.5 billion social media images. We obtain state-of-the-art performance on Flickr30k and COCO, and strong performance on our new task. Finally, online evaluations validate that our task and models are engaging to humans, with our best model close to human performance. If we want machines to communicate with humans, they must be able to capture our interest, which means spanning both the ability to understand and the ability to be engaging, in particular to display emotion and personality as well as conversational function BID17 BID18 BID41 BID19.Communication grounded in images is naturally engaging to humans BID15, and yet the majority of studies in the machine learning community have so far focused on function only: standard image captioning BID36 requires the machine to generate a sentence which factually describes the elements of the scene in a neutral tone. Similarly, visual question answering BID2 and visual dialogue BID6 require the machine to answer factual questions about the contents of the image, either in single turn or dialogue form. They assess whether the machine can perform basic perception over the image which humans take for granted. Hence, they are useful for developing models that understand content, but are not useful as an end application unless the human cannot see the image, e.g. due to visual impairment BID13.Standard image captioning tasks simply state the obvious, and are not considered engaging captions by humans. For example, in the COCO BID5 and Flickr30k BID52 tasks, some examples of captions include "a large bus sitting next to a very tall building" and "a butcher cutting an animal to sell", which describe the contents of those images in a personality-free, factual manner. However, humans consider engaging and effective captions ones that "avoid stating the obvious", as shown by advice to human captioners outside of machine learning.1 For example, "If the bride and groom are smiling at each other, don't write that they are smiling at each other. The photo already visually shows what the subject is doing. Rephrase the caption to reflect the story behind the image". Moreover, it is considered that "conversational language works best. Write the caption as though you are talking to a family member or friend".2 These instructions for human captioners to engage human readers seem to be in direct opposition to standard captioning datasets. In this work we focus on image captioning that is engaging for humans by incorporating personality. As no large dataset exists that covers the range of human personalities, we build and release a new dataset, PERSONALITY-CAPTIONS, with 201,858 captions, each conditioned on one of 215 Standard captioning output: A plate with a sandwich and salad on it. Our model with different personality traits: Sweet That is a lovely sandwich. This sandwich looks so delicious! My goodness! Anxious I'm afraid this might make me sick if I eat it. Sympathetic I feel so bad for that carrot, about to be consumed. Arrogant I make better food than this Optimistic It will taste positively wonderful! Money-minded I would totally pay $100 for this plate. Figure 1: Comparison of a standard captioning model compared to our TransResNet model's predictions on the same image conditioned on various personality traits. Our model is trained on the new PERSONALITY-CAPTIONS dataset which covers 215 different personality traits. The standard captioning system used for comparison is the best COCO UPDOWN model described in Section 4.2. different possible personality traits. We show that such captions are far more engaging to humans than traditional ones. We then develop model architectures that can simultaneously understand image content and provide engaging captions for humans. To build strong models, we consider both retrieval and generative variants, and leverage state-of-the-art modules from both the vision and language domains. For image representations, we employ the work of BID28 that uses a ResNeXt architecture trained on 3.5 billion social media images which we apply to both. For text, we use a Transformer sentence representation following BID32 ) trained on 1.7 billion dialogue examples. Our generative model gives a new state-of-the-art on caption generation on COCO, and our retrieval architecture, TransResNet, yields the highest known hits@1 score on the Flickr30k dataset. To make the models more engaging to humans, we then adapt those same architectures to the PERSONALITY-CAPTIONS task by conditioning the input image on the given personality traits, giving strong performance on our new task. In particular, when compared to human captions, annotators preferred our retrieval model's captions over human ones 49.5% of the time, where the difference is not statistically significant. A large body of work has focused on developing image captioning datasets and models that work on them. In this paper we also perform experiments on the COCO BID5 and Flickr30k BID52 datasets, comparing to a range of models, including both generative models such as in BID45 BID49 BID1 and retrieval based such as in BID12 BID10 BID34. These setups measure the ability of models to understand the content of an image, but do not address more natural human communication. A number of works have tried to induce more engaging captions for human readers. One area of study is to make the caption personalized to the reader, e.g. by using user level features such as location and age BID7 or knowledge of the reader's active vocabulary BID38. Our work does not address this issue. Another research direction is to attempt to produce amusing captions either through wordplay (puns) BID4 or training on data from humour websites BID50. Our work focuses on a general set of personality traits, not on humour. Finally, closer to our work are approaches that attempt to model the style of the caption. Some methods have tried to learn style in an unsupervised fashion, as a supervised dataset like we have built in this work was not available. As a , evaluation was more challenging in those works, see e.g. BID30. Others such as BID51 have used small datasets like SentiCap BID31 with ∼800 images to inject sentiment into captions. BID11 collect a somewhat bigger dataset with 10,000 examples, FlickrStyle10K, but only covers two types of style (romantic and humorous). In contrast, our models are trained on the PERSONALITY-CAPTIONS dataset that has 215 traits and ∼200,000 images. Our work can also be linked to the more general area of human communication, separate from just factual captioning, in particular image grounded conversations between humans (Mostafazadeh . In those tasks, simple word overlap based automatic metrics are shown to perform weakly BID24 due to the intrinsically more diverse outputs in the tasks. As in those domains, we thus also perform human evaluations in this work to measure the engagingness of our setup and models. In terms of modeling, image captioning performance is clearly boosted with any advancements in image or text encoders, particularly the former. In this work we make use of the latest advancements in image encoding by using the work of BID28 which provides state-of-the-art performance on Imagenet image classification, but has so far not been applied to captioning. For text encoding we use the latest advances in attention-based representations using Transformers BID42 ; in particular, their use in retrieval models for dialogue by large-scale pretraining is adapted here for our captioning tasks. The PERSONALITY-CAPTIONS dataset is a large collection of (image, personality trait, caption) triples that we collected using crowd-workers, and will be made publicly available upon acceptance. We considered 215 possible personality traits which were constructed by selecting a subset from a curated list of 638 traits 3 that we deemed suitable for our captioning task. The traits are categorized into three classes: positive (e.g., sweet, happy, eloquent, humble, perceptive, witty), neutral (e.g., old-fashioned, skeptical, solemn, questioning) and negative (e.g., anxious, childish, critical, fickle, frivolous). Examples of traits that we did not use are allocentric, insouciant, flexible, earthy and invisible, due to the difficulty of their interpretation with respect to captioning an image. We use a randomly selected set of the images from the YFFC100M Dataset 4 to build our training, validation and test sets, selecting for each chosen image a random personality trait from our list. In each annotation round, an annotator is shown an image along with a trait. The annotators are then asked to write an engaging caption for the image in the context of the personality trait. It was emphasized that the personality trait describes a trait of the author of the caption, not properties of the content of the image. See Section D in the appendix for the exact instructions given to annotators. 4 ) trained on 3.5 billion Instagram pictures following the procedure described by BID28, which we refer to in the rest of the paper as ResNeXt-IG-3.5B. The authors provided the weights of their trained model to us. Both networks embed images in a 2048-dimensional vector which is the input for most of our models. In some of the caption generation models that make use of attention, we keep the spatial extent of the features by adapting activation before the last average pooling layer, and thus extract features with 7 × 7 × 2048 dimensions. We re-implemented three widely used previous/current state-of-the-art methods BID45 BID49 BID1 for image captioning as representatives of caption generation models. We refer them as SHOWTELL, SHOWATTTELL and UPDOWN respectively. We extract the image representation r I using the aforementioned image encoders. The SHOWTELL model uses image features with 2048 dimensions and the other models use image features with 7 × 7 × 2048 dimensions. In the case where we augment our models with personality traits, we learn an embedding for each trait, which is concatenated with each input of the decoder. Caption Decoders The SHOWTELL model first applies a linear projection to reduce image features into a feature vector with 512 dimensions. Similar to BID45, this embedding is the input for a LSTM model that generates the output sequence. In SHOWATTTELL, while the overall architecture is similar to BID49, we adopt the modification suggested by BID39 and input the attention-derived image features to the cell node of the LSTM. Finally, we use the UPDOWN model exactly as described in BID1. We perform a two-stage training strategy to train such caption generation models as proposed by BID39. In the first stage, we train the model to optimize the standard cross-entropy loss. In the second stage, we perform policy gradient with REINFORCE to optimize the non-differentiable reward function (CIDEr score in our case). During inference, we apply beam search (beam size=2) to decode the caption. We define a simple yet powerful retrieval architecture, named TransResNet. It works by projecting the image, personality, and caption in the same space S using image, personality, and text encoders. Image and Personality Encoders The representation r I of an image I is obtained by using the 2048-dimensional output of the image encoder described in Sec. 4.1 as input to a multi-layer perceptron with ReLU activation units and a final layer of 500 dimensions. To take advantage of personality traits in the PERSONALITY-CAPTIONS task, we embed each trait to a 500-dimensional vector to obtain its representation r P. Image and personality representations are then summed. Caption Encoders Each caption is encoded into a vector r C of the same size using a Transformer architecture BID42, followed by a two layer perceptron. We try two sizes of Transformer: a larger architecture (4 layers, 300 hidden units, 6 attention heads) and a smaller one (2 layers, 300 hidden units, 4 attention heads). We consider either training from scratch or pretraining our models. We either pretrain only the word embeddings, i.e. where we initialize word vectors trained using fastText BID3 ) trained on Wikipedia, or pretrain the entire encoder. For the latter, we follow the setup described in BID32: we train two encoders on a next-utterance retrieval task on a dataset of dialogs containing 1.7 billion pairs of utterances, where one encodes the context and another the candidates for the next utterance, their dot product indicates the degree of match, and they are trained with negative log-likelihood and k-negative sampling. We then initialize our system using the weights of the candidate encoder only, and then train on our task. For comparison, we also consider a simple bag-of-words encoder (pretrained or not). In this case, r C is the sum of the 300-dimensional word embeddings of the caption. In each case, given an input image and personality trait (I, P) and a candidate caption C, the score of the final combination is then computed as s(I, P, C) = (r I + r P) · r C. Figure 2: Our architecture TransResNet, used for our retrieval models. Training and Inference Given a pair I, P, and a set of candidates (c 1, .., c N), at inference time the predicted caption is the candidate c i that maximizes the score s(I, P, c i). At training time we pass a set of scores through a softmax and train to maximize the log-likelihood of the correct responses. We use mini-batches of 500 training examples; for each example, we use the captions of the other elements of the batch as negatives. Our overall TransResNet architecture is detailed in Figure 2. We first test our architectures on traditional caption datasets to assess their ability to factually describe the contents of images in a neutral tone. We then apply the same architectures to PERSONALITY-CAPTIONS to assess their ability to produce engaging captions conditioned on personality. The latter is tested with both automatic metrics and human evaluation of engagingness. Generative Models For our generative models, we test the quality of our implementations of existing models (SHOWTELL, SHOWATTTELL and UPDOWN) as well as the quality of our image encoders, where we compare ResNet152 and ResNeXt-IG-3.5B. We report performance on the COCO caption dataset BID23. We evaluate BLEU BID37, ROUGE-L BID22, CIDEr and SPICE BID0 and compare model's performances to state-of-the-art models under BID20's setting. The are shown in TAB3. Models trained with ResNeXt-IG-3.5B features consistently outperform their counterparts with ResNet152 features, demonstrating the effectiveness of ResNeXt-IG-3.5B beyond the original image classification and detection in BID28. More importantly, our best model (UPDOWN) either outperforms or is competitive with state-ofthe-art single model performance BID1 Retrieval Models We compare our retrieval architecture, TransResNet, to existing models reported in the literature on the COCO caption and Flickr30k tasks. We evaluate retrieval metrics R@1, R@5, R@10, and compare our model performance to state-of-the-art models under the setting of BID20 ). The are given in Table 4 (for more details, see TAB0 in the appendix for COCO and Flickr30k, respectively). For our model, we see large improvements using ResNeXt-IG-3.5B compared to Resnet152, and stronger performance with a Transformer-based text encoding compared to a bag-of-words encoding. Pretraining the text encoder also helps substantially (see Appendix A for more analysis of pretraining of our systems). Our best models are competitive on COCO and are state-of-the-art on Flickr30k by a large margin (68.4 R@1 for our model vs. 56.8 R@1 for the previous state-of-the-art). Generative models We first train the aforementioned caption generation models without using the personality traits. This setting is similar to standard image captioning, and TAB5 shows that the three caption generation models that we considered are ranked in the same order, with the UPDOWN model being the most effective. The best are again obtained using the ResNeXt-IG-3.5B features. Adding the embedding of the personality trait allows our best model to reach a CIDEr score of 22.0, showing the importance of modeling personality in our new task. Note that all scores are lower than for the COCO captioning task. Indeed standard image captioning tries to produce text descriptions that are semantically equivalent to the image, whereas PERSONALITY-CAPTIONS captures how a human responds to a given image when speaking to another human when both can see the image -which is rarely to simply state its contents. Hence, PERSONALITY-CAPTIONS has intrinsically more diverse outputs, similar to found in other human communication tasks BID24. For that reason we perform human evaluation in Section 5.3 in addition to automatic evaluations. Retrieval models Similarly we compare the effect of various configurations of our retrieval model, TransResNet. The models are evaluated in terms of R@1, where for each sample there are 100 candidates to rank: 99 randomly chosen candidates from the test set plus the true label. TAB6 shows the scores obtained on the test set of PERSONALITY-CAPTIONS. Again, the impact of using the image encoder trained on billions of images is considerable, we obtain 53.5% for our best ResNeXt-IG-3.5B model, and 34.4% for our best Resnet152 model. Conditioning on the personality traits is also very important (53.5% vs. 38.5% R@1 for the best variants with and without conditioning). Transformer text encoders also outperform bag-of-word embeddings encoders, Table 4: Retrieval model performance on Flickr30k and COCO caption using the splits of BID20 where pretraining for either type of encoder helps. For Transformers pretraining the whole network performed better than just pretraining the word embeddings, see Appendix A.Example predictions of our best model, TransResNet (ResNeXt-IG-3.5B), are given in TAB2. The goal of PERSONALITY-CAPTIONS is to be engaging to human readers by emulating human personality traits. We thus test our task and models in a set of human evaluation studies. Evaluation Setup Using 500 random images from the YFCC-100M dataset that are not present in PERSONALITY-CAPTIONS, we obtain captions for them using a variety of methods, as outlined in the sections below, including both human authored captions and model predicted captions. Using a separate set of human annotators, comparisons are then done pairwise: we show each image, with two captions to compare, to five separate annotators and ask them to choose the "more engaging" caption. For experiments where both captions are conditioned on a personality, we show the annotator the personality; otherwise, the personality is hidden. We then report the percentage of the time one method is chosen over the other. The are summarized in FIG0. We compare human authored PERSONALITY-CAPTIONS captions to human authored traditional neutral (COCO-like) captions. Captions conditioned on a personality were found to be significantly more engaging than those that were neutral captions of the image, with a win rate of 64.5%, which is statistically significant using a binomial two-tailed test. We compare the best-performing models from Section 5.2 to human authored PERSONALITY-CAPTIONS captions. For each test image we condition both human and model on the same (randomly-chosen) personality trait. Our best TransResNet model from Sec. We also compare our models in a pairwise fashion directly, as measured by human annotators. The given in FIG0 (all statistically significant) show the same trends as we observed before: TransResNet with ResNext-IG-3.5B outperforms the same model with ResNet152 features with a win rate of 55.2%, showing the importance of image features. Additionally, TransResNetwith ResNext-IG-3.5B image features (with no pretraining) also substantially outperforms the UPDOWN model using ResNext-IG-3.5B with a winrate of 80.1%. In this work we consider models that can simultaneously understand image content and provide engaging captions for humans. To build strong models, we first leverage the latest advances in image and sentence encoding to create generative and retrieval models that perform well on standard image captioning tasks. In particular, we attain a new state-of-the-art on caption generation on COCO, and introduce a new retrieval architecture, TransResNet, that yields the highest known hits@1 score on the Flickr30k dataset. To make the models more engaging to humans, we then condition them on a set of controllable personality traits. To that end, we collect a large dataset, PERSONALITY-CAPTIONS to train such models. Using automatic metrics and human evaluations, we show that our best system is able to produce captions that are close to matching human performance in terms of engagement. Our benchmark will be made publicly available to encourage further model development, leaving the possibility of superhuman performance coming soon in this domain. A IMPACT OF PRETRAINED WORD EMBEDDINGS AND TEXT ENCODERS Table 7: More detailed for retrieval model performance on COCO Captions using the splits of BID20. For our TransResNet models, we compare two types of pretraining: Full indicates a model with a pretrained text encoder, while Word indicates a model with pretrained word embeddings only. Caption retrieval Pretraining R@1 R@5 R@10 Med Rank 1k Images m-CNN BID27 42.8 -84.1 2.0 UVS BID21 43.4 75.7 85.8 2.0 HM-LSTM 43.9 -87.8 2.0 Order Embeddings BID44 46.7 -88.9 2.0 Embedding Net BID46 50.4 79.3 69.4 -DSPE+Fisher Vector 50.1 -89.2 -sm-LSTM BID16 53.2 83.1 91.5 1.0 VSE++ (ResNet, FT) BID10 64.6 90.0 95.7 1.0 GXN (i2t+t2i) BID12 68. BID44 23.3 -65.0 5.0 VSE++ (ResNet, FT) BID10 41.3 71.1 81.2 2.0 GXN (i2t+t2i) BID12 42. Table 8: Retrieval model performance on Flickr30k using the splits of BID20. For our models, we compare two types of pretraining: Full indicates a model with a pretrained text encoder, while Word indicates a model with pretrained word embeddings only. Caption retrieval Pretraining R@1 R@5 R@10 Med Rank UVS BID21 23.0 50.7 62.9 5.0 UVS (Github) 29.8 58.4 70.5 4.0 Embedding Net BID46 40.7 69.7 79.2 -DAN BID34 41.4 73.5 82.5 2.0 sm-LSTM BID16 42.5 71.9 81.5 2.0 2WayNet BID8 49.8 67.5 --VSE++ (ResNet, FT) BID10 52.9 80.5 87.2 1.0 DAN (ResNet) BID34 55.0 81.8 89.0 1.0 GXN (i2t+t2i) BID12 56. Engaging-only Captions Instead of asking to author a caption based on a personality trait, we can ask humans to simply write an "engaging" caption instead, providing them with no personality cue. We found that human annotators overall preferred captions written by those unconditioned on a personality by a slight margin (∼ 54%). To further understand this difference, we split the images into three subsets based on the personality on which the PERSONALITY-CAPTIONS annotator conditioned their caption, i.e. whether the personality was positive, negative, or neutral. We then examined the engagingness rates of images for each of these subsets. In the set where PERSONALITY-CAPTIONS annotators were provided with positive personalities, which totaled 185 out of the 500 images, we found that human annotators preferred the captions conditioned on the personality to those that were not. However, in the other two sets, we found that the unconditioned captions were preferred to the negative or neutral ones. For these two subsets, we believe that, without the context of any personality, annotators may have preferred the inherently more positive caption provided by someone who was asked to be engaging but was not conditioned on a personality. Diversity of captions We found that the captions written via our method were not only more engaging for positive personality traits, but also ed in more diversity in terms of personality traits. To measure this diversity, we constructed a model that predicted the personality of a given comment. The classifier consists in the same Transformer as described in 4.3, pre-trained on the same large dialog corpus, followed by a softmax over 215 units. We then compare the total number of personality types as predicted by the classifier among each type of human-labeled data: "engaging" captions conditioned on personalities, "engaging" captions not conditioned on personalities, and traditional image captions. That is, we look at each caption given by the human annotators, assign it a personality via the classifier, and then look at the total set of personalities we have at the end for each set of human-labeled data. For example, out of the 500 human-generated traditional captions, the classifier found 63% of all possible positive personalities in this set of captions. As indicated in TAB0, the human annotators who were assigned a personality produce more diverse captions, particularly negatively and neutrally conditioned ones, as compared to human annotators who are just told to be "engaging" or those who are told to write an image caption. The ultimate test of our generative and retrieval models on PERSONALITY-CAPTIONS is performed using human evaluations. Comparing them using automatic metrics is typically difficult because retrieval methods perform well with ranking metrics they are optimized for and generative models perform well with word overlap metrics they are optimized for, but neither of these necessarily correlate with human judgements, see e.g..Nevertheless, here we compare our generative and retrieval models directly with automatic metrics on COCO. We computed the BLEU, CIDEr, SPICE, and ROUGE-L scores for our best TransResNet model. The comparison is given in TAB0. TAB0: Generative and retrieval model performance on COCO caption using the test split of BID20 That is so cool! I I love street art! OptimisticThe future is bright for people who can dream in artistic ways. Critical I do believe this taggers verbage is a tad junvenile Charming What a charming wall. Adventurous I think I could create art like that, I will go learn and take action. The color of this flower is absolutely astounding. I can't believe it. Wishful I always wish I could grow these types of flowers. Sweet Beautiful flowers! I would give them to you. RomanticThe pink flowers would make a beautiful bouquet for my wife. Oh my, what a lovely purple color of nature's new sprouts! TAB0: More example predictions from our best TRANSRESNET model on the PERSONALITY-CAPTIONS validation set.

We develop engaging image captioning models conditioned on personality that are also state of the art on regular captioning tasks.

We propose a new video understanding benchmark, with tasks that by-design require temporal reasoning to be solved, unlike most existing video datasets.


Question: Represent the article for finding a single-sentence TLDR

Question: Machine learning (ML) models trained by differentially private stochastic gradient descent (DP-SGD) have much lower utility than the non-private ones. To mitigate this degradation, we propose a DP Laplacian smoothing SGD (DP-LSSGD) to train ML models with differential privacy (DP) guarantees. At the core of DP-LSSGD is the Laplacian smoothing, which smooths out the Gaussian noise used in the Gaussian mechanism. Under the same amount of noise used in the Gaussian mechanism, DP-LSSGD attains the same DP guarantee, but a better utility especially for the scenarios with strong DP guarantees. In practice, DP-LSSGD makes training both convex and nonconvex ML models more stable and enables the trained models to generalize better. The proposed algorithm is simple to implement and the extra computational complexity and memory overhead compared with DP-SGD are negligible. DP-LSSGD is applicable to train a large variety of ML models, including DNNs. Many released machine learning (ML) models are trained on sensitive data that are often crowdsourced or contain private information (; ;). With overparameterization, deep neural nets (DNNs) can memorize the private training data, and it is possible to recover them and break the privacy by attacking the released models . For example, Fredrikson et al. demonstrated that a model-inversion attack can recover training images from a facial recognition system . Protecting the private data is one of the most critical tasks in ML. Differential privacy (DP) is a theoretically rigorous tool for designing algorithms on aggregated databases with a privacy guarantee. The idea is to add a certain amount of noise to randomize the output of a given algorithm such that the attackers cannot distinguish outputs of any two adjacent input datasets that differ in only one entry. For repeated applications of additive noise based mechanisms, many tools have been invented to analyze the DP guarantee for the model obtained at the final stage. These include the basic and strong composition theorems and their refinements (; 2010;), the moments accountant , etc. Beyond the original notion of DP, there are also many other ways to define the privacy, e.g., local DP , concentrated/zeroconcentrated DP , and Rényi-DP (RDP) . Differentially private stochastic gradient descent (DP-SGD) reduces the utility of the trained models severely compared with SGD. As shown in Figure 1, the training and validation losses of the logistic regression on the MNIST dataset increase rapidly when the DP guarantee becomes stronger. The convolutional neural net (CNN) 1 trained by DP-SGD has much lower testing accuracy than the non-private one on the MNIST. We will discuss the detailed experimental settings in Section 4. A natural question raised from such performance degradations is: Can we improve DP-SGD, with negligible extra computational complexity and memory cost, such that it can be used to train general ML models with improved utility? We answer the above question affirmatively by proposing differentially private Laplacian smoothing SGD (DP-LSSGD) to improve the utility in privacy-preserving empirical risk minimization (ERM). DP-LSSGD leverages the Laplacian smoothing as a post-processing to smooth the injected Gaussian noise in the differentially private SGD (DP-SGD) to improve the convergence of DP-SGD in training ML models with DP guarantee. The main contributions of our work are highlighted as follows: • We propose DP-LSSGD and prove its privacy and utility guarantees for convex/nonconvex optimizations. We prove that under the same privacy budget, DP-LSSGD achieves better utility, excluding a small term that is usually dominated by the other terms, than DP-SGD by a factor that is much less than one for convex optimization. • We perform a large number of experiments logistic regression and CNN to verify the utility improvement by using DP-LSSGD. Numerical show that DP-LSSGD remarkably reduces training and validation losses and improves the generalization of the trained private models. In Table 1, we compare the privacy and utility guarantees of DP-LSSGD and DP-SGD. For the utility, the notationÕ(·) hides the same constant and log factors for each bound. The constants d and n denote the dimension of the model's parameters and the number of training points, respectively. The numbers γ and β are positive constants that are strictly less than one, and D 0, D σ, G are positive constants, which will be defined in Section 3. σ, we will discuss this in detail in Section 4. There is a massive volume of research over the past decade on designing algorithms for privacypreserving ML. Objective perturbation, output perturbation, and gradient perturbation are the three major approaches to perform ERM with a DP guarantee.; considered both output and objective perturbations for privacy-preserving ERM, and gave theoretical guarantees for both privacy and utility for logistic regression and SVM. numerically studied the effects of learning rate and batch size in DP-ERM. studied stability, learnability and other properties of DP-ERM. proposed an adaptive per-iteration privacy budget in concentrated DP gradient descent. Variance reduction techniques, e.g., SVRG, have also been introduced to DP-ERM. The utility bound of DP-SGD has also been analyzed for both convex and nonconvex smooth objectives (; . analyzed the excess empirical risk of DP-ERM in a distributed setting. Besides ERM, many other ML models have been made differentially private. These include: clustering (; Y. ;), matrix completion , online learning , sparse learning , and topic modeling . exploited the ill-conditionedness of inverse problems to design algorithms to release differentially private measurements of the physical system. considered sparse linear regression in the local DP models. proposed distributed selective SGD to train deep neural nets (DNNs) with a DP guarantee in a distributed system, however, the obtained privacy guarantee was very loose. considered applying DP-SGD to train DNNs in a centralized setting. They clipped the gradient 2 norm to bound the sensitivity and invented the moment accountant to get better privacy loss estimation. proposed Private Aggregation of Teacher Ensembles/PATE based on the semi-supervised transfer learning to train DNNs, and this framework improves both privacy and utility on top of the work by. introduced new noisy aggregation mechanisms for teacher ensembles that enable a tighter theoretical DP guarantee. The modified PATE is scalable to the large dataset and applicable to more diversified ML tasks. Laplacian smoothing (LS) can be regarded as a denoising technique that performs post-processing on the Gaussian noise injected stochastic gradient. Denoising has been used in the DP earlier: Postprocessing can enforce consistency of contingency table releases and leads to accurate estimation of the degree distribution of private network . showed that post-processing by projecting linear regression solutions, when the ground truth solution is sparse, to a given 1 -ball can remarkably reduce the estimation error. used Expectation-Maximization to denoise a class of graphical models' parameters. showed that in the output perturbation based differentially private algorithm design, denoising dramatically improves the accuracy of the Gaussian mechanism in the high-dimensional regime. To the best of our knowledge, we are the first to design a denoising technique on the Gaussian noise injected gradient to improve the utility of the trained private ML models. We use boldface upper-case letters A, B to denote matrices and boldface lower-case letters x, y to denote vectors. For vectors x and y and positive definite matrix A, we use x 2 and x A to denote the 2 -norm and the induced norm by A, respectively; x, y denotes the inner product of x and y; and λ i (A) denotes the i-th largest eigenvalue of A. We denote the set of numbers from 1 to n by [n]. N (0, I d×d) represents d-dimensional standard Gaussian. This paper is organized in the following way: In Section 2, we introduce the DP-LSSGD algorithm. In Section 3, we analyze the privacy and utility guarantees of DP-LSSGD for both convex and nonconvex optimizations. We numerically verify the efficiency of DP-LSSGD in Section 4. We conclude this work and point out some future directions in Section 5. 2 PROBLEM SETUP AND ALGORITHM In this paper, we consider empirical risk minimization problem as follows. Given a training set S = {(x 1, y 1),..., (x n, y n)} drawn from some unknown but fixed distribution, we aim to find an empirical risk minimizer that minimizes the empirical risk as follows, where F (w) is the empirical risk (a.k.a., training loss), f i (w) = (w; x i, y i) is the loss function of a given ML model defined on the i-th training example (x i, y i), and w ∈ R d is the model parameter we want to learn. Empirical risk minimization serves as the mathematical foundation for training many ML models that are mentioned above. The LSSGD for solving is given by where η is the learning rate, ∇f i k denotes the stochastic gradient of F evaluated from the pair of input-output {x i k, y i k}, and B k is a random subset of size b from [n]. Let A σ = I − σL for σ ≥ 0 being a constant, where I ∈ R d×d and L ∈ R d×d are the identity and the discrete one-dimensional Laplacian matrix with periodic boundary condition, respectively. Therefore, When σ = 0, LSSGD reduces to SGD. Note that A σ is positive definite with condition number 1 + 4σ that is independent of A σ's dimension, and LSSGD guarantees the same convergence rate as SGD in both convex and nonconvex optimization. Moreover, Laplacian smoothing (LS) can reduce the variance of SGD on-thefly, and lead to better generalization in training many ML models including DNNs . T and * is the convolution operator. By the fast Fourier transform (FFT), we have A, where the division in the right hand side parentheses is performed in a coordinate wise way. DP ERM aims to learn a DP model, w, for the problem. A common approach is injecting Gaussian noise into the stochastic gradient, and it ing in the following DP-SGD where n is the injected Gaussian noise for DP guarantee. Note that the LS matrix A −1 σ can remove the noise in v. If we assume v is the initial signal, then A −1 σ v can be regarded as performing an approximate diffusion step on the initial noisy signal which removes the noise from v. We will provide a detailed argument for the diffusion process in the appendix. As numerical illustrations, we consider the following two signals: We reshape v 2 into 1D with row-major ordering and then perform LS. Figure 2 shows that LS can remove noise efficiently. This noise removal property enables LSSGD to be more stable to the noise injected stochastic gradient, therefore improves training DP models with gradient perturbations. We propose the following DP-LSSGD for solving with DP guarantee In this scheme, we first inject the noise n to the stochastic gradient ∇f i k (w k), and then apply the LS operator A −1 σ to denoise the noisy stochastic gradient, ∇f i k (w k) + n, on-the-fly. We assume that each component function f i in is G-Lipschitz. The DP-LSSGD for finite-sum optimization is summarized in Algorithm 1. Compared with LSSGD, the main difference of DP-LSSGD lies in injecting Gaussian noise into the stochastic gradient, before applying the Laplacian smoothing, to guarantee the DP. initial guess of w, (, δ): the privacy budget, η: the step size, T: the total number of iterations. and ν is defined in Theorem 1, and In this section, we present the privacy and utility guarantees for DP-LSSGD. The technical proofs are provided in the appendix. Definition 1 ((, δ)-DP). A randomized mechanism M: S N → R satisfies (, δ)-DP if for any two adjacent datasets S, S ∈ S N differing by one element, and any output subset O ⊆ R, it holds that Theorem 1 (Privacy Guarantee). Suppose that each component function f i is G-Lipschitz. Given the total number of iterations T, for any δ > 0 and privacy budget 2 ≤ 5T log(1/δ)b 2 /n 2, DP-LSSGD, with injected Gaussian noise N (0, ν 2) for each coordinate, satisfies (, δ)-DP with ν 2 = 8T αG 2 /(n 2), where α = 2 log(1/δ)/ + 1. Remark 1. It is straightforward to show that the noise in Theorem 1 is in fact also tight to guarantee the (, δ)-DP for DP-SGD. For convex ERM, DP-LSSGD guarantees the following utility in terms of the gap between the ergodic average of the points along the DP-LSSGD path and the optimal solution w *. Theorem 2 (Utility Guarantee for convex optimization). Suppose F is convex and each component function Aσ and w * is the global minimizer of F, the DP-LSSGD outputw =, where ω = 2σ+1− √ 4σ+1 2σ < 1. That is, γ converge to 0 almost exponentially as the dimension, d, increases. Remark 2. In the above utility bound for convex optimization, for different σ (σ = 0 corresponds to DP-SGD), the only difference lies in the term γ(D σ + G 2). The first part γD σ depends on the gap between initialization w 0 and the optimal solution w *. The second part γG 2 decrease monotonically as σ increases. σ should be selected to get an optimal trade-off between these two parts. Based on our test on multi-class logistic regression for MNIST classification, σ = 0 always outperforms the case when σ = 0. For nonconvex ERM, DP-LSSGD has the following utility bound measured in gradient norm. Theorem 3 (Utility Guarantee for nonconvex optimization). Suppose that F is nonconvex and each component function f i is G-Lipschitz and has L-Lipschitz continuous gradient. Given with w * being the global minimum of F, then the DP-LSSGD outputw = T −1 k=0 w k /T satisfies the following utility Proposition 2. In Theorem 3, β = It is worth noting that if we use the 2 -norm instead of the induced norm, we have the following utility guarantee In the 2 -norm, DP-LSSGD has a bigger utility upper bound than DP-SGD (set σ = 0 in ζ). However, this does not mean that DP-LSSGD has worse performance. To see this point, let us consider the following simple nonconvex function For two points a 1 = and a 2 = (1, √ 3/2), the distance to the local minima a * = are 2 and √ 7/2, while ∇f (a 1) 2 = 1 and ∇f (a 2) 2 = √ 13/2. So a 2 is closer to the local minima a * than a 1 while its gradient has a larger 2 -norm. In this section, we verify the efficiency of DP-LSSGD in training multi-class logistic regression and CNNs for MNIST and CIFAR10 classification. We use v ← v/ max (1, v 2 /C) to clip the gradient 2 -norms of the CNNs to C. The gradient clipping guarantee the Lipschitz condition for the objective functions. We train all the models below with (, 10 −5)-DP guarantee for different. For Logistic regression we use the privacy budget given by Theorem 1, and for CNNs we use the privacy budget in the Tensorflow privacy (Andrew & et al., 2019). We checked that these two privacy budgets are consistent. We ran 50 epochs of DP-LSSGD with learning rate scheduled as 1/t with t being the index of the iteration to train the 2 -regularized (regularization constant 10 −4) multi-class logistic regression. We split the training data into 50K/10K with batch size 128 for cross-validation. We plot the evolution of training and validation loss over iterations for privacy budgets (0.2, 10 −5) and (0.1, 10 −5) in Figure 3. We see that the training loss curve of DP-SGD (σ = 0) is much higher and more oscillatory (log-scale on the y-axis) than that of DP-LSSGD (σ = 1, 3). Also, the validation loss of the model trained by DP-LSSGD decays faster and has a much smaller loss value than that of the model trained by DP-SGD. Moreover, when the privacy guarantee gets stronger, the utility improvement by DP-LSSGD becomes more significant. Next, consider the testing accuracy of the multi-class logistic regression trained with (, 10 −5)-DP guarantee by DP-LSSGD includes σ = 0, i.e., DP-SGD. We list the test accuracy of logistic regression trained in different settings in Table 2. These reveal that DP-LSSGD with σ = 1, 2, 3 can improve the accuracy of the trained private model and also reduce the variance, especially when the privacy guarantee is very strong, e.g., (0.1, 10 −5). We know that the step size in DP-SGD/DP-LSSGD may affect the accuracy of the trained private models. We try different step size scheduling of the form {a/t|a = 0.5, 1.0, 1.5, 2.0, 2.5, 3.0}, where t is again the index of iteration, and all the other hyper-parameters are used the same as before. Figure. 4 plots the test accuracy of the logistic regression model trained with different learning rate scheduling and different privacy budget. We see that the private logistic regression model trained by DP-LSSGD always outperforms DP-SGD. In this subsection, we consider training a small CNN 2 with DP-guarantee for MNIST classification. We implement DP-LSSGD and DP-LSAdam (simply replace the noisy gradient in DP-Adam in the Tensorflow privacy with the Laplacian smoothed surrogate) into the Tensorflow privacy framework (Andrew & et al., 2019). We use the default learning rate 0.15 for DP-(LS)SGD and 0.001 for DP-(LS)Adam and decay them by a factor of 10 at the 10K-th iteration, norm clipping, batch size, and micro-batches. We vary the noise multiplier (NM), and larger NM guarantees stronger DP. As shown in Figure 5, the privacy budget increases at exactly the same speed (dashed red line) for four optimization algorithms. When the NM is large, i.e., DP-guarantee is strong, DP-SGD performs very well in the initial period. However, after a few epochs, the validation accuracy gets highly oscillatory and decays. DP-LSSGD can mitigate the training instability issue of DP-SGD. DP-Adam outperforms DP-LSSGD, and DP-LSAdam can further improve validation accuracy on top of DP-Adam. Next, we consider the effects of the LS constant (σ) and the learning rate in training the DP-CNN for MNIST classification. We fixed the NM to be 10, and run 60 epochs of DP-SGD and DP-LSSGD with different σ and different learning rate. We show the comparison of DP-SGD with DP-LSSGD with different σ in the left panel of Figure 6, and we see that as σ increases it becomes more stable in training CNNs with DP-guarantee even though initially it becomes slightly slower. In the middle panel of Figure 6, we plot the evolution of validation accuracy curves of the DP-CNN trained by DP-SGD and DP-LSSGD with different learning rate, where the solid lines represent for DP-LSSGD and dashed lines for DP-SGD. DP-LSSGD outperforms DP-SGD in all learning rates tested, and DP-LSSGD is much more stable than DP-SGD when a larger learning rate is used. Finally, we go back to the accuracy degradation problem raised in Figure 1. As shown in Figure 3, LS can efficiently reduce both training and validation losses in training multi-class logistic regression for MNIST classification. Moreover, as shown in the right panel of Figure 6, DP-LSSGD can improve the testing accuracy of the CNN used above significantly. In particular, DP-LSSGD improves the testing accuracy of CNN by 3.2% and 5.0% for (0.4, 10 −5) and (0.2, 10 −5), respectively, on top of DP-SGD. DP-LSAdam can further boost test accuracy. All the accuracies associated with any given privacy budget in Figure 6 (right panel), are the optimal ones searched over the obtained in the above experiments with different learning rate, number of epochs, and NM. Due to page limitation, we put the of DP-CNN for CIFAR10 classification in the appendix. In this paper, we integrated Laplacian smoothing with DP-SGD for privacy-presrving ERM. The ing algorithm is simple to implement and the extra computational cost compared with the DP-SGD is almost negligible. We show that DP-LSSGD can improve the utility of the trained private ML models both numerically and theoretically. It is straightforward to combine LS with other variance reduction technique, e.g., SVRG . To prove the privacy guarantee in Theorem 1, we first introduce the following 2 -sensitivity. Definition 2 (2 -Sensitivity). For any given function f (·), the 2 -sensitivity of f is defined by where S − S 1 = 1 means the data sets S and S differ in only one entry. We will adapt the concepts and techniques of Rényi DP (RDP) to prove the DP-guarantee of the proposed DP-LSSGD. Definition 3 (RDP). For α > 1 and ρ > 0, a randomized mechanism M: S n → R satisfies (α, ρ)-Rényi DP, i.e., (α, ρ)-RDP, if for all adjacent datasets S, S ∈ S n differing by one element, we have where the expectation is taken over M(S). Lemma 1. ) Given a function q: S n → R, the Gaussian Mechanism M = q(S) + n, where n ∼ N (0, ν 2 I), satisfies (α, α∆ 2 (q)/(2ν 2))-RDP. In addition, if we apply the mechanism M to a subset of samples using uniform sampling without replacement, M satisfies Moreover, the input of the i-th mechanism can be based on outputs of the previous (i − 1) mechanisms. Lemma 3. If a randomized mechanism M: S n → R satisfies (α, ρ)-RDP, then M satisfies (ρ + log(1/δ)/(α − 1), δ)-DP for all δ ∈. With the definition (Def. 3) and guarantees of RDP (Lemmas 1 and 2), and the connection between RDP and (, δ)-DP (Lemma 3), we can prove the following DP-guarantee for DP-LSSGD. Proof of Theorem 1. Let us denote the update of DP-SGD and DP-LSSGD at the k-th iteration starting from any given points w k andw k, respectively, as where B k is a mini batch that are drawn uniformly from [n], and |B k | = b is the mini batch size. We will show that with the aforementioned Gaussian noise N (0, ν 2) for each coordinate of n, the output of DP-SGD,w, after T iterations is (, δ)-DP. Let us consider the mechanismM k = b. According to Lemma 1, if we add noise with variance the mechanism M k will satisfy α, (n 2 /b 2) log(1/δ)/ 2(α − 1)T -RDP. By post-processing theorem, we immediately have that under the same noise, According to Lemma 1,M k will satisfy α, log(1/δ)/(α−1)T -RDP provided that ν 2 ≥ 1/1.25, because τ = b/n. Let α = 2 log(1/δ)/ + 1, we obtain thatM k satisfies 2 log(1/δ)/ + 1, /(2T) -RDP as long as we have Therefore, the following condition suffices Therefore, according to Lemma 2, we have w k satisfies 2 log(1/δ)/ + 1, k /(2T) -RDP. Finally, by Lemma 3, we have w k satisfies k /(2T) + /2, δ -DP. Therefore, the output of DP-SGD,w, is (, δ)-DP. Remark 3. In the above proof, we used the following estimate of the 2 sensitivity σ g, then according to we have where d is the dimension of d, and Moreover, if we assume the g is randomly sampled from a unit ball in a high dimensional space, then a high probability estimation of the compression ratio of the 2 norm can be derived from Lemma. 5., so for the above noise, it can give much stronger privacy guarantee. To prove the utility guarantee for convex optimization, we first show that the LS operator compresses the 2 norm of any given Gaussian random vector with a specific ratio in expectation. Lemma 4. Let x ∈ R d be the standard Gaussian random vector. Then Proof of Theorem 2. Recall that we have the following update rule w, where i k are drawn uniformly from [n], and n ∼ N (0, ν 2 I). Observe that Taking expectation with respect to i k and n given w k, we have where the second inequality is due to the convexity of F, and Lemma 4. It implies that Now taking the full expectation and summing up over T iterations, we have where According to the definition ofw and the convexity of F, we obtain To prove the utility guarantee for nonconvex optimization, we need the following lemma, which shows that the LS operator compresses the 2 norms of any given Gaussian random vector with a specific ratio in expectation. Lemma 5. Let x ∈ R d be the standard Gaussian random vector. Then Proof of Lemma 5. Let the eigenvalue decomposition of A −1 Proof of Theorem 3. Recall that we have the following update rule w, where i k are drawn uniformly from [n], and n ∼ N (0, ν 2 I). Since F is L-smooth, we have Taking expectation with respect to i k and n given w k, we have where the second inequality uses Lemma 5 and the last inequality is due to 1 − η k L/2 > 1/2. Now taking the full expectation and summing up over T iterations, we have 2. If we choose fix step size, i.e., η k = η, and rearranging the above inequality, and using which implies that B CALCULATIONS OF β AND γ To prove Proposition 1, we need the following two lemmas. Lemma 6 (Residue Theorem). Let f (z) be a complex function defined on C, then the residue of f around the pole z = c can be computed by the formula where the order of the pole c is n. Moreover, where {c i} be the set of pole(s) of f (z) inside {z||z| < 1}. The proof of Lemma 6 can be found in any complex analysis textbook. Lemma 7. For 0 ≤ θ ≤ 2π, suppose has the discrete-time Fourier transform of series f [k]. Then, for integer k, Proof. By definition, We compute by using Residue theorem. First, note that because; therefore, it suffices to compute) for nonnegative k. Set z = e iθ. Observe that cos(θ) = 0.5(z + 1/z) and dz = izdθ. Substituting in and simplifying yields that where the integral is taken around the unit circle, and are the roots of quadratic −σz 2 + (2σ + 1)z − σ. Note that α − lies within the unit circle; whereas, α + lies outside of the unit circle. Therefore, because k is nonnegative, α − is the only singularity of the integrand in within the unit circle. A straightforward application of the Residue Theorem, i.e., Lemma 6, yields that This completes the proof. Proof of Proposition 1. First observe that we can re-write γ as It remains to show that the above summation is equal to. This follows by lemmas 7 and standard sampling in Fourier analysis (i.e. sampling θ at points {2πj/d} ). Nevertheless, we provide the details here for completeness: Observe that that the inverse discrete-time Fourier transform of is given by otherwise. The proof is completed by substituting the of lemma 7 in the above sum and simplifying. We list some typical values of γ in Table 1.. Therefore, we have We list some typical values of β in Table 2. where v 0 is the discretization of f (x), and v ∆t is the numerical solution of at time ∆t. Therefore, we have v ∆t = (I − ∆tL) which is the LS with σ = ∆t. In this section, we will show that LS can also improve the utility of the DP-CNN trained by DP-SGD and DP-Adam for CIFAR10 classification. We simply replace the CNN architecture used above for MNIST classification with the benchmark architecture in the Tensorflow tutorial 3 for CIFAR10 classification. Also, we use the same set of parameters as that used for training DP-CNN for MNIST classification except we fixed the noise multiplier to be 2.0 and clip the gradient 2 norm to 3. As shown in Figure 7, LS can significantly improve the validation accuracy of the model trained by DP-SGD and DP-Adam, and the DP guarantee for all these algorithms are the same (dashed line in Figure 7).

We propose a differentially private Laplacian smoothing stochastic gradient descent to train machine learning models with better utility and maintain differential privacy guarantees.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

GANs are shown to provide us a new effective robust mean estimate against agnostic contaminations with both statistical optimality and practical tractability.


Question: represent the article for finding a single-sentence tldr

Question: We study the robust one-bit compressed sensing problem whose goal is to design an algorithm that faithfully recovers any sparse target vector $\theta_0\in\mathbb{R}^d$ \emph{uniformly} from $m$ quantized noisy measurements. Under the assumption that the measurements are sub-Gaussian, to recover any $k$-sparse $\theta_0$ ($k\ll d$) \emph{uniformly} up to an error $\varepsilon$ with high probability, the best known computationally tractable algorithm requires\footnote{Here, an algorithm is ``computationally tractable'' if it has provable convergence guarantees. The notation $\tilde{\mathcal{O}}(\cdot)$ omits a logarithm factor of $\varepsilon^{-1}$.} $m\geq\tilde{\mathcal{O}}(k\log d/\varepsilon^4)$. In this paper, we consider a new framework for the one-bit sensing problem where the sparsity is implicitly enforced via mapping a low dimensional representation $x_0$ through a known $n$-layer ReLU generative network $G:\mathbb{R}^k\rightarrow\mathbb{R}^d$. Such a framework poses low-dimensional priors on $\theta_0$ without a known basis. We propose to recover the target $G(x_0)$ via an unconstrained empirical risk minimization (ERM) problem under a much weaker \emph{sub-exponential measurement assumption}. For such a problem, we establish a joint statistical and computational analysis. In particular, we prove that the ERM estimator in this new framework achieves an improved statistical rate of $m=\tilde{\mathcal{O}} (kn\log d /\epsilon^2)$ recovering any $G(x_0)$ uniformly up to an error $\varepsilon$. Moreover, from the lens of computation, we prove that under proper conditions on the ReLU weights, our proposed empirical risk, despite non-convexity, has no stationary point outside of small neighborhoods around the true representation $x_0$ and its negative multiple. Furthermore, we show that the global minimizer of the empirical risk stays within the neighborhood around $x_0$ rather than its negative multiple. Our analysis sheds some light on the possibility of inverting a deep generative model under partial and quantized measurements, complementing the recent success of using deep generative models for inverse problems. Quantized compressed sensing investigates how to design the sensing procedure, quantizer and reconstruction algorithm so as to recover a high dimensional vector from a limited number of quantized measurements. The problem of one-bit compressed sensing, which aims at recovering a target vector θ 0 ∈ R d from single-bit observations y i = sign(a i, θ 0), i ∈ {1, 2, · · ·, m}, m d and random sensing vectors a i ∈ R d, is particularly challenging. Previous theoretical successes on this problem (e.g. ;) mainly rely on two key assumptions: The Gaussianity of the sensing vector a i, The sparsity of the vector θ 0 on a given basis. However, the practical significance of these assumptions are rather limited in the sense that it is difficult to generate Gaussian vectors and high dimensional targets in practice are often distributed * Equal Contribution 1 Here, an algorithm is "computationally tractable" if it has provable convergence guarantees. The notatioñ O(·) omits a logarithm factor of ε −1. near a low-dimensional manifold rather than sparse on some given basis. The goal of this work is to make steps towards addressing these two limitations. Specifically, we introduce a new framework for robust dithered one-bit compressed sensing where the structure of target vector θ 0 is represented via a ReLU network G: Building upon this framework, we propose a new recovery algorithm by solving an unconstrained ERM. We show this algorithm enjoys the following favorable properties: • Statistically, when taking measurements a i to be sub-exponential random vectors, with high probability and uniformly for any is the ball of radius R > 0 centered at the origin, the solution G(x m) to the ERM recovers the true vector G(x 0) up to error ε when the number of samples m ≥ O(kn log 4 (ε −1)(log d + log(ε −1))/ε 2 ). In particular, our does not require REC type assumptions adopted in previous analysis of generative signal recovery works and at the same time weakens the known sub-Gaussian assumption adopted in previous one-bit compressed sensing works. When the number of layers n is small, this meets the minimax optimal rate (up to a logarithm factor) for sparse recovery and simultaneously improves upon the best knownÕ(k log d/ε 4) statistical rate for computationally tractable algorithms. • Computationally, we show that solving the ERM and approximate the true representation x 0 ∈ R k is tractable. More specifically, we prove with high probability, there always exists a descent direction outside two small neighborhoods around x 0 and its negative multiple with radius O(ε 1/4), uniformly for any x 0 ∈ B k 2 (R) with R = (0.5+ε) −n/2 R, when the ReLU network satisfies a weight distribution condition with parameter ε > 0 and m ≥ O(kn log 4 (ε −1)(log d + log(ε −1))/ε 2 ). Furthermore, when ε is small enough, one guarantees that the solution x m stays within the neighborhood around x 0 (rather than its negative multiple). Our is achieved without assuming the REC type conditions and under quantization errors, thereby improving upon previously known computational guarantees for ReLU generative signal recovery in linear models with small noise. From a technical perspective, our proof makes use of the special piecewise linearity property of ReLU network. The merits of such a property in the current scenario are two folds: It allows us to replaces the generic chaining type bounds commonly adopted in previous works (e.g. Dirksen and Mendelson (2018a) ) by novel arguments that are "sub-Gaussian free". From a hyperplane tessellation point of view, we show that for a given accuracy level, a binary embedding of 2 (R) into Euclidean space is "easier" in that it requires less random hyperplanes than that of a bounded k sparse set. Notations. Throughout the paper, let S d−1 and B(x, r) denotes the unit sphere and the ball of radius r centered at We say a random variable is sub-exponential if its ψ 1 -norm is bounded. A random vector x ∈ R d is sub-exponential if there exists a a constant C > 0 such that sup t∈S d−1 x, t ψ1 ≤ C. We use x ψ1 to denote the minimal C such that this bound holds. Furthermore, C, C, c, c 1, c 2, c 3, c 4, c 5 denote absolute constants, their actual values can be different per appearance. In this paper, we focus on one-bit recovery model in which one observes quantized measurements of the following form where a ∈ R d is a random measurement vector, ξ ∈ R is a random pre-quantization noise with an unknown distribution, τ is a random quantization threshold (i.e. dithering noise), and x 0 ∈ R k is the unknown representation to be recovered. We are interested the high-dimensional scenario where the dimension of the representation space k is potentially much less than the ambient dimension d. The function G: R k → R d is a fixed ReLU neural network of the form: where σ(x) = max(x, 0) and σ • (x) denotes the entry-wise application of σ(·). We consider a scenario where the number of layers n is smaller than d, Throughout the paper, we assume that G(x 0) is bounded, i.e. there exists an R ≥ 1 such that G(x 0) 2 ≤ R, and we take τ ∼ Uni[−λ, +λ], i.e. a uniform distribution bounded by a chosen parameter λ > 0. Let {(a i, y i)} m i=1 be i.i.d. copies of (a, y). Our goal is to compute an We propose to solve the following ERM forx m: where It is worth mentioning, in general, there is no guarantee that the minimizer of L(x) is unique. Nevertheless, in Section §2.2, we will show that any solutionx m to this problem must stay inside small neighborhoods around the true signal x 0 and its negative multiple with high probability. Our statistical guarantee relies on the following assumption on the measurement vector and noise: Assumption 2.1. The measurement vector a ∈ R d is mean 0, isotropic and sub-exponential. The noise ξ is also a sub-exponential random variable. Under this assumption, we have the following main statistical performance theorem: Theorem 2.1. Suppose Assumption 2.1 holds and consider any ε ∈. Set the constant C a,ξ,R = max{c 1 (R a ψ1 + ξ ψ1), 1}, λ ≥ 4C a,ξ,R · log(64C ψ,ξ,R · ε −1) and Then, with probability at least 1 − c 3 exp(−u), ∀u ≥ 0, any solutionx m to satisfies for all x 0 such that G(x 0) 2 ≤ R, where c 1, c 2, c 3 ≥ 1 are absolute constants. Remark 2.1. It is easy to verify that the sample complexity enforced by holds when where C is a large enough absolute constant. This gives the m = O(kn log 4 (ε −1)(log d + log(ε −1))/ε 2 ) statistical rate. The question whether or not the dependency on n is redundant (comparing to that of sparse recovery guarantees) warrants further studies. Remark 2.2. Note that our is a uniform recovery in the sense that the bound G(x m) − G(x 0) 2 ≤ ε holds with high probability uniformly for any target x 0 ∈ R k such that G(x 0) 2 ≤ R. This should be distinguished from known bounds (; ; ;) on sparse one-bit sensing which hold only for a fixed sparse vector. Furthermore, though assuming boundedness of G(x 0), our recovery algorithm solves for the minimizer without knowing this bound, which is favorable for practice. Before presenting the on the global landscape, we first introduce some notations used in the rest of this paper. For any fixed x, we define W +,x:= diag(W x > 0)W, in which we set the rows of W having negative product with x to be zeros. We further define W i,+,x:= diag(W i W i−1,+,x · · · W 1,+,x x > 0)W i, where only active rows of W i are kept. Thus, we can represent the RuLU network by G(x) = (Π n i=1 W i,+,x)x:= W n,+,x W n−1,+,x · · · W 1,+,x x. Definition 2.1 (Weighted Distribution Condition (WDC) ). The matrix W ∈ R d ×k satisfies the Weighted Distribution Condition with constant ε wdc if for any nonzero vectors x 1, x 2 ∈ R k, where we have θ x,z = ∠(x, z) and Mx ↔ẑ is the matrix that transformsx toẑ,ẑ tox, and ϑ to 0 for any ϑ ∈ span({x, z}) ⊥. Here we definex:= x/ x 2,ẑ:= z/ z 2. Before presenting Theorem 2.2 and 2.3, we define the directional derivative along non-zero z as where {x N} is a sequence such that x N → x and L(x) is differentiable at any x N. Such sequence must exist due to the piecewise linearity of G(x). For any x such that L(x) is differentiable, the gradient of L(x) is can be easily computed as Next, we will present Theorem 2.2 to show that under certain conditions, local minimum can only lie in small neighborhoods of two points x 0 and its negative multiple −ρ n x 0. Theorem 2.2. Suppose that G is a ReLU network with W i satisfying WDC with error ε wdc for all i = 1,..., n where n > 1. With probability 1 − c 1 exp(−u), for any nonzero x 0 satisfying x 0 2 ≤ R(1/2 + ε wdc) −n/2, if we set 88πn wdc, for any nonzero x, set v x = lim x N →x ∇L(x N) where {x N} is the sequence such that ∇L(x N) exists for all x N (and v x = ∇L(x) if L(x) is differentiable at x), then, there exists a constant ρ n ≤ 1 such that the directional derivative satisfies: where ρ n = n−1 i=0 Note that in the above theorem, case 1 indicates that the when the magnitude of the true representation x 0 2 2 is larger than the accuracy level ε wdc, the global minimum lies in small neighborhoods around x 0 and its scalar multiple −ρ n x 0, while for any point outside the neighborhoods of x 0 and −ρ n x 0, one can always find a direction with a negative directional derivative. Note that x = 0 is a local maximum due to D w L < 0 along any non-zero directions w. One the other hand, case 2 implies that when x 0 2 2 is smaller than ε wdc, the global minimum lies in the neighborhood around 0 (and thus around x 0). Moreover, in the following theorem, we further pin down the global minimum of the loss function for case 1. Theorem 2.3. Suppose that G is a ReLU network with W i satisfying WDC with error ε wdc for all i = 1,..., n where n > 1. Assume that c 1 n 3 ε 1/4 wdc ≤ 1, and x 0 is any nonzero vector satisfying x 0 2 ≤ R(1/2 + ε wdc) −n/2. Then, with probability 1 − 2c 4 exp(−u), for any x 0 such that x 0 2 2 ≥ 2 n ε wdc, setting λ ≥ 4C a,ξ,R · log(64C ψ,ξ,R · ε −1 wdc), and m ≥ c 2 a 2 ψ1 λ 2 log 2 (λm)(kn log(ed) + k log(2R) + k log m + u)/ε 2 wdc, we have L(x) < L(z), ∀x ∈ B(φ n x 0, c 3 n −5 x 0 2), and ∀z ∈ B(−ζ n x 0, c 3 n −5 x 0 2), where c 1, c 2, c 3, c 4 are absolute constants, φ n, ζ n are any scalars in [ρ n, 1]. Particularly, we have c 3 n −5 < min n≥2 ρ n such that the radius c 3 n −5 x 0 2 < ρ n x 0 2 for any n. Remark 2.3. The significance of Theorem 2.3 are two folds: first, it shows that the value of ERM is always smaller around x 0 compared to its negative multiple −ρ n x 0; second, when the accuracy level ε wdc is small, one can guarantee that the global minimum of L(x) stays around x 0. In particular, by Theorem 2.2 and 2.3, our theory implies that if ε wdc ≤ cn −76 for some constant c, then the global minimum of the proposed ERM is in B(φ n x 0, c 3 n −5 x 0 2). Since we do not focus on optimizing the order of n here, further improvement of such a dependency will be one of our future works.

Represent the short summary for finding the corresponding scientific article (long)

We provide statistical and computational analysis of one-bit compressed sensing problem with a generative prior.

We prove that for a large class of functions f there exists an interval certified robust network approximating f up to arbitrary precision.