Question: Represent the scientific article to find a short TLDR

Question: To understand how object vision develops in infancy and childhood, it will be necessary to develop testable computational models. Deep neural networks (DNNs) have proven valuable as models of adult vision, but it is not yet clear if they have any value as models of development. As a first model, we measured learning in a DNN designed to mimic the architecture and representational geometry of the visual system (CORnet). We quantified the development of explicit object representations at each level of this network through training by freezing the convolutional layers and training an additional linear decoding layer. We evaluate decoding accuracy on the whole ImageNet validation set, and also for individual visual classes. CORnet, however, uses supervised training and because infants have only extremely impoverished access to labels they must instead learn in an unsupervised manner. We therefore also measured learning in a state-of-the-art unsupervised network (DeepCluster). CORnet and DeepCluster differ in both supervision and in the convolutional networks at their heart, thus to isolate the effect of supervision, we ran a control experiment in which we trained the convolutional network from DeepCluster (an AlexNet variant) in a supervised manner. We make predictions on how learning should develop across brain regions in infants. In all three networks, we also tested for a relationship in the order in which infants and machines acquire visual classes, and found only evidence for a counter-intuitive relationship. We discuss the potential reasons for this. Humans are helpless for a long time during infancy when compared with most other species. During this helpless period, the brain and mind develop as a of genetically programmed maturation in concert with experience from the environment. One way to measure development during the helpless period is by examining infants' capabilities through behaviour. However, despite great ingenuity in experimental design, due to the limited behavioural repertoire of infants, it has proven difficult to obtain more than a basic understanding of how the brain and mind are developing. Furthermore, some brain systems may undergo extended development prior to any manifestation in behaviour . Therefore, in addition to measuring behaviour a new and promising complementary tool is magnetic resonance imaging (MRI), which can measure the structure, wiring and function of the infant brain. To understand why infants' brains and minds change in the way they do, we need to create models of the mechanisms of development. This is critical if we are to understand how development is disrupted in infants with brain injury. Infants that are born very preterm, or that have potentially negative event during birth are admitted to the neonatal intensive care unit (NICU). At present, a significant proportion of NICU infants develop cognitive, behavioural or social impairments later in life. To detect these impairments earlier, and to design effective targeted interventions, we need to create models of brain and mind development in young infants. As it is difficult to intuitively imagine what the mind of a developing infant is like, it is necessary to create computational models of infant learning that can be tested against real data. A promising first brain system for which computational models of infant learning could be built is the object recognition system in the ventral visual stream. This system is relatively well understood in adults and there are proven neuroimaging methods that can be used to probe its representations and to test the hypotheses that will be generated by our computational models in due course. Indeed, in adults, there is substantial work on modelling the ventral visual stream. A key discovery from this work has been that the most straightforward strategy, of first collecting neural data and then creating models to fit it, has not been very effective . The issue is that it is only feasible to acquire a relatively small quantity of neuronal data, but the computational models capable of performing object vision have many millions of parameters. A more effective strategy, therefore, is to instead use computational models that are optimised to perform object vision to high accuracy, and then to use these as models of neural activity . Using this strategy, it has been shown that DNNs designed for the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) can predict patterns of neural activity in the ventral visual stream, as measured with functional magnetic resonance imaging (fMRI) (; Güçlü & van Gerven;), electroencephalography (EEG) and behavioural studies . It is important to note that the claim in these studies is not that there is a one-to-one mapping between artificial neurons and biological neurons. Rather, it is that some of the macroscopic aspects of the activity patterns in the DNNs and the brain are similar. For example, the ventral visual stream forms a hierarchy, with visual input from the eyes being passed forwards through a number of brain structures, and gradually being transformed from a representation of the visual input into a representation of semantic category. A quantitatively similar hierarchy of representations is found through the layers of the DNN (Güçlü & van Gerven;). Some aspects of biological object vision are innate, such as a preference for looking at faces, which is apparent in the first hour after birth . However, much must be learned through visual experience, as the genetic code alone is limited in capacity (estimated to be 150-750 MB 1) and would be substantially exhausted by the parameters of deep learning models (50-150 MB ). Even if sufficient storage were available, many objects relevant to infants today (e.g., baby bottle, or a Pokémon ) were invented too recently for innate codes to have evolved for them. A great deal of object vision knowledge, therefore, is learned. There is evidence that much of this learning is already happening in early infancy. Infants aged 3-4 months old can identify statistical regularities in sequences of pictures . And, by 6 months, infants start to look at a visual class corresponding to a concurrently spoken label 2. In adults, fMRI of the ventral visual stream has found that there are regions that are selective for particular visual classes, such as faces, body parts or places (; ;). In 4-6 month old infants, at least partial selectivity is already present in the ventral visual stream , although it continues to develop for many years . DNNs have provided a useful model of adult object vision. As these DNNs learn from visual "experience" they are therefore candidate models of infant learning. We do not expect them to capture the precise details of the learning process as, for example, the learning rule of back propagation is not biologically plausible (although biologically plausible mechanisms can approximate it ). Rather, our overarching hypothesis is that DNNs might capture some of the broad macro-scale characteristics of learning. In this work, we characterise two macro-scale properties of a number of computational models, with the aim of generating testable measurements for a future large-scale neuroimaging study. 1. Should we expect representations in brain regions of the visual hierarchy to develop simultaneously or asynchronously? An established principle of infant development is that brain regions underlying simpler functions develop first, and are followed by those underlying more complex functions (Charles A. Nelson in) 3. To provide an initial prediction of how this might happen in the visual hierarchy we examined how representations developed in different layers of DNNs during training. 2. Are visual classes that are learned earlier by infants also learned earlier by DNNs? In infants, the acquisition of visual classes can be estimated by the onset of the receptive or expressive use of the words for the classes. It has been found that, when measured this way, some visual classes are learned before others, e.g., body parts and vehicles precede food and clothing . The link between the acquisition of words and brain representations in the ventral visual stream is supported by fMRI examining the visual representations of mental imagery have shown that object classes can be decoded from brain representations when participants are tasked with reading a concrete noun . We ask whether some of this ordering is attributable to visual complexity, as reflected in the DNN classification performance. To date, studies which have shown a parallel between DNNs and the adult brain have used networks trained in a supervised manner (; Güçlü & van Gerven; ; ;). To remain consistent with these studies, we started with a supervised network. Specifically, we used CORnet-S, as this was especially designed to meet the Brain-Score benchmark by capturing the architectural principles and showing strong predictivity of neural data from the ventral visual stream, while still achieving good classification performance. It has four layers mapping onto regions in the ventral visual stream (V1, V2, V4 and IT). Infants' access to labels is extremely impoverished, making supervised learning an unlikely training curriculum for infant learning. In order to develop computational models that capture the dynamics of infant learning it is necessary to evaluate unsupervised training. One current state-of-the-art unsupervised strategy for learning visual features for object recognition is DeepCluster , which adjusts convolutional weights to create clusters of images that yield similar patterns of activation. This has a parallel with behavioural of infant learning, in that they are known to create clusters of similar images, and to be sensitive to deviants from that cluster . DeepCluster uses a simple but elegant technique for self-supervised learning, in which the start of an epoch sees each image (from ImageNet) being passed forward through a convolutional network, and the ing output activations are then clustered across all of the images using k-means. These clusters are assigned labels, which are subsequently used to learn the weights of the convolutional network with stochastic gradient descent on batches of images in the typical way. As in we used 10,000 clusters. We used AlexNet as the convolutional network, containing five layers each with a single convolutional layer. These layers were modified as in with the local response normalisation layers removed and batch normalisation used instead . Also like , a fixed linear transformation based on Sobel filters was used on the input to remove colour and increase local contrast. CORnet was trained in a supervised way, and DeepCluster in an unsupervised way. Any difference in the might be due to this difference in supervision. However, the convolutional networks at the heart of these two networks also differ, which could cause further differences. To control for this, we therefore repeated the experiments using the same AlexNet variant as in DeepCluster, but trained in a supervised way. The value for object recognition of the representations was assessed for each of the layers in the networks (4 layers for CORnet, and 5 layers for DeepCluster and Alexnet). Specifically, we quantified the explicit representation of object class using the method by of freezing weights in the convolutional layers, and training a linear decoder on the output of each layer to decode the ImageNet categories. This was done across epochs in the learning process, to capture the development of the representations in each of the layers. To summarise the learning curves of the individual classes we fitted the performance with the curve where p is top-5 precision, t the epoch, A the asymptotic level of performance, and k the learning rate. Fitting minimised least squares with the addition to the cost function of two regularisation terms equal to k 2 (to discourage implausibly high learning rates) and (A < 100) * (A − 100) 4 to discourage A values greater than 100%. We also compared the order in which visual classes are acquired in infants and machines. Unfortunately, there is no data available at present of the order in which infants acquire different visual classes. Therefore, we used a proxy, which is the estimate of the age of acquisition (AoA) of the word for the class . While a number of linguistic factors are known to affect when words are first used, including the frequency of the word in language and its number of phonemes, critically the second strongest factor is the "concreteness" of the word . Concreteness is the degree to which a word is associated with a perceptual representation, and is typically obtained by asking people to rate it. The concept has been experimentally validated in many studies, and concrete words have been shown to more readily evoke visual representations in the brain which are capable of being decoded as visual classes using fMRI and EEG . Thus, the association of AoA with concreteness suggests that the strength of the visual representation of a class may have an effect on when its label is acquired. This might happen due to one clear constraint; a child cannot name a visual class before the representation of that visual class has been developed. So, we tested if the visual classes that are labelled earlier in infants are learned more quickly by DNNs. Using the Natural Language Toolkit (NLTK), the WordNet synsets for the 1000 ImageNet classes were compared to database of 30,000 English words with AoA ratings using semantic similarity metric. The classes with the highest similarity score were considered as matching, and manually inspected for any incorrect comparisons or synset definitions. These were deleted, leaving a total of 308 classes on which further analyses were conducted. To provide a visualisation of when different types of classes were learned, we clustered the 308 classes using's metric and then clustering (scipy.cluster.hierarchy.fcluster) to yield 20 classes. By visual inspection, we then attached a label to each of the class clusters. Training was run on AWS using the Deep Learning AMI version 24.0 on either a p2.8xlarge instance (8 x NVIDIA K80 GPUs with 488 GB of RAM) or a p3.8xlarge instance type (4 NVIDIA Tesla V100 GPUs and 244 GB RAM), using Python 3.6 with Pytorch 1.1. Spot instances were used to reduce cost. The three networks were trained from scratch. The ISLVRC 2012 set was used for training and validation. The CORnet-S code was obtained from https://github.com/ dicarlolab/CORnet. DeepCluster, Alexnet and the linear classifier implementation were from https://github.com/facebookresearch/DeepCluster. 3.1.1 CORNET Explicit representation of object class in the four layers of the CORnet network during training is shown in Fig. 1a. The earlier layers in the hierarchy (V1, V2 and V4) reached their asymptotic level quickly (around epoch 1), but IT continued to learn until at least epoch 25. However, although IT took longer to reach its asymptote, even after minimal training (epoch 1) it contained greater explicit information than the lower layers. In human infants, such a learning scheme would therefore be be indicated by earlier maturation of lower-order visual processing regions (V1, V2 and V4), than higher-order brain regions (e.g., IT) in the developing brain. However, even early in development, infant IT would be expected to contain stronger explicit representations of object class as was seen for the IT layer of CORnet. It may be thought that this is inherent to how IT is built to function; it was optimised to create explicit representations of objects, therefore this signature is found throughout training. However, when examining the from later models we find that this is not the case. The colour of the curves shows AoA of the class's name for infants (blue to red for low to high AoA). Centre: The learning curves were parameterised with a fit (Eqn. 1), shown here. Right: Distributions of the fit parameters. They were correlated for all networks (CORnet r=0.62, p<0.001; DeepCluster r=0.36, p<0.01; AlexNet r=0.39, p<0.001), showing that classes that were learned more quickly were also converging on a higher asymptote. In contrast, for DeepCluster, representations mature in a more "bottom-up" manner (Fig. 1b). Specifically, the explicit representation of object class does not monotonically increase with layereven at the end of 60 training epochs, layer 3 contains stronger representations of object class than layer 4. Furthermore, the order of the layers varies through training, with layer 2 stronger than layer 3 early in training, and layer 1 containing stronger representations of object class than layer 4. Extrapolating to the brain, this more developmentally plausible unsupervised strategy would be indicated not only by higher-order regions in the ventral visual stream developing more slowly, but by earlier regions initially leading in the presence of representations of object class. This hypothesis is testable in future fMRI experiments. In supervised training, object labels are provided at the top layer of the network, and so it is perhaps not surprising that even at early epochs the entire network, including the upper layers, are maturing. This contrasts to unsupervised learning during which the only source of information is the visual input, and so it is perhaps not surprising that maturation proceeds in a more bottom up manner; until good representations have developed in the early layers, there is poorer information at higher layers. This is consistent with the simple-to-complex maturation theory (Charles A. Nelson in). CORnet and DeepCluster are not just different in their training strategies, but also in the convolutional networks at their heart. To control for this, we repeated training with AlexNet. The in Fig. 1c show that even when the same convolutional network as DeepCluster is used, but instead with a supervised training strategy, the bottom-up learning trajectories of DeepCluster are eliminated. Strikingly, explicit object representation in the lower layers of AlexNet actually reduced from epochs 3-5 onwards when assessed with top-5 or loss (Appendix B, Fig. 5c), an effect which was also seen in the supervised CORnet albeit much more weakly. In DeepCluster, this reduction in explicit representation did not appear, suggesting that it may in fact be a feature of supervised learning. In the second aim, we compared machine and human learning across visual classes. The learning curves were fit well by the model (Eqn. 1, left two columns of Fig. 2). The joint distribution (Fig. 2, right column) showed that classes which were learned quickest were ultimately learned best, as the two fit parameters were strongly correlated for all three models (CORnet r=0.62; p<0.001; DeepCluster r=0.36, p<0.01; AlexNet r=0.39, p<0.001) These fit parameters were then used to compare the machine with human learning. Paradoxically, classes learned more precisely by the model were if anything learned later by infants (Fig. 3, correlation of AoA and parameter A, CORnet r=0.11 p=0.06; DeepCluster r=0.10, p=0.09; AlexNet r=0.14, p<0.02). Although classes which were learned more precisely were in general learned more quickly in the model, there was no relationship observed between learning rate parameter (k) and infant AoA. In Aim 1, the macroscale distribution of object representations was found to be affected by learning strategy, with supervised training concentrating object representations near the output layer, while unsupervised training yielded distributed representations across layers, peaking in the penultimate layer. Learning strategy also led to layerwise differences in the order of learning, with unsupervised learning showing a bottom-up sweep in the object representation, but supervised learning an unchanging order with the top layer leading throughout. A control experiment showed that these effects were due to the learning strategy rather than the DNN architecture. In future work, it will be important to extend the investigations to further DNNs. These could test generalisation to variants on the strategies, such as local aggregation, a recently proposed unsupervised training objective, which like DeepCluster learns a visual embedding based on clustering of images . It would be informative to investigate a wider range of objectives, for example testing DNNs that exploit other structures in visual input such as temporal prediction or cross-modal learning . Future work could also investigate the benefit of making DNNs more similar to brains at the implementation level, for example by using an alternative to batch normalisation applicable to online learning 4. These macroscale differences in layerwise learning could then be tested in future infant neuroimaging, to identify which models best approximate human learning. In Aim 2, we found only a counter-intuitive relationship between the visual classes that are learned best by DNNs and those that are named first by infants. This might be because of the limitations of this measure in infants. The AoA of a label is only an approximation to when a visual class is acquired, as it is affected by other properties including the number of phonemes in the label. In future work, we will measure when infants acquire visual categories. There are other factors that influence when infants learn to identify a visual class in addition to the class's visual "accessibility". One is the frequency of the class in the environment. The ImageNet classes are esoteric and unecological, with a high preponderance of dog breeds, for example. More human-like learning will probably require more human-like (or baby-like) training sets. Another factor is the innate or learned reward value of the class to the baby (e.g., the mother's face or their milk bottle may be learned earlier). The current state-of-the-art DNNs for visual recognition use supervised training, but there is strong interest in developing unsupervised strategies, as unlabelled data is more plentiful and cheaper. Diverse unsupervised strategies have been demonstrated, such as cross-channel colour prediction and feature counting . Given the enormous space of possible unsupervised learning strategies, it is unclear how to direct investigations in a principled way. Knowledge from the human visual system such as the nature of motion and colour representations and how they develop in infancy could guide this search, and the layerwise trajectories of development could provide a constraint for biomimetic DNNs. There is recent evidence that DNNs use textures rather than shape to recognise objects, while it is the reverse in humans (a). The specific features relied upon by DNNs is thought to underlie the lack of robustness to noise, as is evident in adversarial attacks (b). This difference in feature weighting might be one cause of the different trajectories of learning for different visual classes between machine and human found in Aim 2. Assessing DNNs in how similar their acquisition order is to humans might therefore provide a valuable benchmark to guide the development of more noise-robust DNNs. DNNs were inspired by the brain. Although DNNs learn like humans from large quantities of data, there is little work to build formal connections between infant and machine learning. Such connections have the potential to bring considerable insight to both fields but the challenge is to find defining characteristics that can be measured in both systems. This paper has addressed this challenge by measuring two characteristic features in DNNs that can be measured in infants. A APPENDIX: DETERMINING NUMBER OF TRAINING EPOCHS FOR THE OBJECT DECODER Training the object decoders was the most computationally expensive part of this project, as one was trained for every layer across many epochs and models. It was therefore necessary to use as few training epochs as possible. To evaluate how many were needed, we trained decoders for 5 epochs on features from a sample of convolutional training epochs and all layers (Fig. 4). It was found that while there was a steady increase in decoding performance up to (and presumably beyond) the 5 epochs, the relative performance across different layers, or epochs, was broadly captured by epoch 2. For further analyses we therefore used 2 epochs of training for the decoding layer. Fig. 1 showed the layerwise changes in top-5 precision through learning. Fig. 5 shows the corresponding changes in cross-entropy loss. As DeepCluster learned more slowly than the supervised networks, we extended training to 70 epochs (Fig. 6). It can be seen that it was continuing to learn, particularly in the higher layers, but the order of the layers did not change within this range. Figure 6: The unsupervised network, DeepCluster, learned more slowly than the supervised networks (Fig. 5). To test if the layers with the strongest explicit object representation changed over a longer period of extended learning, we trained the convolutional layers to 70 epochs.

represent the short summary for finding the corresponding scientific article (long).
For instance, <<We learn an efficient lossy image codec that can be optimized to facilitate reliable photo manipulation detection at fractional cost in payload/quality and even at low bitrates.>>  to "Detection of photo manipulation relies on subtle statistical traces, notoriously removed by aggressive lossy compression employed online. We demonstrate that end-to-end modeling of complex photo dissemination channels allows for codec optimization with explicit provenance objectives. We design a lightweight trainable lossy image codec, that delivers competitive rate-distortion performance, on par with best hand-engineered alternatives, but has lower computational footprint on modern GPU-enabled platforms. Our show that significant improvements in manipulation detection accuracy are possible at fractional costs in bandwidth/storage. Our codec improved the accuracy from 37% to 86% even at very low bit-rates, well below the practicality of JPEG (QF 20). dard JPEG compression, and we extended it to support trainable codecs. We show a generic version of the updated model in Fig. 1 with highlighted potentially trainable elements. In this study, we fixed the camera model, and jointly optimize the FAN and a deep compression network (DCN). We describe the design of our DCN codec, and its pre-training protocol below. Our DCN model follows the general auto-encoder architecture proposed by , but uses different quantization, entropy estimation and entropy coding schemes (Section 3.2). The model is fully convolutional, and consists of 3 sub-sampling (stride-2) convolutional layers, and 3 residual blocks in between (Fig. 2). We do not use any normalization layers (such as GDN), and rely solely on a single trainable scaling factor. Distribution shaping occurs organically thanks to entropy regularization (see Fig. A.3b in the appendix). The decoder mirrors the encoder, and implements up-sampling using sub-pixel convolutions (combination of convolutional and depth-to-space layers). We experimented with different variants of latent representation quantization, eventually converging on soft-quantization with a fixed codebook of integers with a given maximal number of bits per feature (bpf). We used a 5-bpf uniform codebook (M = 32 values from -15 to 16). We show the impact of codebook size in the appendix (Fig. A.3a). The model is trained to minimize distortion between the input and reconstructed images regularized by entropy of the latent representation: where X is the input image, and E, Q, and D denote the encoder, quantization, and decoder, respectively. We used a simple L 2 loss in the RGB domain as the distortion measure d(·, ·), a differentiable soft estimate of entropy H (Section 3.2), and SSIM as the validation metric. We developed our own quantization and entropy estimation mechanism, because we found existing approaches unnecessarily complicated and/or lacking in accuracy. Some of the most recent solutions include: addition of uniform random noise to quantized samples and non-parametric entropy modeling by a fitted piece-wise linear model (Ballé et al., 2016); differentiable entropy upper bound with a uniform random noise component ; regularization by penalizing norm of quantized coefficients and differences between spatial neighbors ; PixelCNN for entropy estimation and context modeling . Our approach builds upon the soft quantization used by , but is extended to address numerical stability problems, and allow for accurate entropy estimation. Let z be a vectorized latent representation Z of N images, i.e.: z k = z n,i,j,c where n, i, j, c advance sequentially along an arbitrary memory layout (here image, width, height, channel). quantization asz = Wc. Hard quantization replaces an input value with the closest available codeword, and corresponds to a rounding operation performed by the image codec. Soft quantization is a differentiable relaxation, which uses a linear combination of all code-words -as specified by the weight matrix. A detailed comparison of both quantization modes, along with an illustration of potential numerical pitfalls, can be observed in the top row of Fig. A.1 in the appendix. The hard and soft quantization are used in the forward and backward passes, respectively. In Tensorflow, this can be implemented as z = tf.stop gradient(ẑ -z) +z. The weights for individual code-words in the mixture are computed by applying a kernel κ to the distances between the values and the code-words, which can be organized into a distance matrix D: The most commonly used implementations use a Gaussian kernel: which suffers from numerical problems for edge cases overflowing the codebook range (see Fig. A .1 top row in the 4-th and 5-th columns). To alleviate these problems, we adopt a t-Student kernel: which behaves much better in practice. We do not normalize the kernels, and ensure correct proportions of the weights by numerically normalizing rows of the weight matrix. We estimate entropy of the quantized values by summing the weight matrix along the sample dimension, which yields an estimate of the histogram w.r.t. codebook entries (comparison with an actual histogram is shown in Fig. A. 3):h This allows to estimate the entropy of the latent representation by simply: We assess the quality of the estimate both for synthetic random numbers (1,000 numbers sampled from Laplace distributions of various scales) and an actual latent representation of 128 × 128 px RGB image patches sampled from the clic test set (see Section 3.5 and examples in Fig. 4a). For the random sample, we fixed the quantization codebook to integers from −5 to 5, and performed the experiment numerically. For the real patches, we fed the images through a pre-trained DCN model (a medium-quality model with 32 feature channels; 32-C) and used the codebook embedded in the model (integers from −15 to 16). Fig. 3 shows the entropy estimation error (both absolute and relative) and scatter plots of real entropies vs. their soft estimates using the Gaussian and t-Student kernels. It can be observed that the t-Student kernel consistently outperforms the commonly used Gaussian. The impact of the kernels' hyperparameters on the relative estimation error is shown in Fig. A.2. The best combination of kernel parameters (v = 50, γ = 25) is highlighted in red and used in all subsequent experiments. Published as a conference paper at ICLR 2020 Figure 4: Example images from the considered clic, kodak and raw test sets (512×512 px). We used a state-of-the-art entropy coder based on asymmetric numeral systems and its finite state entropy (FSE) implementation . For simplicity and computational efficiency, we did not employ a context model 1 and instead encode individual feature channels (channel EC). Bitrate savings w.r.t. global entropy coding (global EC) vary based on the model, image size and content. For 512 × 512 px images, we observed average savings of ≈ 12%, but for very small patches (e.g., 128 px), it may actually in overhead (Tab. A.2). This can be easily addressed with a flag that switches between different compression modes, but we leave practical design of the format container for future work. We use a simple structure of the bit-stream, which enables variable-length, per-channel entropy coding with random channel access (Tab. A.1). Such an approach offers flexibility and scalability benefits, e.g.: it allows for rapid analysis of selected feature channels ; it enables trivial parallel processing of the channels to speed up encoding/decoding on modern multi-core platforms. We pre-trained the DCN model in isolation and minimize the entropy-regularized L 2 loss (equation 1) on mixed natural images (MNI) from 6 sources: native camera output from the RAISE and MIT-5k datasets ; photos from the Waterloo exploration database ; HDR images ; computer game footage ; city scapes ; and the training sub-set of the CLIC professional dataset . In total, we collected 32,000 square crops ranging from 512 × 512 to 1024 × 1024 px, which were subsequently down-sampled to 256 × 256 px and randomly split into training and validation subsets. We used three augmentation strategies: we trained on 128 × 128 px patches randomly sampled in each step; we flip the patches vertically and/or horizontally with probability 0.5; and we apply random gamma correction with probability 0.5. This allowed for reduction of the training set size, to ≈10k images where the performance saturates. We used batches of 50 images, and learning rate starting at 10 −4 and decaying by a factor of 0.5 every 1,000 epochs. The optimization algorithm was Adam with default settings (as of Tensorflow 1.12). We train by minimizing MSE (L 2 loss) until convergence of SSIM on a validation set with 1,000 images. We control image quality by changing the number of feature channels. We consider three configurations for low, medium, and high quality with 16, 32, and 64 channels, respectively. Standard Codecs: As hand-crafted baselines, we consider three codecs: JPEG from the libJPEG library via the imageio interface, JPEG2000 from the OpenJPEG library via the Glymur interface, and BPG from its reference implementation . Since our model uses full-resolution RGB channels as input, we also use full-resolution chrominance channels whenever possible (JPEG and BPG). To make the comparison as fair as possible, we measure effective payload of the codecs. For the JPEG codec, we manually seek byte markers and include only the Huffman tables and Huffman-coded image data. For JPEG2000, we add up lengths of tile-parts, as reported by jpylyzer. For BPG, we seek the picture data length marker. Rate-distortion Trade-off: We used 3 datasets for the final evaluation (Fig. 4): (raw) 39 images with native camera output from 4 different cameras ; (clic) 39 images from the professional validation subset of; (kodak) 24 images from the standard Kodak dataset. All test images are of size 512 × 512px, and were obtained either by cropping directly without re-sampling (raw, kodak) or by resizing a central square crop (clic). We measured PSNR and SSIM using the scikit-image package and MS-SSIM using sewar. We used the default settings and only set the data range to. The values are computed as the average over RGB channels. The bit-rate for hand-crafted codecs was computed using the effective payload, as explained above. For the DCN codec, we completely encoded and decoded the images (Section A.1). Fig. 5 shows rate-distortion curves (SSIM vs. effective bpp) for the clic and raw datasets (see appendix for additional ). We show 4 individual images (Fig. 4) and averages over the respective datasets. Since standard quality control (e.g., quality level in JPEG, or number of channels in DCN) leads to a wide variety of bpps, we fit individual images to a parametric curve f (x) = (1 + e −αx β +γ) −1 − δ and show the averaged fits. It can be observed that our DCN model delivers significantly better than JPEG and JPEG2000, and approaches BPG. Processing Time: We collected DCN processing times for various platforms (Table 1), including desktops, servers, and low-power edge AI. We report network inference and complete encoding/decoding times on 512 × 512 px and 1920 × 1080 px images, averaged over the clic dataset (separate runs with batch size 1) and obtained using an unoptimized Python 3 implementation 2. On GPUenabled platforms, the inference time becomes negligible (over 100 fps for 512 × 512 px images, and over 20 fps for 1920 × 1080 px images on GeForce 1080 with a 2.6 GHz Xeon CPU), and entropy coding starts to become the bottleneck (down to 13 and 5 fps, respectively). We emphasize that the adopted FSE codec is one of the fastest available, and significantly outperforms commonly used arithmetic coding . If needed, channel EC can be easily parallelized, and the ANS codec could be re-implemented to run on GPU as well (Weißenberger &). As a reference, we measured the processing times of 1920 × 1080 px images for the standard codecs on the i7-7700 CPU @ 3.60GHz processor. JPEG coding with 1 thread takes between 0.061 s (Q=30) and 0.075 s (Q=90) (inclusive of writing time to RAM disk; using the pillow library). JPEG 2000 with 1 thread takes 0.61 s regardless of the quality level (inclusive of writing time to RAM disk; glymur library). BPG with 4 parallel threads takes 2.4 s (Q=1), 1.25 s (Q=20) and 0.72 s (Q=30) (inclusive of PNG reading time from RAM disk; bpgenc command line tool). While not directly comparable and also not optimized, some state-of-the-art deep learned codecs require minutes to process even small images, e.g., 5-6 min for 768 × 512 px images from the Kodak dataset reported by. The fastest state-of-the-art learned codec is reported to run at ≈100 fps on images of that size on a GPU-enabled desktop computer . We consider the standard photo manipulation detection setup where an adversary uses contentpreserving post-processing, and a forensic analysis network (FAN) needs to identify the applied operation or confirm that the image is unprocessed. We use a challenging real-world setup, where the FAN can analyze images only after transmission through a lossy dissemination channel (Fig. 1). In such conditions, forensic analysis is known to fail . We consider several versions of the channel, including: standard JPEG compression, pre-trained DCN codecs, and trainable DCN codecs jointly optimized along with the FAN. We analyze 128 × 128 px patches, and don't use down-sampling to isolate the impact of the codec. We consider 6 benign post-processing operations which preserve image content, but change lowlevel traces that can reveal a forgery. Such operations are commonly used either during photo manipulation or as a masking step afterwards. We include: (a) sharpening -implemented as an unsharp mask operator applied to the luminance channel in the HSV color space; (b) resampling involving successive down-sampling and up-sampling using bilinear interpolation and scaling factors 1:2 and 2:1; (c) Gaussian filtering with a 5 × 5 filter and standard deviation 0.83; (d) JPEG compression using a differentiable dJPEG model with quality level 80; (e) AWGN noise with standard deviation 0.02; and (f) median filtering with a 3 × 3 kernel. The operations are difficult to distinguish visually from native camera output -even without lossy compression (Fig. 7). The FAN is a state-of-the-art image forensics CNN with a constrained residual layer . We used the model provided in the toolbox , and optimize for classification (native camera output + 6 post-processing classes) of RGB image patches. In total, the model has 1.3 million parameters. We jointly train the entire workflow and optimize both the FAN and DCN models. Let M c denote the c-th manipulation (including identity for native camera output), and F denote the output of the FAN with F c being the probability of the corresponding manipulation class c. Let also C denote the adopted lossy compression model, e.g., D • Q • E for the DCN. We denote sRGB images rendered by the camera ISP as X. The FAN model is trained to minimize a cross-entropy loss: and the DCN to minimize its combination with the original fidelity/entropy loss (equation 1): where λ c is used to control the balance between the objectives (we consider values from 10 −3 to 1). We start from pre-trained DCN models (Section 3.4). The FAN model is trained from scratch. When JPEG compression was used in the channel, we used the differentiable dJPEG model from the original study , but modified it to use hard-quantization in the forward pass to ensure equivalent to libJPEG. We used quality levels from 10 to 95 with step 5. We followed the same training protocol as , and trained on native camera output (NCO). We used the DNet pipeline for Nikon D90, and randomly sampled 128 × 128 px RGB patches from 120 full-resolution images. The remaining 30 images were used for validation (we sampled 4 patches per image to increase diversity). We used batches of 20 images, and trained for 2,500 epochs with learning rate starting at 10 −4 and decaying by 10% every 100 epochs. For each training configuration, we repeated the experiment 3-5 times to validate training stability. We summarize the obtained in Fig. 8 which shows the trade-off between effective bpp (rate), SSIM (distortion), and manipulation detection accuracy. The figure compares standard JPEG compression (diamond markers), pre-trained basic DCN models (connected circles with black border), and fine-tuned DCN models for various regularization strenghts λ c (loose circles with gray border). Fine-tuned models are labeled with a delta in the auxiliary metric (also encoded as marker size and color), and the text is colored in red/green to indicate deterioration or improvement. Fig. 8a shows how the manipulation detection capability changes with effective bitrate of the codec. We can make the following observations. Firstly, JPEG delivers the worst trade-off and exhibits irregular behavior. The performance gap may be attributed to: (a) better image fidelity for the DCN codec, which retains more information at any bitrate budget; (b) presence of JPEG compression as one of the manipulations. The latter factor also explains irregular drops in accuracy, which coincide with the quality level of the manipulation and unfavorable multiples of the quantization tables (see also Fig. B.1). Secondly, we observe that fine-tuning the DCN model leads to a sudden increase in payload requirements, minor improvement in quality, and gradual increase in manipulation detection accuracy (as λ c → 0). We obtain significant improvements in accuracy even for the lowest JPG JPG JPG JPG JPG (70 JPG JPG JPG JPG (quality level (37% → 85%; at such low bitrates JPEG stays below 30%). Interestingly, we don't see major differences in payload between the fine-tuned models, which suggests that qualitative differences in encoding may be expected beyond simple inclusion of more information. Fig. 8b shows the same from a different perspective, and depicts the standard rate-distortion trade-off supplemented with auxiliary accuracy information. We can observe that DCN fine-tuning moves the model to a different point (greater payload, better quality), but doesn't seem to visibly deteriorate the rate-distortion trade-off (with the exception of the smallest regularization λ c = 0.001 which consistently shows better accuracy and worse fidelity). To explain the behavior of the models, we examine frequency attenuation patterns. We compute FFT spectra of the compressed images, and divide them by the corresponding spectra of uncompressed images. We repeat this procedure for all images in our raw test set, and average them to show consistent trends. The are shown in Fig. 9 for the pre-trained DCN model (1st column) and fine-tuned models with decreasing λ c (increasing emphasis on accuracy). The plots are calibrated to show unaffected frequencies as gray, and attenuated/emphasized frequencies as dark/bright areas. The pre-trained models reveal clear and gradual attenuation of high frequencies. Once the models are plugged in to the dissemination workflow, high frequencies start to be retained, but it does not suffice to improve manipulation detection. Increasing importance of the cross-entropy loss gradually changes the attenuation patterns. Selection of frequencies becomes more irregular, and some bands are actually emphasized by the codec. The right-most column shows the most extreme configuration where the trend is clearly visible (the outlier identified in quantitative analysis in Section 4.3). The changes in codec behavior generally do not introduce visible differences in compressed images (as long as the data distribution is similar, see Section 5). We show an example image (raw test set), its compressed variants (low-quality, 16-C), and their corresponding spectra in Fig. 10. The spectra follow the identified attenuation pattern (Fig. 9). The compressed images do not reveal any obious artifacts, and the most visible change seems to be the jump in entropy, as predicted in Section 4.3. While the proposed approach can successfully facilitate pre-screening of photographs shared online, further research is needed to improve model generalization. We observed that the fine-tuning procedure tends bias the DCN/FAN models towards the secondary image dataset, in our case the native camera output (NCO). The baseline DCN was pre-trained on mixed natural images (MNI) with extensive augmentation, leading to competitive on all test sets. However, fine-tuning was performed on NCO only. Characteristic pixel correlations, e.g., due to color interpolation, bias the codec and lead to occasional artifacts in MNIs (mostly in the clic test set; see Appendix B), and deterioration of the rate-distortion trade-off. The problem is present regardless of λ c, which suggests issues with the fine-tuning protocol (data diversity) and not the forensic optimization objective. We ran additional experiments by skipping photo acquisition and fine-tuning directly on MNI from the original training set (subset of 2,500 RGB images). We observed the same behavior (see Appendix C), and the optimized codec was artifact-free on all test sets. (Although, due to a smaller training set, the model loses some of its performance; cf. MNI in Fig. A.6 .) However, the FANs generalized well only to clic and kodak images. The originally trained FANs generalized reasonably well to different NCO images (including images from other 3 cameras) but not to clic or kodak. This confirms that existing forensics models are sensitive to data distribution, and that further work will be needed to establish more universal training protocols (see detailed discussion in Appendix D). Short fine-tuning is known to help , and we leave this aspect for future work. We are also planning to explore new transfer learning protocols . Generalization should also consider other forensic tasks. We optimized for manipulation detection, which serves as a building block for more complex problems, like processing history analysis or tampering localization (; ; ; a). However, additional pre-screening may also be needed, e.g., analysis of sensor fingerprints , or identification of computer graphics or synthetic content (b). Our study shows that lossy image codecs can be explicitly optimized to retain subtle low-level traces that are useful for photo manipulation detection. Interestingly, simple inclusion of high frequencies in the signal is insufficient, and the models learns more complex frequency attenuation/amplification patterns. This allows for reliable authentication even at very low bit-rates, where standard JPEG compression is no longer practical, e.g., at bit-rates around 0.4 bpp where our DCN codec with lowquality settings improved manipulation detection accuracy from 37% to 86%. We believe the proposed approach is particularly valuable for online media platforms (e.g., Truepic, or Facebook), who need to pre-screen content upon reception, but need to aggressively optimize bandwidth/storage. The standard soft quantization with a Gaussian kernel works well for rounding to arbitrary integers, but leads to numerical issues for smaller codebooks. Values significantly exceeding codebook endpoints have zero affinity to any of the entries, and collapse to the mean (i.e., ≈ 0 in our case; Fig. A.1a). Such issues can be addressed by increasing numerical precision, sacrificing accuracy (due to larger kernel bandwidth), or adding explicit conditional statements in the code. The latter approach is inelegant and cumbersome in graph-based machine learning frameworks like Tensorflow. We used a t-Student kernel instead and increased precision of the computation to 64-bits. This doesn't solve the problem entirely, but successfully eliminated all issues that we came across in our experiments, and further improved our entropy estimation accuracy. Fig. A.2 shows entropy estimation error for Laplace-distributed random values, and different hyper-parameters of the kernels. We observed the best for a t-Student kernel with 50 degrees of freedom and bandwidth γ = 25 (marked in red). This kernel is used in all subsequent experiments. We experimented with different codebooks and entropy regularization strengths. Fig. A.3a shows how the quantized latent representation (QLR) changes with the size of the codebook. The figures also compare the actual histogram with its soft estimate (equation 6). We observed that the binary codebook is sub-optimal and significantly limits the achievable image quality, especially as the number of feature channels grows. Adding more entries steadily improved quality and the codebook with M = 32 entires (values from -15 to 16) seemed to be the point of diminishing returns. Our entropy-based regularization turned out to be very effective at shaping the QLR (Fig. A.3b) and dispensed with the need to use other normalization techniques (e.g., GDN). We used only a single scalar multiplication factor responsible for scaling the distribution. All baseline and finetuned models use λ H = 250 (last column). Fig. A.4 visually compares the QLRs of our baseline low-quality codec (16 feature channels) with weak and strong regularization. We used a rudimentary bit-stream structure with the essential meta-data and markers that allow for successful decoding (Tab. A.1). Feature channels are entropy-coded independently (we refer to this approach as channel EC), and can be accessed randomly after decoding a lookup Figure A.6: Rate-distortion performance for standard codecs and all DCN versions including the pre-trained baselines (b) and 3 fine-tuned models with the the weakest (λ c = 1) and the strongest emphasis on manipulation detection (λ c = 0.005 and 0.001). The latter incure only fractional cost in payload/quality but bring significant benefits for manipulation detection. The top (bottom) rows show for SSIM (MS-SSIM), respectively, and include DCN models fine tuned on native camera output (raw) and mixed natural images (clic). Fig. 8 visualizes the trade-offs in image compression and forensic analysis performance. Here we show direct impact of image compression and fine-tuning settings on the achievable manipulation detection accuracy and its variations (Fig. B.1). For the JPEG codec, we observe nearly perfect manipulation detection for the highest quality level (QF=95), and a steady decline starting immediately below. The sudden drop in accuracy corresponds to the quality level of JPEG as one of the manipulations (QF=80). For DCN models, we clearly see steady improvement of fine-tuning w.r.t. the baseline models (on the right). Interestingly, the high-quality model shows a slight decline at first. Qualitative Analysis: The learned frequency attenuation/amplification patterns for all of the considered quality levels are shown in Fig. B.2. The visualizations were computed in the FFT domain and show the relative magnitude of individual frequencies w.r.t. original uncompressed images (averaged over all test images). In all cases, we observe complex behavior beyond simple inclusion of high-frequency content. The pattern seem to have a stable trajectory, despite independent runs of the experiment with different regularization strengths λ c. The same patterns will also be visible in individual image spectra (Fig. B.4 -Fig. B.6). Generalization and Imaging Artifacts: While our baseline DCN models were pre-trained on a large and diverse training set, the fine-tuning procedure relied on the complete model of photo acquisition and dissemination. Photo acquisition with digital cameras yields characteristic imaging artifacts, e.g., due to color filtering and interpolation. This leads to a characteristic distribution of native camera output (NCO), and ultimately biases the codec. Fig. B.3 shows the differences in SSIM between the baseline models and the models fine-tuned with a very weak cross-entropy objective (leading to no improvement in manipulation detection accuracy). For NCO (raw test set), we observe improvement of image quality (and corresponding increase in bitrate). For the kodak set, the quality remains mostly unaffected (with an increased bitrate). On the clic set, we observe minor quality loss, and occasional artifacts (see examples in Fig. B.4). In Fig. B.4 -Fig. B.6 we collected example images from all test sets (clic, kodak, and raw) and compress them with baseline and fine-tuned models. The images are ordered by SSIM deterioration due to weak fine-tuning (quality loss without gains in accuracy; Fig. B. 3) -the worst cases are shown at the top. (Note that some of the artifacts are caused by JPEG encoding of the images embedded in the PDF, and some geometric distortions were introduced by imperfect scaling in matplotlib.) In the first clic image (1st row in Fig. B.4), we can see color artifacts along the high-contrast wires. In the second image, we see distortions in the door blinds, and a subtle shift in the hue of the bike. For the remaining two images, SSIM remains the same or better and we do not see any artifacts. In the kodak set, the worst image quality was observed for kodim05 (1st row in Fig. B .5), but we don't see any artifacts. Figure C.1: Visualization of the rate-distortion-accuracy trade-off on the clic dataset after fine-tuning on mixed natural images. As discussed in Section 5, we ran additional experiments by skipping photo acquisition and finetuning directly on mixed natural images (MNI) -a subset of the original DCN training set (2,500 images). Images in this dataset tend to have more details and depict objects at a coarser scale, since they were all down-sampled to 256 × 256 px (from various original sizes). This required adjusting manipulation strength to maintain visual similarity between photo variations. In particular, we used weaker sharpening, Gaussian filtering with a smaller standard deviation (0.5), down&up-sampling via 75% of the image size (instead of 50%), Gaussian noise with standard deviation 0.012, and JPEG quality level 90. We fine-tuned for 600 epochs. We summarize the obtained in Fig C. 1 using images from the clic test set. In this experiment, the gap in manipulation detection accuracy between JPEG and baseline DCN has disappeared, except for remaining sudden drops at selected JPEG quality levels (corresponding to the manipulation quality factor 90). We still observe significant improvement for fine-tuned DCN models, but here it tends to saturate around 86%, which might explain negligible improvement of the high-quality 64-C model. By inspecting confusion matrices, we see most of the confusion between native, sharpen and awgn classes where the differences are extremely subtle. The fine-tuned DCN models remain close to the baseline rate-distortion behavior. Interestingly, except for the weakest regularization (λ c = 0.001), all fine-tuned models seem to be equivalent (w.r.t. the trade-off). We did not observe any obvious artifacts, even in the most aggressive models. The only image with deteriorated SSIM is the alejandro-escamilla-6 image from clic, which was consistently the most affected outlier in nearly all fine tuned models for both NCO and MNI (1st row in Fig. C.2). In some replications it actually managed to improve, e.g., for the shown cases with λ c = 0.005 and 0.001. However, we don't see any major differences between these variations. Visualization of frequency attenuation patterns (Fig. C.3) shows analogous behavior, but the changes are more subtle on MNI. We included additional difference plots w.r.t. baseline and weakly finetuned models, where the changes are better visible. On NCO, due to less intrinsic high-frequency content, the behavior is still very clear (cf. bottom part of Fig. C.3). Published as a conference paper at ICLR 2020 In this study, we considered two broad classes of images: native camera output (NCO) and mixed natural images (MNI) which exhibit significant differences in pixel distribution. For DCN pretraining, we relied on a large MNI dataset with images down-sampled to 256 × 256 px (Section 3.4). Fine-tuning was then performed on either NCO from a single camera (Nikon D90; Section 4) or a smaller sub-set of the original training MNI (2,500 images; Appendix C). Finally, we considered three test sets: raw with NCO from 4 different cameras; clic and kodak with MNI. We observed that the FAN models tend to have limited generalization capabilities to images with a different pixel distribution. We ran additional experiments to quantify this phenomenon, and show the obtained in Fig. D.1 where we compare test vs validation accuracy for various test sets (we also included a version of the clic set resized to 256×256 px). In the top row, we show for models trained on NCO from a single camera. We can see imperfect, but reasonable generalization to the output of various cameras (red markers). Once the data distribution changes, the performance drops significantly. Analogously, FAN models trained on MNI generalize well to other MNI datasets (clic and kodak), but fail on NCO. We see an additional bias towards images down-sampled to the same resolution as the training data (compare clic vs clic-256 images), but here the difference is small and consistent -between 5.2 -6% based on a linear fit to the data."

Unsupervised networks learn from bottom up; machines and infants acquire visual classes in different orders

Represent the TLDR for finding the corresponding scientific article.

Our work shows positional information has been implicitly encoded in a network. This information is important for detecting position-dependent features, e.g. semantic and saliency.