Abstract
Abstract-Through various means of structural and synaptic plasticity enabling online learning, neural networks are constantly reconfiguring their computational functionality. Neural information content is embodied within the configurations, representations, and computations of neural networks. To explore neural information content, we have developed metrics and computational paradigms to quantify neural information content. We have observed that conventional compression methods may help overcome some of the limiting factors of standard information theoretic techniques employed in neuroscience, and allows us to approximate information in neural data. To do so we have used compressibility as a measure of complexity in order to estimate entropy to quantitatively assess information content of neural ensembles. Using Lempel-Ziv compression we are able to assess the rate of generation of new patterns across a neural ensemble's firing activity over time to approximate the information content encoded by a neural circuit. As a specific case study, we have been investigating the effect of neural mixed coding schemes due to hippocampal adult neurogenesis.
I. INTRODUCTION
Unlike artificial neural networks with explicit training and testing operating modes, the brain is continuously adapting through various means of structural and synaptic plasticity enabling online learning. Effectively, biological neural networks are constantly re-configuring their computational functionality, and are comprised of a high dimensional, distributed representation. Neural information content is embodied within the configurations, representations, and computations of neural networks. The representational capacity of neural circuits is an unknown but desirable trait to measure. A better understanding of neural representations has the potential to impact adaptive representations and encoding, machine learning, and neuromorphic hardware designs. Even though it only occurs in a few brain regions (hippocampus and olfactory bulb), of particular interest is the process of adult neurogenesis by which new neurons are added to an existing neural circuit.
I. INTRODUCTION
As follows, we present an overview of the adult neurogenesis process motivating the need for a quantitative understanding of its representational capacity and then introduce information theory as one approach to do so while discussing limitations to how it has been applied to neural circuits. We then present an extension of an approach based upon compression theory applicable to neural ensembles and use it to assess the effect of adult neurogenesis in terms of information representation.
II. BIOLOGICAL PROCESS OF ADULT NEUROGENESIS
The study of the biological process of adult neurogenesis only began relatively recently; while observations of the neurogenesis process in the adult go back to the 1950s [1] , it was only in the 1990s that extensive characterization and appreciation of the process occurred [2] [3] . Since then, the neuroscience community has rapidly characterized the neurogenesis process at all scales ranging from molecular and cellular studies to systems and behavioral levels.
II. BIOLOGICAL PROCESS OF ADULT NEUROGENESIS
The process of adult neurogenesis in the dentate gyrus (DG) region (shown in Fig. 1 ) is reviewed extensively in Aimone et al. [4] , but we briefly survey it here. Each day, roughly 1,000 new neurons are born from a stem cell population that resides locally within the DG. This rate is highly regulated by a number of intrinsic and extrinsic factors, as is the ultimate survival of the neurons that are born. While numbers differ somewhat from study to study, within several weeks about half of the neurons that are born no longer exist, most likely due to activation of apoptotic pathways (an internal gene signaling cellular death mechanism). If a neuron lives to about four to six weeks old, it most likely will persist indefinitely.
II. BIOLOGICAL PROCESS OF ADULT NEUROGENESIS
In rodents, new neurons take approximately two months to achieve maturity. During this time, they progress from a neuroblast cell phenotype, which lacks the projections commonly associated with neurons, to fully functional granule cells that are indistinguishable from those born at earlier ages. Once new input and output synapses start to form at about 14 to 16 days old, the cells mature rapidly, obtaining new synapses at a rapid pace. By about two months old, the neurons have about 5,000 to 6,000 input glutamatergic synapses from both internal and external cortical populations. A final key observation is that synapses on young neurons are more plastic, i.e., more amenable to learning, than those on mature neurons.
II. BIOLOGICAL PROCESS OF ADULT NEUROGENESIS
In addition to this difference in connectivity and synaptic plasticity, young neurons are distinct from the mature neurons in their basic electrophysiological properties. Young neurons typically have a higher membrane resistance, which allows individual synapses to have a higher relative impact than a similar weighted (same maximum conductance) synapse on a mature neuron. The combination of these properties the physiology, connectivity, and plasticity of young neurons has led to a widespread acceptance that these cells are more active or hyperexcitable compared to the mature population [4] .
II. BIOLOGICAL PROCESS OF ADULT NEUROGENESIS
This increased excitability for young neurons is notable given the existing hypotheses for DG function. Prior to the widespread acknowledgement that the DG hosted new neurons, theorists and computational neuroscientists had generally come to a consensus that the DG was responsible for two key functions (pattern separation and conjunctive encoding), both related to a key memory encoding role. Since the 1980s, the DG has been thought to be critical for driving the encoding of memories in the downstream CA3 region [5] [6] . This training function was due in part to its sparse but powerful output synapses, which were potentially capable of individually driving a downstream CA3 pyramidal neuron [7] . As a result, the DG could act like a quasi-supervised trainer to the rest of the hippocampus' memory formation. The question has remained what the DG is training. One hypothesis is that the DG is simply responsible for selecting as orthogonal a sparse representation in the downstream CA3 network as possible to minimize interference between memories [6] . At the extreme, this is effectively a hash coding hypothesis; each memory gets randomly assigned a subset of neurons in the DG, which then results in a random CA3 ensemble. In the high dimensions that are in play in the DG and hippocampus, a somewhat sparse activation could yield effectively unlimited number of combinations and a very low potential rate of interference. The other hypothesis is more information based; the DG is not producing an information-free hash code, but rather a very sparse code that is a function of the cortical inputs that drive the whole hippocampus. In this view, the information content of the DG can be extremely high, but the outcome is the same; a nearly orthogonal representation ensemble in the CA3 to encode whatever events are occurring.
II. BIOLOGICAL PROCESS OF ADULT NEUROGENESIS
The difference between these two hypotheses is subtle, and indeed it has caused a significant amount of debate in the broader hippocampus field [8] . Notably, the implications for neurogenesis are potentially quite substantial. The bulk of neurogenesis behavioral studies, which are explained in detail in Aimone et al. [4] , have interpreted their results using the pattern separation hypothesis, however it is quite possible that alternative interpretation based on information coding may yet prove informative. There is considerable debate about whether the activity of young neurons directly contributes to the DG pattern separation function [9] or whether their impact is to increase information content in episodic memories [10] . These two ideas have been discussed extensively in the literature, but there has been little attempt to develop metrics to quantify how an ensemble of neurons may provide either a pattern separation function or be increasing information content sent into the hippocampus. Most studies on the pattern separation ability of the DG have focused on looking at average correlations of the output population as compared to the correlations of an input layer [11] . This reduction in similarity is typically measured without reference to the overall information encoded in the system. For example, if the EC contains substantial information about two events that may have occurred at a park, is it sufficient for the DG to simply encode only one feature that may have differed between the two events (say one had a dog involved and another a cat)? While this is potentially optimal from a pattern separation perspective, it would potentially trivialize the ultimate memory formed. Rather, the DG likely needs to maintain a balance between information coding and separation; effectively minimizing correlation between DG outputs is likely best thought of as a constraint as opposed to a function in and of itself [10] .
II. BIOLOGICAL PROCESS OF ADULT NEUROGENESIS
This more sophisticated view of DG function requires both simultaneously assessing DG information content and correlations in a quantifiable, parametric sense. While there are a number of techniques to assess similarity (correlations, cosyne angles, Hamming distance, etc.), the ideal metric is not immediately obvious, as this is likely going to be a function of how network similarity impacts the downstream CA3 region. Further, the quantification of information content is a major challenge in systems neuroscience. Examining the nature of information representation in a model of thousands or millions of neurons requires the development of new approaches to ascertain the information content of large populations of neurons. Next we briefly introduce information theory and describe how we have used it to quantify the impact of adult neurogenesis in terms of neural information content.
III. INFORMATION THEORY
Pioneered by Claude Shannon in 1948, information theory was developed as a formal mathematical approach to measure the information in a message [12] . Precise notions about information, not to mention tools that could be used to study it, did not exist prior to this time. Shannon suggested information be measured as the base two logarithm of the inverse of the probability. Intuitively, the more surprising a message is, the more information it conveys.
III. INFORMATION THEORY
There have been many other attempts to use formal Shannon information theory on neural networks such as: Plugin Entropy, Jacknife debiased, Asympotically debiased, Ma bound, Bayesian/Dirichlet prior, Coverage-adjusted, Best upper bound, etc. For a detailed overview of these and other techniques see [13] . However, these approaches have had varying levels of success and there are limitations to such approaches. Entropy (and many other concepts from information theory) calculations require knowledge of the firing behavior probability distributions for the neurons. However, these distributions are unknown and difficult to estimate. It is also intractable to simply record firing behaviors directly due to limitations of in vivo recording capabilities of all the neurons in an ensemble during an experimental task of interest. And additionally, neural plasticity alters the response properties of neurons, effectively creating non-stationary distributions which complicates the recording challenge. And additionally, there is some determinism to neurons as their behavior is influenced by the inputs they receive (requiring conditional probabilities), as opposed to being independently stochastic. These challenges are further amplified as many of the techniques of information theory are applicable to single neurons but not large ensembles of neurons.
III. INFORMATION THEORY
Rather, we approached this problem by identifying compression methods used broadly in computing as a potential route to quantify the redundancy in the network representation over time, thus providing an estimate for independent information communicated by a network. Compression can take a number of different forms. Here, we were concerned with lossless compression, which guarantees that the inputs are fully retrievable from the outputs (i.e., no information is lost in the compression process), at the possible expense of a suboptimal compression. The alternative -lossy compression is widely used in file formats such as TIFF images and MP3 audio files works by allowing information that is not perceived by the end user to be eliminated as well, which typically results in a more compressed signal, but one which contains less information than the original source. For the purposes of our neural analysis, we are concerned with obtaining an estimate of total independent information content, which lossy compression will fail to provide. Specifically, we have investigated the use of online, dictionary based compression.
IV. MATERIAL & METHODS

A. Lempel-Ziv & Normalized Complexity Analysis
Lempel-Ziv (LZ) coding is an online, adaptive class of techniques for source coding [14] [15] [16] . It consists of adaptive dictionary compression algorithms which are universally optimal in that their asymptotic compression rate approaches the entropy rate of the source for any stationary ergodic source [17] . Rather than building an optimal coding based upon known a priori knowledge of the frequency of occurrence of the symbols being encoded (such as Huffman coding does), the LZ algorithm parses a string and builds dictionary entries based upon the shortest phrase not yet seen. Repeated substrings result in larger dictionary entries, so effectively the LZ algorithm is able to dynamically generate more efficient representations for the most prevalent sub-strings.
A. Lempel-Ziv & Normalized Complexity Analysis
Applied to the neural domain, this approach allows us to analyze the encoding of neural ensembles without knowing firing behaviour probability distributions of each neuron. Rather, we have used complexity as a measure of compressibility in order to estimate entropy to quantitatively assess the information content of a signal. Szczepanski et al. applied the general Lempel-Ziv complexity (LZ-Complexity) measure to estimate entropy of real and simulated neurons [18] . But unlike the work of Szczepanski et al., rather than applying LZ-Complexity analysis to an individual neuron's spike train, we have applied the approach to a neural population as a whole. LZ-Complexity is based upon measuring the rate of generation of new patterns along a sequence of characters in a string being compressed [14] . Applied to neuron spike trains, this technique looks for repeated spiking behavior over time. Instead, by applying it across an entire neural ensemble, we assessed repeated patterns of neural activity. Figure 2 illustrates this concept using compression as the function C(T (N )) to estimate the neural infomration content for neural ensemble N .
A. Lempel-Ziv & Normalized Complexity Analysis
We have explored a couple of approaches to analyzing the multidimensional signal comprised of an ensemble of neurons firing over time. Our first approach was to take the co-activity of all neurons in the ensemble at an instance in time and concatenate each of these temporal segments into a single spike signal. This approach is depicted in Fig. 3 . Alternatively, rather than concatenating each segment into a single spike signal we also investigated a piecewise analysis in which a single segment (whether temporal or ensemble) at a time is passed to the dynamically expanding dictionary. An illustration of this approach is shown in Fig. 4 . Using this piecewise approach, the same dictionary is repeatedly utilized and updated with each segment presented. However, dictionary entries cannot span different segments (as is the case with the concatenated single spike signal).
A. Lempel-Ziv & Normalized Complexity Analysis
Regardless of which approach is used, the spike signal is converted into a binary signal where an action potential is encoded as a one and the absence of activity by a zero. The normalized complexity may then be computed as follows:
A. Lempel-Ziv & Normalized Complexity Analysis
Normalized complexity measures the generation rate of new patterns along a word of length n with letters from an alphabet of size α (in this case two). Additionally, it can be proven [17] that as the string length (our series of neural firings in this case) goes to infinity, the supremum of the normalized complexity approaches the entropy of the signal S:
A. Lempel-Ziv & Normalized Complexity Analysis
Consequently, this provides us with a technique to approximate the information content encoded within a neural ensemble as expressed by the firing behavior the neurons exhibit.
B. Approximate Function Understanding Through Sampling
As a simplification overlooking the vast intricacies involved in their operation, neural behavior may be described as a function. Neurons fire in response to input stimuli which are able to drive the potential of the neuron beyond threshold. Ignoring the complexities of various learning mechanisms which facilitate the dynamic modification of neural responses to a given stimuli, the behavior of a neuron is a functional response. Each neuron yields a mapping from inputs to its functional output (namely whether or not to fire). As a simple, yet related, mathematical expression consider the canonical Boolean function. As the fundamental basis of digital logic and computing Boolean functions describe the behavior the function exhibits over all possible binary inputs. This is typically represented by a truth table such as that shown in the upper left of Fig. 5 . In this example, there are three inputs (A, B, and C) which allows for eight possible binary permutations. Three arbitrary Boolean functions of the inputs are defined by the columns N 1 , N 2 , and N 3 . In a small idealized scenario such as this, it is tractable to furthermore specify the minimal functional representation as well.
B. Approximate Function Understanding Through Sampling
In the neural domain this idealized analysis is not possible. In general neurons typically have 10,000 input connections. Even for a single neuron with binary synaptic connectivity, it is not tractable to consider 2 10,000 possible unique input permutations. Rather, whether in a computational neural model or physiology recordings one can only sample the neural response behavior over a small subset of the possible space. This is analogous to only sampling a subset of the full Boolean truth table such as the selection of the red boxes leading to the breakout table show in Fig. 5 .
B. Approximate Function Understanding Through Sampling
Without complete knowledge there are limits to what information may be inferred from this sampling. For example, in conjunction the three Boolean functions (N 1 , N 2 ,  N 3 ) have five unique response patterns over all possible inputs(000,010,011,100, and 101). But the arbitrary sampling shown only captures four of the five possibilities and cannot completely infer the full functionality encoded by these Boolean functions. Rather, one can only estimate the behavioral properties of the combined function. Compression analysis, as previously described, is one means of inferring the complexity of the functionality and in effect estimating the encoding of the composite function. How well the functionality may be inferred also depends upon the sampling provided. This is the exact same limitations imposed by the neural domain. Rather than having absolute knowledge of the inputs and outputs of all neurons over the full set of possible permutations, instead a typical neural recording is analogous to the breakout table in Fig. 5 .
C. Control Study Experimental Paradigm
As a control study to investigate the accuracy of our compression based information estimation method, we have implemented an experimental paradigm which allows us to vary neural input resolution to control what the information content of the ensemble should be. Our analysis technique is a general technique and not specific to any neural region, but for our experimental paradigm we are examining information encoding within the DG, with the input to the DG coming from entorhinal cortex (EC). Grid cells of the EC encode a path trajectory through space and serve as the inputs to the DG place cells. By varying the precision of the EC grid cell encoding of the input space we can define to what fidelity the neural encoding is able to distinguish the input path. This notion is captured in the upper portion of Fig. 6 . The top square of the figure illustrates an arbitrary trajectory through space. The middle three squares portray DG grid cell encodings of various fidelities. As can be seen in the leftmost square with a resolution of four (partitioning the space into fourths along each dimension), this low resolution cannot distinguish between the twists and turns of the trajectory. Rather, anytime the input path lies within the region that grid is activated. Conversely, the higher resolution partitioning of the space (such as that shown by the other two samples) encodes a higher fidelity representation of the original path allowing it to distinguish precision such as the large loop in the upper right quadrant. As a result, the increased resolution provides more information in the sense that you are more certain about the precise position in physical space. The various grid cell encodings then serve as the input to DG place cells whose firing activity are the neural samplings we are interested in analyzing for information content. In this paradigm, the DG place cells are randomly placed topologically. And additionally, each place cell has a field width parameter which dictates the extent of which inputs are able to drive it to fire. The bottom portion of Fig. 6 illustrates various configurations of place cell widths corresponding to encoding schemes. The leftmost example corresponds to a neural ensemble consisting entirely of mature, tightly tuned neurons that respond to precise inputs. Conversely, the middle example illustrates a neural ensemble consisting solely of broadly tuned neurons which respond to a broad set of inputs. The rightmost example portrays a mixed coding scheme, such as that hypothesized by neurogenesis where young neurons are broadly tuned and hyper receptive, but as they mature become tightly tuned to respond to specific inputs.
C. Control Study Experimental Paradigm
This experimental paradigm allows us to generate neural spiking outputs which we can then run our compression techniques on to approximate the information content in a controlled manner. For our assessments we have primarily analyzed ensembles of 100 to 500 randomly placed neurons over 1,000 to 10,000 timesteps. Fig. 7 depicts one path intended to loosely resemble Brownian motion across the space. Fig. 8 illustrates the resulting firing rates of these neurons over this path for resolutions of 4 and 100. With fairly narrow place cell widths, only neurons closest to the active grid location are driven to fire. Consequently, as can be seen in the top half of the figure, the neurons which happen to be located central to grid positions with the coarse binning resolution of four fire frequently. Alternatively when the resolution is much finer, such as captured in the lower half of the figure, a lot more neurons fire, but less frequently. In this sense, the neural encoding of the path is distributed across multiple neurons and has the ability to represent more information.
C. Control Study Experimental Paradigm
Keeping the neuron positions, path, and place cell widths fixed for a given analysis we have then varied the resolution and estimated the information content based upon the observed neural firings. The resolution of the EC grid cell encoding provides a theoretical upper bound as to what precision the compression analysis approximation can estimate -namely log 2 (resolution 2 ). However, not all paths fully cover the space nor do they cover each region uniformly. And so a more precise theoretical upper bound may be computed for a specific path by using the frequency of occurrence of a particular resolution region as the probability of the event that that particular region will be active. Treating the regions of space as outcomes, these frequentist inferences allow us to calculate the actual entropy of a specific path through space Applying this compression analysis of neural information content yields the approximations captured in Fig. 9 compared with the theoretical information content upper bound (shown in red). Across the x-axis are increasing resolutions from 1 to 10, and the y-axis is the information estimate (entropy).
C. Control Study Experimental Paradigm
We have also applied the approach to ensembles of mixed place cell widths to investigate the effect on information content with a mixed coding scheme. As follows are results of varying the mix ratio all with a path resolution of 25 and fixed neuron positions and path for all experiments. In these experiments we used 100 neurons and averaged the results of 25 random paths. Figure 10 depicts this computational paradigm. Figure 11 illustrates the results of an assessment of the impact of neurogenesis rates for a fixed mixed coding ratio. In the chart, the place cell width listed along a row slice is the majority cell type with the resolution at a column intersection corresponding to the mixed in minority cell type. For example, in a given row column intersection with a 10 percent mix there will be 90 neurons with place cell widths given by the row and 10 neurons with that given by the column. Increasing the place cell width size simulates increased neurogenesis as the broader place cells equate to young, more excitable neurons. The main diagonal along the plot corresponds to a pure coding at a single given resolution. See the Results Tables Appendix for a tabular view of the neural information content estimates of this mixed coding experiment as well as three other mixed coding ratios.
VI. DISCUSSION

A. Conclusions
We have presented a general approach to analyze the encoding of neural ensembles by using complexity as a measure of compressibility in order to estimate entropy to quantitatively assess the information content of a signal. As a general technique, it may be applied to a desired neural region of interest as opposed to serving as a custom analysis which is not re-usable. Key properties of the compression based approach are that it requires little a priori knowledge, and it may be applied on ensembles without having to individually compute and combine single channel information content values.
A. Conclusions
As a specific case study, we have used this technique to asses the impact of adult neurogenesis on DG information encoding. We have discovered that adult neurogenesis quantifiably increases neural information content, but saturates given too many young neurons. This is captured in Fig. 11 in which the slice plots increase information content with the inclusion of broader place width neurons (simulating young neurons) but diminishes with too many broad place width neurons. Intuitively, having some young neurons which are highly plastic and amenable to encoding novelty makes sense, and likewise having too many of these less specialized neurons is analogous to imprecision in a representation (decreasing the overall neural information content). This general trend held across the various mixed coding ratios as well, with the greater mix ratios resulting in steeper slopes to the gain as well as decline in neural information content at the extremes. These results are consistent with Aimone et. al's 2011 study [10] which argued that neurogenesis is used to improve information content; not simply to increase pattern separation as suggested by Sahay et al. [9] .
B. Next Steps
Our next steps for this research include evaluating the use of alternative compression techniques to explore any insights alternative techniques may identify. Additionally, alternative compression techniques may prove to be a better estimate of neural information content. It is possible that the intrinsic properties of different compression algorithms may prove to be better suited for certain neural ensembles if the neural circuit of interest has known properties such as temporal periodicity. Additionally, sensitivity analysis techniques may help characterize the techniques robustness to paramaterizations such as noise in the neural firing signal or ordering effects in the algorithmic approach to computing neural information content.
APPENDIX

Results Tables
As follows are the full experimental results of assessing the neural information content of the DG while varying the mixed coding ratio. For each of the experiments 100 neurons were used with a fixed EC sampling resolution providing the inputs to the DG model.
Results Tables
In each of the charts, the place cell width listed along the rows is the majority cell type with the resolution at the column intersection corresponding to the mixed in minority cell type. For example, in a given row column intersection with a five percent mix there will be 75 neurons with place cell widths given by the row and 25 neurons with that given by the column. The main diagonal of each table corresponds to a pure coding at a given resolution. 

{'paper_id': '17142263', 'title': 'Quantifying neural information content: A case study of the impact of hippocampal adult neurogenesis', 's2_url': 'https://api.semanticscholar.org/CorpusID:17142263', 'source': 'S2ORC'}