---
abstract: 'This paper takes a thermodynamic approach to addressing the problem of how to represent data efficiently and meaningfully, a problem at the heart of learning and adaptation in both biological and artificial systems. Thermodynamic analysis of an information engine's cyclic operation reveals information theoretic quantities that are setting limits on performance. If run at fixed temperature, dissipation is lower bounded by a term proportional to <span>*irrelevant*</span> information. Data representation strategies that are optimal in the sense of minimizing dissipation must therefore strive to retain only <span>*relevant*</span> information. When an information engine is allowed to make use of a temperature difference, it can produce net work output, for which an upper bound is derived. Maximizing the bound yields a direct derivation of the Information Bottleneck method, a known technique in signal processing and machine learning, used precisely to filter relevant information from irrelevant clutter.'
author:
- Susanne Still
bibliography:
- 'TCBDR.bib'
title: Thermodynamic cost and benefit of data representations
---

What is the best representation of reality that an observer can come up with? At first glance, this question may seem outside the realm of physics, but observers are physical entities, and understanding how they process data may ultimately allow us to put constraints on theories by which observers describe the universe in the language of theoretical physics. This question is equivalent to asking what would be the optimal way for an adaptive system to represent environmental input data. To answer the question, we need context: optimal with respect to what? Thermodynamics offers a natural starting point, as all observers, biological and man-made alike, must expend energy to operate, and hence there must be minimal energy requirements for data representation.

Information theory, an area that has shaped our modern "information age" unlike any other, addresses how to produce efficient data representations to maintain a desired level of fidelity @Shannon48 [@Shannon59; @Berger71; @CoverThomas]. Information theory has its roots in concepts originating from physics, such as entropy, the theory's central quantity @Shannon48 [@Wiener48cybernetics; @CoverThomas]. Since the very beginning of this field its precise connections to statistical physics and thermodynamics have been discussed at some length @Shannon48 [@Wiener48cybernetics; @Jaynes57; @CoverThomas; @mezard2009information]. In particular, information processing is crucial for the operation of Maxwell's thought experiment—the "demon" that may open a trap door to defy the second law of thermodynamics @maxwell-demon. The thermodynamic cost of the demon's operation offsets any gains, thereby ensuring that the second law is not violated @brillouin1951maxwell. Details of this process have been discussed for over a hundred years @demonbook90. Recently, interest in these issues has spiked with the increasing ability to control bio-molecular machines, and the advent of nanotechnology @berut2012expLandauer [@mandal2014; @koski2014experimental; @exp-landauer2014; @parrondo2015thermodynamics; @martinez2016brownian; @gavrilov2016erasure; @gavrilov2017direct; @mandal2012work].

To answer our question, we will limit the discussion to energy efficiency in information engines. An observer is then part of an engine, utilizing information to extract work (see Fig. \[fig\]).

![Sketch of an information engine<span data-label="fig"></span>](infoengine.jpg){width="0.75\linewidth"}

Several recent papers view information as some sort of fuel that can be turned into work, e.g. @mandal2012work [@boyd2016identifying; @merhav2017relations]. Certainly, it costs energy to produce this "information fuel" @vedral17. In an information engine, minimum energy requirements limit how much work is necessary to produce information by means of a data representation @parrondo2015thermodynamics [@Still-LiL]. Some of the captured information can then be utilized to extract work. The net work done by the information engine is work extracted in excess of work needed to produce information.

Now we are in a position to sharpen up our initial question: which data representation maximizes the information engine's maximally achievable net work output? To answer this, we derive an upper bound on work output for a cyclicly run engine (equivalently, a lower bound on dissipation). Importantly, the bound contains information theoretic quantities that depends on the data representation, suggesting we find the data representation that optimizes the bound.

Information Engines
===================

The general idea for information engines goes back over a century @demonbook90. A simple and elegant setup proposed by L<span>é</span>o <span>Szilárd</span> @szilard was particularly influential, (e.g. @zurek1986maxwell [@magnasco1996szilard; @lloyd1997quantum; @kim2011quantum; @vaikuntanathan2011modeling]). It was originally conceived to illuminate the concept behind Maxwell's demon without having to discuss the detailed physiology of an observer. <span>Szilárd</span>'s setup is illustrated in Fig. \[S-fig\]. The work medium is a one-particle gas in a box of volume $V$, connected to a heat bath at temperature $T$. The work extraction mechanism consists of two elements: a partition that is inserted in the middle of the box, so that the particle's accessible volume is reduced to $V/2$, and a contraption that allows for the extraction of work as the partition moves towards the empty side of the box. Clearly this can only be done when information about the particle's location is used: one needs to know which side of the box is empty. <span>Szilárd</span> pointed out that this information has to be available until the work extraction process is completed, thus his setup requires not only measurement, but also memory @szilard. The physical mechanisms involved in measurement <span>*and*</span> memory result in what we may call a <span>*representation*</span> of observable data.

![Sketch of <span>Szilárd</span>'s Engine. The heat bath is not drawn, but ought to be imagined connected to the engine.<span data-label="S-fig"></span>](figure-szillard.jpg){width="1\linewidth"}

To make the discussion precise, let an information engine consist of the following three components . For simplicity, let them all be connected to the same heat bath, at temperature $T$, and assume the heat bath is very large, compared to the engine.

1\) A work medium that also acts as an information source; let its observable aspects be denoted by the variable $x$, with probability distribution $p(x)$. For simplicity of the exposition, assume $x$ is a discrete random variable, since it is not essential for the arguments made in this paper that $x$ be continuous.

2\) A work extraction device. The physics of this device determine which aspects of the work medium need to be known so that work can be extracted. Let us denote those by the variable $y$. In the example of <span>Szilárd</span>'s engine, $y$ is simply a coarse graining of the particle's position along the axis that is separated by the physical partition. However, the arguments made in this paper generalize naturally to the case in which $y$ may not have such a simple dependence on the observable quantity [^1]. The relationship between what we can measure, $x$, and what we need to know and change to extract work, $y$, is, in general, characterized by the joint probability $p(x,y)$.

3\) A data representation apparatus that consists of at least one measurement device, an encoding scheme, and a memory. Altogether, it enables the observer to produce and keep a stable representation, $z$, of the observed data.

One cycle of an information engine at fixed temperature, $T$, then consists of: (i) data representation, and (ii) work extraction.

We model the thermodynamics of data representation by imagining that parameters of the system that produces and stores $z$ change during the time interval in which the data representation is produced. Their trajectory during this time interval (which we may view as a "protocol"), depends on the information source's state at the start of the time interval, that is, on the "raw data", $x$. Data representation is described by the probabilistic map $p(z|x)$. Executing the protocol may require energy, a fact that cannot be ignored in the overall energy bill. Arguments have been made in the literature that the energetic costs encountered for the use of a memory are most apparent when that memory is erased @bennett2003notes. Other arguments have been put forth, explaining that whether energy loss is attributed to the making or the erasing of memory depends on the details of the specific setup, e.g. @Ouldridge [@boyd2016maxwell].

Here, we do not reset the memory, because we set up the work extraction step as follows: information captured by the observer about the work medium is used to run a work extraction protocol by changing external parameters on the work medium with the help of a given work extraction device. The protocol associated with this is a function of the available knowledge, $z$. The engine exploits all useable information, and is then brought back to its initial macro-state. This means that at the beginning of a new cycle, there are no correlations between the current state of the information source and the state of the data representation left over from the last cycle. Hence, it is not necessary to reset the memory. Sometimes, the memory will not have to be updated to produce the next data representation, because it is accidentally left in the state it ought to be in next. But other times the observer needs to make changes to the state of the memory to update the data representation. In general, running the data representation machinery, on average, costs the observer energy.

The work extraction step allows for a potential gain of useful energy, turned into work output. We expect that if run cyclicly at fixed temperature, then the engine cannot produce net work output, in accordance with the second law of thermodynamics. The efficiency of such an engine is determined by the amount of dissipated work. Now we will derive a lower bound that contains quantities reflecting the machine's information processing.

To that end, let us look at the free energy changes involved in individual steps of the engine cycle. The following analysis makes use of non-equilibrium, or generalized @Crooks07, free energy, and thus applies to systems that can be operating arbitrarily far from thermodynamic equilibrium. Recently, generalized free energy has found increasingly widespread use (e.g. @deffner2016quantum [@lostaglio2015description] and references therein). It can be motivated by noting that detailed information about a non-equilibrium system allows for the extraction of more work than the limit set by the equilibrium free energy of the system, $F_{\rm eq} = \langle E \rangle_q - kT H[q]$ (here $q$ denotes the equilibrium distribution, and Shannon entropy $H[q(x)] := - \sum_x q(x) \log{\left( q(x)\right)}$ is proportional to Gibbs entropy @Jaynes57 [@gavrilov2017direct]). Additional free energy available in a non-equilibrium system is proportional to the relative entropy between the actual non-equilibrium distribution, $p$, and the corresponding equilibrium distribution: $F_{\rm add}[p] := kT \sum_x p(x) \log{\left[ {p(x) \over q(x)} \right]}$ @shaw1984dripping [@takara2010generalization]. The overall free energy of a non-equilibrium system is thus the sum $F[p] := F_{\rm eq} + F_{\rm add}[p] = \langle E \rangle_p - kT H[p]$, where $T$ denotes the temperature of the heat bath [^2].

Free energy change involved in data representation
--------------------------------------------------

During the data representation step, the information engine's free energy, $F[p(x,z)] = \langle E(x,z) \rangle_{p(x,z)} - k T H[p(x,z)]$, changes by $\Delta F_{\rm DR} = \Delta E_{\rm DR} - kT \Delta H_{\rm DR}$. According to the first law of thermodynamics, the change in average energy, $\Delta E_{\rm DR} = Q_{\rm DR} + W_{\rm DR}$, equals the average heat flow into the engine, $Q_{\rm DR}$, plus average work, $W_{\rm DR}$, going into the engine [^3]. To compute the internal entropy change $\Delta H_{\rm DR} := H_{\rm DR}^f - H_{\rm DR}^i$, note the following: at the start of the data representation step, raw data and its representation are not dependent on each other, i.e. $p(x,z) = p(x)p(z)$, meaning that the initial joint entropy factorizes: $H_{\rm DR}^i = H[X]+H[Z]$, whereas $H_{\rm DR}^f = H[X,Z]$. The internal entropy of the engine thus decreases by the amount of <span>*information*</span> captured: $\Delta H_{\rm DR} = H[X,Z] - H[X] - H[Z] = - I[X,Z]$. Mutual information is given by $I[X,Z] = \sum_{x,z} p(x,z) \log{\left[p(x,z) \over p(x) p(z)\right]}$. Altogether, the free energy change is $$\Delta F_{\rm DR} = W_{\rm DR} + Q_{\rm DR} + kT I[X,Z]~. \label{dFDRT}$$ The second law of thermodynamics can be written as $W_{\rm DR} \geq \Delta F_{\rm DR}$. It tells us that the free energy change determines the smallest amount of work necessary to obtain the data representation, or, in other words, the <span>*least effort*</span> @Still-LiL. Typically, one designs a data representation procedure such that the change in average energy vanishes @Ouldridge [@parrondo2015thermodynamics], $\Delta E_{\rm DR} = 0$. In that case, the least effort required for data representation is proportional to the information retained about the data @Still-LiL.

The Second Law furthermore tells us that, in general, heat at least in the amount of $kT$ times the captured information must be produced: $$- Q_{\rm DR} \geq kT I[X,Z] \geq 0~, \label{Qdr}$$ in agreement with the arguments put forth by <span>Szilárd</span>, Landauer, and many others. Mutual information is non-negative. On average, data representation comes along with heat production.

Irrelevant information and lost work potential
----------------------------------------------

The observer has captured $I[X,Z]$ bits of information about the work source, but can all of this information be used to extract work? There are two ways in which information can be useless: (i) it is lost before work extraction begins, and (ii) it is information about something that cannot change during work extraction.

To use the available information, one needs to infer the value of the relevant quantity, $y$, from the existing data representation, $z$. That can be done by computing $p(y|z)= \sum_x p(y|x,z) p(x|z)$. In this paper, we assume that representing data $x$ in terms of $z$ does not change $y$, therefore $p(y|x,z) = p(y|x)$. <span>Szilárd</span>'s box is like that, because the location of the partition is fixed <span>*a priori*</span> [^4]. The inference then simplifies to computing $$\begin{aligned}
p(y|z) &=& { \sum_x p(y|x) p(z|x) p(x) \over \sum_x p(z|x) p(x)} ~,\label{inference}\end{aligned}$$ which requires knowledge of the structure of the work extraction mechanism, i.e. of $p(x,y)$. The observer's inference then contains <span>*relevant*</span> information in the amount of . The remainder, $I[X,Z] - I[Y,Z]$ is <span>*irrelevant*</span> with respect to extracting work. Importantly, relevant information is always information about the relevant quantity <span>*in the future*</span>, because time passes between data representation and the beginning of work extraction.

Only relevant information can be used to extract work. To show that, let us write $x=(\hat{x},v,y)$ in terms of $y$, and two other random variables: $\hat{x}$ describes all aspects that are not stabilized, and $v$ denotes aspects that are stabilized but cannot be changed. As an example, consider <span>Szilárd</span>'s box, and report on the three dimensional position of the particle, coarse grained to some arbitrary resolution. Only information about the particle's location along the axis divided by the physical partition, coarse grained to two outcomes, "left" or "right" (or, equivalently $y \in \{-1, 1\}$) is both stable and useful. Imagine that another partition would be inserted perpendicular to the original one, but this partition could not move. Then, the variable $v$ would have two possible outcomes, denoting on which side of the second partition the particle is found. The distribution $p(v|z)$ would change neither in the time between data representation and work extraction, nor during work extraction. At the end of the work extraction protocol, the work medium has to be restored to its original macro-state. This means the fixed partition has to be removed and reinserted. After that, $p(v,z) = p(v)p(z)$. We may summarize finer grained information about the particle's location along these two axis, together with position information along the third axis, in the variable $\hat{x}$. This information would decay in the time that passes before work extraction starts, due to the motion of the particle. How quickly it decays depends on how the system's relaxation time compares to the observer's reaction time, a detail that we do not wish to analyze here. For simplicity, we will assume that the decay happens completely in the time interval between data representation and work extraction. The distribution $p(x,z) = p(\hat{x},v,y,z)$ thus decays to $p(\hat{x}|v,y)p(v,y|z)p(z) = p(z|v,y)p(x)$ at the time when work extraction starts. The internal entropy is then $H^i_{\rm WE} = H[Z|V,Y] + H[X]$. If we compare that to $H_{\rm DR}^f$, we see that the change in internal entropy due to the decay of unstable information is $\Delta H_{\rm relax} := H^i_{\rm WE} - H^f_{\rm DR} = H[Z|V,Y] - H[Z|X] = I[X, Z] - I[\{V,Y\}, Z]$. We can rewrite the distribution at the beginning of work extraction as $p(\hat{x}|v,y)p(v|y,z)p(y,z)$, and thus the entropy as $H^i_{\rm WE} = H[\hat{X}|V,Y] + H[V|Y,Z] + H[Y,Z]$. The work extraction protocol then changes only the distribution $p(y,z)$ to $p(y)p(z)$, so that the entropy becomes $H^f_{\rm WE} = H[\hat{X}|V,Y] + H[V|Y,Z] + H[Y] + H[Z]$. Finally, at the end of work extraction, things are restored to the same macro-state, whereby all correlations are lost. This means that the entropy goes back to $H^i_{\rm DR} = H[X]+H[Z] = H[\hat{X}|V,Y] + H[V|Y] + H[Y] + H[Z]$, and the amount of information lost due to this procedure is $\Delta H_{\rm restore} := H^i_{\rm DR} - H^f_{\rm WE} = H[V|Y] - H[V|Y,Z] = I[V,\{Y,Z\}] - I[V,Y] = I[\{V,Y\},Z] - I[Y,Z]$. Altogether, irrelevant information is lost:

Free energy change involved in work extraction
----------------------------------------------

The work extraction protocol effects changes in the work medium, specifically changes in the distribution of $y$. The resulting entropic change is $\Delta H_{\rm WE} := H^f_{\rm WE} - H^i_{\rm WE} = H[Y] + H[Z] - H[Y,Z] = I[Y,Z]$, and the associated change in average energy is the sum of average heat flow and work: $\Delta E_{\rm WE} = Q_{\rm WE} + W_{\rm WE}$. The total free energy change involved in work extraction is hence $$\Delta F_{\rm WE} = W_{\rm WE} +  Q_{\rm WE} - k_B T I[Y,Z]~.$$ The Second Law tells us that relevant information sets an upper bound on how much heat can be absorbed: $$Q_{\rm WE} \leq kT I[Y,Z] \label{Qwe}$$ Recall that, by convention, work <span>*extracted*</span> from the work medium has a negative sign. The Second Law then reminds us that the maximum amount of work which can be extracted is given by the reduction in free energy: $-W_{\rm WE} \leq - \Delta F_{\rm WE}$. If things are arranged such that the average energy change during this step is zero, $\Delta E_{\rm WE} =0$, then this <span>*work potential*</span>, $- \Delta F_{\rm WE}$, is proportional to relevant information retained in the data representation @Still-LiL, and we have $-W_{\rm WE} \leq k_B T I[Y,Z]$.

Dissipation at fixed temperature
--------------------------------

The total average work performed on the engine in a cycle, consists of work necessary to produce the observer's data representation, $W_{\rm in} := W_{\rm DR}$, minus work extracted from the work medium, using the observer's knowledge, $W_{\rm out} := -W_{\rm WE}$. Over a cycle, changes in internal entropy cancel out. The total free energy change therefore equals the total average energy change, which must be zero, since we are running a cycle. Hence, net work flowing into the engine has to equal net heat flowing out: $W_{\rm in} - W_{\rm out} = - Q_{\rm DR} - Q_{\rm WE}$. The Second Law, applied to the entire cycle, tells us that the process must be dissipative, $W_{\rm in} - W_{\rm out} = - Q_{\rm DR} - Q_{\rm WE} \geq 0$, i.e. our knowledge of the work medium does not allow us to extract more work than what it costs to produce the data representation that physically embodies this knowledge.

Now we use inequalities (\[Qdr\]) and (\[Qwe\]), to obtain a tighter lower bound on dissipation [^5]: $$W_{\rm in} - W_{\rm out} \geq kT (I[X,Z] - I[Y,Z]) \geq 0~.$$ The amount of irrelevant information captured by the data representation limits minimally achievable dissipation. Therefore, a data representation that is <span>*optimal*</span> in the sense of allowing for minimal dissipation has to <span>*discard irrelevant information*</span>. The ideal observer should capture only relevant information, so that $I[X,Z] = I[Y,Z]$.

But overall, the machine can do no net work, because it is run at fixed temperature. At best, it can extract as much work as it needs to perform the data representation [^6]. Minimizing dissipation at fixed temperature, in the absence of other constraints, does not force the observer to do anything: there are multiple optimal solutions, one of which is keeping no information at all and extracting no work, i.e. $I[X,Z] = I[Y,Z] = 0$. However, if other constraints are present, then minimizing dissipation in an information engine that runs at a fixed temperature may make sense.

Example: Szilard Engine with redundancy
---------------------------------------

Imagine reporting on which <span>*third*</span> of the relevant axis the particle is found in (see Fig \[Sm-fig\]). This particular data representation would dissipate, on average, heat at least in the amount of $kT \ln(3)$. How much work could be extracted using the knowledge it produces? Two thirds of the time, the particle would be either on the left or the right, and work could be extracted in the amount of at most $kT \ln(2)$, on average. But one third of the time, the particle would be reported to be in the middle. This contains no relevant information about which side of the partition is empty. Thus, the overall maximum average work that could be extracted would be ${2\over 3} kT \ln(2)$, and dissipation would be limited by $kT\left( \ln(3) - {2\over 3} \ln(2)\right) > 0$.

![Modified <span>Szilárd</span> Engine: Report on the particle's location to resolution $n=3$. The heat bath is not drawn, but ought to be imagined.<span data-label="Sm-fig"></span>](figure-3m2e-szillard.jpg){width="1\linewidth"}

The argument generalizes to reporting the particle's location to an n-th of the length of the relevant axis, encountering a lower bound on dissipation of $kT \left(\ln\left({n \over 2}\right) + {1\over n}\ln(2)\right)$ for odd $n>2$. For even $n>2$, energy gets wasted by a redundant data representation, but the work potential is still maximal, namely $kT \ln(2)$, and we encounter dissipation in the amount of at least $kT \ln\left({n \over 2}\right)$. If the constraints allow the observer only to choose $n \geq 2$, then the choice $n=2$ is optimal in terms of minimizing the smallest achievable dissipation. But if $n=1$ becomes an option, then we have two solutions with a minimally achievable dissipation of zero, with the difference that the $n=2$ solution captures more relevant information than the $n=1$ solution, $\ln(2) > 0$, and thus allows us to do something (extract work).

Two temperatures, $T<T'$.
-------------------------

To produce net work output, we may run the data representation at a lower temperature, $T$, than the temperature at which we extract work, $T'$. This can be implemented by changing the temperature of the heat bath. We then need to include two new steps into the engine's cycle, cooling and heating. Table \[table2\] summarizes the situation.

= \[draw=black,inner sep=6pt\]

(box)

<span>0.4</span>

$$\begin{aligned}
&\Delta F_{\rm C} = Q_{\rm C} - kT \left(H[X] + H[Z] \right)\notag \\
& \;\;\;\;\;\;\;\;\;\;\;\;\;\; + \;kT' \left(H'[X]  + H'[Z]\right) \label{deltaFC} \\
&\Delta F_{\rm DR} = W_{\rm DR} + Q_{\rm DR} + kT I[X,Z] \label{deltaFDC}\\
&\Delta F_{\rm relax} = - kT \left( I[X,Z] - I[Y,Z] \right) \label{dFrelax}\\
&\Delta F_{\rm H} = Q_{\rm H} - kT' \left( H'[Z|Y] + H'[X] \right)  \notag \\
& \;\;\;\;\;\;\;\;\;\;\;\;\;\;+ \;kT \left( H[Z|Y]  + H[X] \right) \label{deltaFH}\\
&\Delta F_{\rm WE} = W_{\rm WE} + Q_{\rm WE} - kT' I[Y,Z] \label{deltaFWE} \end{aligned}$$

;

When cooling from $T'$ to $T$ before the measurement, heat flows in the amount of $Q_C$. As kinetic energy decreases, so does the internal entropy. To keep track of these changes, we use the notation $H'$ for the entropy at the higher temperature. We assume that no energy is added in the form of work. The free energy change is then given by Eq. (\[deltaFC\]). The free energy associated with data representation is the same as before, compare Eqs. (\[dFDRT\]) and (\[deltaFDC\]), therefore Eq. (\[Qdr\]) holds. Also, in complete analogy to before, we lose work potential proportional to irrelevant information, Eq. (\[dFrelax\]). (We have seen in the previous discussion that nothing essential is added to the treatment by stabilizing aspects of the system that cannot change during work extraction, and for simplicity we now leave this out.) Heating from $T$ to $T'$ increases the internal entropy of the system, while heat flows in the amount of $Q_H$. The free energy change is given by Eq. (\[deltaFH\]). Work gets extracted at the higher temperature, see Eq. (\[deltaFWE\]).

Increasing the temperature can only decrease information, $I'[Y,Z] := H'[Y|Z] - H'[Y] \leq I[Y,Z]$. Clearly, no memory is infinitely stable, and we can burn up the entire machine if we raise the temperature enough: $\lim_{T' \rightarrow \infty} I'[Z,Y] = 0$. In order to operate the information engine to its intended use, we shall limit temperature differences to a "memory preserving" temperature range, meaning that the memory stays stable and we lose no relevant information: $I'[Y,Z] = I[Y,Z]$. In this range, the maximum amount of heat that can be absorbed is, as before, limited by relevant information: $$Q_{\rm WE} \leq kT' I[Y,Z]~. \label{QweT'}$$

We assume that no work is performed during heating and cooling. The Second Law then tells us that the free energy cannot increase during these steps: $ - \Delta F_{\rm H} \geq 0$, and $- \Delta F_{\rm C} \geq 0$. In the memory preserving regime, the combined effects of the heating and cooling, $Q_{\rm HC} \equiv Q_{\rm C} + Q_{\rm H}$, result in an average dissipation of heat (adding Eqs. (\[deltaFC\]) and (\[deltaFH\])): $$\begin{aligned}
\!\!\!\!\!\!\!\! -Q_{\rm HC} &=& -\Delta F_{\rm H} - \Delta F_{\rm C} + k(T'-T) I[Y,Z] \geq 0. \label{QHC}\end{aligned}$$ In the memory preserving temperature regime, internal entropy changes cancel over a cycle. The total free energy change over a cycle is equal to the total average energy change, which has to be zero. The total work done by the engine is thus $W_{\rm out} - W_{\rm in} = Q_{\rm DR} + Q_{\rm WE} + Q_{\rm HC} \leq Q_{\rm DR} + Q_{\rm WE}$, which is upper bound by (using Eqs. (\[Qdr\]) and (\[QweT'\])): $$\begin{aligned}
W_{\rm out} - W_{\rm in} 
&\leq& kT' I[Y,Z] - kT I[X,Z] ~. \label{DissIB}\end{aligned}$$

Information Bottleneck
======================

Eq. (\[DissIB\]) suggests the following optimization: choose that data representation $p^*(z|x)$ which maximizes the achievable work by maximizing the upper bound. That is, solve the following optimization problem: $$\begin{aligned}
&&\max_{p(z|x)} \left( I[Y,Z] - \lambda I[X,Z] \right) \notag \\
&& {\rm subject \; to:} \;\;  \sum_z p(z|x), \; \forall x. \end{aligned}$$ Here $\lambda := T/T' < 1$, and the added constraints simply ensure normalization of the probability distributions $p(z|x)$. This optimization problem is known in data compression and machine learning under the name "Information Bottleneck" (IB) @IBN. From the extremal conditions of the Lagrangian, we know that all optimal solutions have to fulfill @IBN: $$\begin{aligned}
p^*(z|x) &=& {p(z) e^{-{1\over \lambda} {\cal D}[p(y|x) \| p(y|z)] } \over \sum_z p(z) e^{-{1\over \lambda} {\cal D}[p(y|x) \| p(y|z)] }}~. \label{IBsol}
$$ This equation can be iterated numerically together with Eq. (\[inference\]) to find optimal codes @IBN.

The temperature ratio $\lambda = T/T'$, controls the trade-off between the importance of getting energy out during work extraction vs. having to spend energy on the data representation. Equivalently, $\lambda$ controls the trade-off between keeping relevant information and having a concise data representation that does not cost more bits than it has to @IBN. For simplicity, fix the temperature $T'$ at which work is extracted. Then the temperature at which the data representation is produced, $T$, becomes the defining parameter. We find the best solution at low temperature, $T \rightarrow 0$, where assignments of data to representatives become deterministic @Still-IAL09: $p^*(z|x)_{T\rightarrow 0} = \delta_{z z^*(x)}$, with $z^*(x) := {\rm arg}\min_z {\cal D}[p(y|x) \| p(y|z)]$, where $\delta_{ij}$ denotes the Kronecker-$\delta$.

The Information Bottleneck is a fairly general method for optimal data representation, containing within it other methods as special cases (see e.g. @Still-IBPI-2014, and references therein). When used for model discovery from <span>*finite*</span> samples (the case in which $p(x,y)$ is not known, but estimated), attention has to be paid to correcting for errors made due to under-sampling, in order to avoid over-fitting @StillBialek04.

Discussion
==========

The physical approach to signal processing and machine learning followed in this paper is not limited to what we have learned here. It can be used to explore a range of other existing methods from physical arguments, and to design new ones. For example, one could argue that work output is not the only thermodynamically meaningful quantity to maximize. Engine power and efficiency might matter, and optimization of these quantities would not necessarily coincide @proesmans2016brownian. Time constraints certainly play an important role, particularly in the context of biology.

While the optimization here was limited to finding the best data representation, one could, of course, optimize over work extraction devices instead, i.e. over $p(y|x)$. Interestingly, this can be used to derive Linsker's "Infomax principle" @linsker1988self. To that end, note that no data representation can capture more relevant information than what is contained in the data, i.e. $I[Y,Z] \leq I[X,Y]$. Combining this with Eq. (\[DissIB\]), we see that work output can ultimately be no larger than $$W_{\rm out} - W_{\rm in} \leq kT' I[X,Y] - kT I[X,Z] ~.$$ Now, since $I[X,Z]$ does not depend on $p(y|x)$, maximizing this upper bound over $p(y|x)$ alone is then equivalent to finding a (probabilistic) map from $x$ to $y$ that maximizes information: $\max_{p(y|x)} I[X,Y]$. This is exactly the spirit of Infomax, which has been used, for example, to motivate Independent Component Analysis (ICA) @bell1995information. Infomax can thus be interpreted to find the best physical tool to exploit given information. It has not necessarily been used in that context, but this new interpretation may turn out to be a fruitful way of thinking.

The approach could also be used on other known machine learning methods. For example, the "Boltzmann machine" adjusts its weights to minimize the relative entropy between input distribution and model distribution (the neural network's equilibrium distribution) @BM. The neural network stands alone, in the sense that it is not specified how the information it captures can be turned into energy gain. The best the neural net can do is thus to create a stable data representation that captures as much free energy as possible. The remaining free energy is turned to heat during the network's relaxation to equilibrium. The Boltzmann machine minimizes this heat dissipation. Once context is given to the neural network, it may become advantageous to discard information that cannot be exploited.

Summary
=======

It was demonstrated that, in the context of information engines, simple physical arguments are enough to derive an optimal data representation strategy: demanding efficient use of energy means data representations ought to capture <span>*relevant*</span> information. Importantly, the observer needs to infer and predict what the value of the relevant quantity will be <span>*in the future*</span>. The analysis presented here thus generalizes previous findings @StillSBC12 [@grimsmo2013quantum].

Overall, the approach followed here reflects a distinctly <span>*physical*</span> way of thinking about learning systems. Physical limits to information processing let us derive learning strategies from tangible demands. This not only gives us a clear intuition about what the resulting algorithms do, and hence how and why they work, but may also lead to new and better learning methods, and may ultimately become useful in the context of understanding the principles governing biological systems @Bialek12.

Acknowledgements
================

I am greatly indebted to the patient support of Rob Shaw and Elan Stopnitzky, who read and discussed many versions of this manuscript. I have presented this work at various international conferences since 2014, which has naturally lead to countless meaningful and helpful discussions, too many to list. Discussions with Tobi Berger, Bill Bialek, Gavin Crooks, Arne Grimsmo, Massimiliano Esposito, Chris Jarzynski, Matteo Marsili, Marcelo Magnasco, Seth Lloyd, Tom Ouldridge and David Sivak have had particular impact on my thoughts.

[^1]: It is even possible that we cannot directly measure the work source, but instead can only measure an information source that is correlated to it. The analysis performed here applies to this scenario. In general, $x$ refers to observable data, $y$ refers to the quantity that needs to be known to extract work.

[^2]: This equality follows simply from inserting the Boltzmann distribution, $q=e^{-\beta (E - F_{\rm eq})}$, into $F_{\rm add}$, whereby $F_{\rm add}[p] = - kT H[p] + \Sigma_x p(x) E(x) - F_{\rm eq}$

[^3]: We use the convention that energy flows <span>*into*</span> the engine whenever $W$ (or $Q$) are positive, and out of the engine whenever $W$ (or $Q$) are negative.

[^4]: If the observer had control over where to insert the partition, then potentially more information could be used. How much exactly would, however, depend on the speed with which the observer acts. Fixing the partition's location <span>*a priori*</span> allows us to ignore such details. But the control the observer has over the work medium is rather limited. While this level of control is sometimes called "feedback" @SagawaUedaFeedback12, true feedback should include the ability to change the relevant feature ($y$) @Still-IAL09. This more general case will be discussed in a second paper.

[^5]: This bound is in agreement with e.g. @lloyd1997quantum

[^6]: Note that a machine can be constructed such that the observer writes the data representation, $z$, onto a tape and the work extraction is performed by an "information ratchet", such as in @mandal2012work [@boyd2016identifying]. Then, producing a data representation is equivalent to producing "information fuel" for the ratchet, and the thermodynamic cost for information production is governed by the amount of information prepared, Eq. (\[Qdr\]). For example, <span>Szilárd</span>'s box could be prepared such that the particle is on the left side of the piston at the beginning of each cycle. The engine could then run the same work extraction mechanism every cycle, without the need to make a decision based on input data, and hence could work without memory. But the compression of the gas, which is required to prepare the box, would cost energy in the amount of at least $kT \ln(2)$. This preparation step should not be taken out of the overall accounting to make it look as if the engine produces net work output at fixed temperature.

{'source': 'arXiv_out/1705'}