Interactive Information Theory

Definition. The self-information of an event occurring with probability \(p\) is \(I(x) = -\log p(x)\). With base 2 the unit is the bit; with the natural log it is the nat. In machine learning the convention is the natural log because it differentiates cleanly, so every loss function in this article is in nats.

Intuition: the size of a surprise. "The sun will rise tomorrow" tells us nothing — its probability is 1 and its self-information is 0. "It will snow tomorrow" is mildly surprising, and "a meteorite will strike this city tomorrow" is enormously so. The rarer the event, the more the fact that it happened tells us. Self-information puts a number on how much it tells us.

Why a logarithm? It is not an arbitrary choice of function. There are three properties we demand of a quantity of information. ① If the probability is 1 the information is 0. ② The lower the probability, the greater the information (monotonically decreasing). ③ If two independent events both occur, their information adds — the information from flipping a coin twice should be twice that of flipping it once. Independence means probabilities multiply (\(p(x,y)=p(x)p(y)\)), so we need a function turning products into sums. Add natural conditions such as continuity and the possible functions are pinned down to a positive constant multiple of a logarithm, with the base of the log fixing that constant and the unit of information.

Connection to the probability article. The reason we introduced the log-likelihood in the previous article was practical — to avoid underflow and to decompose the derivative. Now we are giving that same \(-\log p\) a meaning. It is not merely computational convenience but a quantity of surprise, and as the next section shows, also the number of bits needed to encode something. To minimize a loss is to make the model less surprised by the data.

Divergence. As \(p\to 0\), \(I\to\infty\). If a model assigns probability 0 to an event that actually occurs, the loss becomes infinite. In exact real arithmetic a softmax probability is never precisely 0, but in actual floating-point computation a very small value can underflow to 0. So implementations use numerically stable operations such as log_softmax, which applies log-sum-exp to the logits, rather than computing probabilities first and taking a log afterwards.

\[\begin{aligned} I(x) &= -\log p(x) \\ I(x,y) &= I(x) + I(y) \quad \text{(when independent)} \\ 1\ \text{nat} &= \log_2 e \approx 1.443\ \text{bit} \end{aligned}\]

Move the p slider from 1 down toward 0. The curve starts at 0 when p=1 and shoots up sharply as p→0. The unit toggle compares bits with nats and shows that only the base of the log changes, not the shape.

Definition. The entropy of a distribution \(p\) is the expectation of self-information: \(H(p) = \mathbb{E}_{x\sim p}[-\log p(x)] = -\sum_x p(x)\log p(x)\). It measures "on average, how surprised will I be by one draw from this distribution," which is precisely that distribution's uncertainty.

When is it largest and when smallest? If one outcome has probability 1 (complete certainty) then \(H=0\) — there is nothing to be surprised by. Conversely, if \(K\) outcomes all share probability \(1/K\), the entropy attains its maximum \(\log K\). The uniform distribution is the hardest to predict. Dragging the bars in the demo moves between these two extremes.

Binary entropy. For \(K=2\), \(H(p)=-p\log p-(1-p)\log(1-p)\) has its maximum \(\log 2\) at \(p=0.5\) and falls to 0 at both ends, a symmetric bell shape. The shape of this one curve summarizes the entire notion of uncertainty — it is drawn below.

Connection to the probability article: why the Gaussian? The second reason we gave for choosing a Gaussian was "maximum entropy." Now we can state that precisely — under the constraint that the mean and variance are given, the continuous distribution maximizing entropy is exactly the Gaussian. Likewise, with no constraints at all the answer is the uniform distribution, and for a positive random variable with only the mean given it is the exponential distribution. The maximum entropy principle is "reflect only what you know and do not invent what you do not," written as a formula.

In machine learning. Entropy appears in objective functions in its own right. Entropy regularization in reinforcement learning (SAC, PPO) adds the policy's entropy to the reward to prevent premature convergence and insufficient exploration, and decision trees use the reduction in entropy (information gain) as a splitting criterion. There are also techniques pushing entropy the other way — entropy minimization in semi-supervised learning encourages the model to make confident predictions.

\[\begin{aligned} H(p) &= -\sum_x p(x)\log p(x) = \mathbb{E}_{p}[-\log p(x)] \\ 0 \le H(p) &\le \log K \\ H(p) &= \log K \iff p \text{ is uniform} \end{aligned}\]

Drag the bars up and down to change the distribution. Pile the probability onto one bar and H goes to 0; spread the five bars evenly and H reaches its maximum log 5 ≈ 1.609. The "Uniform" button jumps straight to the maximum, and the binary entropy curve below marks where the current state sits.

Definition (Shannon's source coding theorem). When symbols drawn from a distribution \(p\) are encoded losslessly, the average code length per symbol cannot be shorter than \(H(p)\) bits. And that bound can be approached arbitrarily closely. An optimal code assigns \(-\log_2 p(x)\) bits to a symbol of probability \(p(x)\).

Why that length? Short codes are scarce — there are only \(2^\ell\) codes of length \(\ell\) bits. So short codes must be spent sparingly, on the symbols that occur often. Assigning a symbol a length \(\ell(x)\) is in effect assuming a probability \(2^{-\ell(x)}\) for it. Since \(-\log_2p(x)\) is usually not an integer it does not exactly equal the real binary code length of a single symbol, and should be understood as an ideal information length. Using block coding over several symbols, or arithmetic coding, brings the average length per symbol arbitrarily close to this value.

This is the bridge between information theory and machine learning. For a model \(q\) to assign probability to data is for it to assign code lengths. Therefore minimizing the negative log-likelihood is exactly the same problem as finding the code that compresses the data most tightly. "A good model" and "a good compressor" are the same phrase — which is where the practice of reporting language model performance as a compression rate comes from.

Why it need not be an integer. \(-\log_2 p\) is usually not an integer, but encoding several symbols together rather than one at a time (block coding) makes the average length per symbol arbitrarily close to the entropy. Arithmetic coding does exactly this in practice. So the entropy is an attainable bound, not a loose one.

\[\begin{aligned} \ell^*(x) &= -\log_2 p(x) \quad \text{(ideal information length)} \\ \mathbb{E}_p[\ell^*(x)] &= -\sum_x p(x)\log_2 p(x) = H(p) \\ \textstyle\sum_x 2^{-\ell(x)} &\le 1 \quad \text{(Kraft's inequality)} \end{aligned}\]

Drag the bars to change the symbol probabilities. The ideal information length −log₂p is shown in bits beside each symbol, and their average is computed below. That ideal average coincides exactly with the entropy, and a real code can approach it through block coding. The next section looks at what happens when this code is built from the wrong distribution.

Definition. The cross-entropy of \(q\) relative to \(p\) is \(H(p,q) = -\sum_x p(x)\log q(x) = \mathbb{E}_{x\sim p}[-\log q(x)]\). Exactly one thing differs from the entropy above — the expectation is taken with \(p\) while \(q\) sits inside the log. That mismatch is the origin of the name "cross."

Read it as coding. \(-\log_2 q(x)\) is the length of a code built in the belief that \(q\) is right. But the symbols actually come from \(p\). The average number of bits spent encoding real data with that code is the cross-entropy. It is the price of using a codebook built from a mistaken belief.

When is it minimal? \(H(p,q)\ge H(p)\), with equality only when \(q=p\). That is, the cross-entropy is minimized when my model matches the true distribution exactly, and that minimum value is the entropy. In the demo, matching \(q\) to \(p\) drives the value down to \(H(p)\) but never below it. The uncertainty inherent in the data cannot be removed by any model — this is the fundamental reason a loss never reaches 0.

It is the same thing as NLL. In training, \(p\) is in practice the empirical distribution of the dataset, which places uniform mass on the observed examples. Then \(H(p,q)=-\frac{1}{N}\sum_i \log q(x^{(i)})\), which is literally the same expression as the average NLL derived in the previous article. Calling the classification loss cross-entropy and calling it negative log-likelihood are two languages naming one object.

How it differs from KL. The cross-entropy \(H(p,q)\) is the total cost of expressing real data with the model \(q\)'s code, while the KL \(D_{\mathrm{KL}}(p\|q)\) is the additional cost after subtracting the unavoidable uncertainty \(H(p)\) of the true distribution itself. Hence \(H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q)\). If the data distribution \(p\) is fixed during training then \(H(p)\) is a constant, so minimizing cross-entropy and minimizing KL find the same \(q\). The values themselves are generally not equal, however, and coincide only in the special case \(H(p)=0\), such as a one-hot target.

\[\begin{aligned} H(p,q) &= -\sum_x p(x)\log q(x) \\ H(p,q) &\ge H(p), \qquad \text{equality} \iff q = p \\ H(\hat{p}_{\text{data}}, q) &= -\frac{1}{N}\sum_{i=1}^{N}\log q(x^{(i)}) = \text{average NLL} \end{aligned}\]

Drag the q bars to match the grey true distribution p. The instant they align exactly, H(p,q) equals H(p), and that is the attainable minimum. Pull them out of alignment and the value rises, but it never drops below H(p).

Definition. \(D_{\mathrm{KL}}(p\,\|\,q) = \sum_x p(x)\log\frac{p(x)}{q(x)} = H(p,q) - H(p)\). It is the excess over the entropy in the cross-entropy, and reading it as "the extra bits per symbol paid for having used a code built from \(q\)" is exactly right.

It is always at least 0. \(D_{\mathrm{KL}}(p\|q)\ge 0\), with equality only when \(p=q\) (Gibbs' inequality). Intuitively it says that using some other code \(q\) instead of the optimal code \(p\) cannot make the average length shorter. Mathematically it is proved from the concavity of the log and Jensen's inequality. This non-negativity is the backbone of the ELBO later on and of many information-theoretic inequalities.

It is not a distance. KL fails to be a distance (a metric) in the mathematical sense for two reasons. First, it is not symmetric: \(D_{\mathrm{KL}}(p\|q)\ne D_{\mathrm{KL}}(q\|p)\). Second, it does not satisfy the triangle inequality. So it is called a divergence rather than a distance. The asymmetry is not a defect but a meaningful property, and an entire section further on is devoted to it.

What training actually minimizes. \(H(p) = H(\hat{p}_{\text{data}})\) is a constant once the data is fixed and does not involve the parameters. Therefore minimizing the cross-entropy and minimizing \(D_{\mathrm{KL}}(\hat{p}_{\text{data}}\|q_\theta)\) are exactly the same optimization problem. Reducing the loss is pulling the model distribution toward the data distribution, and the reason the absolute value of the loss never reaches 0 is the constant term \(H(p)\).

The support problem. If \(p(x)>0\) while \(q(x)=0\), the KL is infinite. For a model to give probability 0 to an event that actually happens is to incur an infinite penalty. The reason an idealized softmax gives positive probability at finite logits, and the reason stabilization such as log-softmax is needed in real implementations, both connect to this extreme penalty.

\[\begin{aligned} D_{\mathrm{KL}}(p\,\|\,q) &= \sum_x p(x)\log\frac{p(x)}{q(x)} = H(p,q) - H(p) \\ D_{\mathrm{KL}}(p\,\|\,q) &\ge 0, \qquad \text{equality} \iff p = q \\ \arg\min_\theta H(p, q_\theta) &= \arg\min_\theta D_{\mathrm{KL}}(p\,\|\,q_\theta) \end{aligned}\]

Move q and watch the three values H(p), H(p,q) and KL together. Always H(p,q) = H(p) + KL, and KL ≥ 0. Press "Swap direction" and the roles of p and q are exchanged — the same two distributions, yet a different value. That is the asymmetry.

Definition. When the correct answer is class \(c\), the cross-entropy between the one-hot distribution \(p=\mathbf{e}_c\) and the model output \(q=\mathrm{softmax}(\boldsymbol{z})\) is \(H(p,q)=-\sum_k p_k\log q_k=-\log q_c\) — only one term survives. This is the classification loss we use every day.

Why they become equal to KL. In general \(H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q)\), so the two are not the same value. But a one-hot target has no uncertainty, so \(H(p)=0\). In this special case, therefore, \(D_{\mathrm{KL}}(p\|q)=H(p,q)\) and cross-entropy and KL coincide exactly.

Why they agree for optimization anyway. Even when the target is not one-hot, if the target distribution \(p\) is fixed during training then \(H(p)\) is a constant with respect to the model parameters. So the numbers for cross-entropy and KL differ by \(H(p)\), yet minimizing either yields the same optimal \(q\) and the same gradients. The interpretive difference remains: cross-entropy is the total cost of prediction, KL the cost added by the model being wrong.

The pressure of a one-hot loss. Driving the loss \(-\log q_c\) to 0 requires \(q_c=1\), which under a softmax requires the gap between the correct logit and the rest to grow without bound. So the model is under pressure to keep enlarging the correct logit, and can fall into overconfidence beyond its actual accuracy.

What label smoothing does. Blurring the target slightly into \(p=(1-\varepsilon)\mathbf{e}_c+\frac{\varepsilon}{K}\mathbf{1}\) makes \(H(p)>0\). Now cross-entropy and KL are no longer the same number, and their difference is \(H(p)\). The optimum also moves from \(q_c=1\) to a finite value, changing the target itself into an instruction to the model: "do not be completely certain."

\[\begin{aligned} H(p,q) &= H(p)+D_{\mathrm{KL}}(p\|q) \\ p=\mathbf e_c &\Rightarrow H(p)=0,\quad H(p,q)=D_{\mathrm{KL}}(p\|q)=-\log q_c \\ p^{\text{smooth}} &= (1-\varepsilon)\mathbf e_c+\tfrac{\varepsilon}{K}\mathbf 1 \end{aligned}\]

Raise the correct logit and watch the loss head toward 0 for a one-hot target. Increasing ε makes the target entropy positive, lifting the floor of the loss curve and moving the minimum to a finite logit.

Setup. Suppose the quantity we want is \(D_{\mathrm{KL}}(q\|p)=\mathbb E_{x\sim q}[\log(q(x)/p(x))]\). If we can draw samples \(x\sim q\) and compute the probability ratio \(r(x)=p(x)/q(x)\) at each sample, we can approximate the KL by replacing the expectation with a sample mean. The direction in the formulas below is fixed as \(q\|p\), so to obtain \(p\|q\) both the sampling distribution and the ratio must be flipped as well.

What an estimator is. The true value \(D\) we want to know is fixed, but the value \(\widehat D=T(x_1,\ldots,x_N)\) computed from a finite sample \(x_1,\ldots,x_N\) changes every time the sample is redrawn. A computational rule that takes a sample and guesses an unknown true value is called an estimator. The reason the point moves when you press "Resample" is not that \(D\) changed but that \(\widehat D\) varies with the sample.

Bias and unbiased estimators. Imagine drawing fresh samples of the same size infinitely many times and averaging the resulting estimates. How far that average is from the true value is the bias, \(\mathrm{Bias}(\widehat D)=\mathbb E[\widehat D]-D\). If the bias is 0 the estimator is called unbiased. Unbiased does not mean "exact on one computation"; it means the errors above and below the true value cancel on average over repetitions. So an unbiased estimator can still swing wildly, or go negative, on a small sample.

Is bias bad? Not necessarily. An estimator's error is usually assessed as \(\mathrm{MSE}=\mathbb E[(\widehat D-D)^2]=\mathrm{Bias}^2+\mathrm{Variance}\). Accepting a little bias in exchange for a large reduction in variance can make the total error smaller. That is why \(k_2\), while not an exactly unbiased estimator, can serve as a stable diagnostic when the two distributions are close.

Standard ways to design an estimator. First, Monte Carlo estimation, replacing the definition of the expectation directly by a sample mean, is the most direct, and \(k_1\) is of this kind. Second, local approximation, Taylor-expanding a hard-to-compute function around a point of interest to obtain a simpler expression, of which \(k_2\) is an example. Third, a control variate, adding or subtracting a term with zero expectation to preserve the mean while reducing the wobble, which is the role \(r-1\) plays in \(k_3\). Fourth, using the relationship between a convex function and its tangent, one can force the per-sample value to be non-negative just like the true divergence. In real design one does not insist on unbiasedness alone but compares bias, variance, computational cost, numerical stability, and support together.

\[\begin{aligned} \widehat D &= T(x_1,\ldots,x_N) \\ \mathrm{Bias}(\widehat D) &= \mathbb E[\widehat D]-D \\ \mathbb E[\widehat D]=D &\quad\Longleftrightarrow\quad \widehat D\text{ is unbiased} \\ \mathrm{MSE}(\widehat D) &= \mathrm{Bias}(\widehat D)^2+\mathrm{Var}(\widehat D) \end{aligned}\]

The first estimator \(k_1=-\log r\). It computes the definition one sample at a time. Since \(\mathbb E_q[k_1]=D_{\mathrm{KL}}(q\|p)\) it is unbiased, but on an individual sample it can be negative whenever \(r>1\). The true KL is non-negative, yet an estimate from a small batch may be negative, and if the tails of \(p\) and \(q\) differ the log-ratio can swing hard and the variance can grow large.

The second estimator \(k_2=\tfrac12(\log r)^2\). It is always non-negative and more stable when the two distributions are close, because for \(p\approx q\) the KL can be read as a second-order approximation in the log-ratio. But it is a generally biased approximation, and as the two distributions move apart it can differ substantially from the real KL. It is a trade between a stable number and an accurate expectation.

The third estimator \(k_3=(r-1)-\log r\). Since \(\log r\le r-1\), we have \(k_3\ge0\) for every sample. At the same time \(\mathbb E_q[r-1]=\int q(x)(p(x)/q(x)-1)\,dx=0\), so it merely adds a mean-zero term to \(k_1\). Hence \(k_3\) is also an unbiased estimator of \(D_{\mathrm{KL}}(q\|p)\) while never being negative on a sample. That \(r-1\) term acts as a control variate, leaving the expectation unchanged while reducing the variance.

Why \(k_3\) is natural. Subtracting from \(-\log r\) its tangent at \(r=1\), namely \(-(r-1)\), leaves \((r-1)-\log r\). A convex function lies above its tangent, so the result is always non-negative and is exactly 0 at \(p=q\), that is \(r=1\). The same idea of subtracting the tangent at \(r=1\) builds non-negative sample estimators for more general \(f\)-divergences too.

Using them in practice. Average over \(N\) samples as \(\widehat D=\frac1N\sum_i k(x_i)\). \(k_1\) is the most direct from the definition, \(k_2\) is useful as a simple diagnostic under the premise that the two distributions are very close, and \(k_3\) is a good choice when unbiasedness and per-sample non-negativity are both wanted. That said, none of these formulas solves the problem of computing the probability ratio stably, nor the support problem of \(p(x)=0\) where \(q(x)>0\).

\[\begin{aligned} r(x)&=\frac{p(x)}{q(x)},\qquad x\sim q \\ k_1&=-\log r &&\text{(unbiased, high variance)} \\ k_2&=\tfrac12(\log r)^2 &&\text{(biased, local approximation)} \\ k_3&=(r-1)-\log r &&\text{(unbiased, nonnegative)} \\ \widehat D_{\mathrm{KL}}(q\|p)&=\frac1N\sum_{i=1}^N k(x_i) \end{aligned}\]

The key point is that "KL is non-negative" and "the estimate from one sample is non-negative" are different requirements. \(k_1\) is accurate on average but individual values can be negative, while \(k_3\) obtains both properties by adding a term that does not change the mean.

The two directions. When approximating a true distribution \(p\) with a model \(q\), minimizing \(D_{\mathrm{KL}}(p\|q)\) is called forward KL (moment matching), and minimizing \(D_{\mathrm{KL}}(q\|p)\) is called reverse KL (information projection).

Forward KL covers. In \(D_{\mathrm{KL}}(p\|q)=\sum p\log(p/q)\) the expectation is taken with \(p\). So wherever \(p(x)\) is large and \(q(x)\) is small there is a heavy penalty, while wherever \(p(x)=0\) the weight is 0 and whatever \(q\) does costs nothing. As a result \(q\) tries to cover everywhere \(p\) has mass, giving a broad, blunt solution that blankets even the empty valleys between peaks. This is called mode-covering.

Reverse KL selects. In \(D_{\mathrm{KL}}(q\|p)\) the expectation is taken with \(q\). So wherever \(q(x)\) is large and \(p(x)\) is small there is a heavy penalty. The best \(q\) can do is avoid the regions where \(p\) is small, so it curls up on a single peak and abandons the rest. This is called mode-seeking, or zero-forcing.

Which is right? It depends on the purpose. If "nothing may be missed" matters, forward is better; if "the generated samples must be plausible" matters, reverse is better. Maximum likelihood training is essentially forward KL, which partly explains why generative models with insufficient capacity tend to produce blurry, averaged samples (for example the blurry images of a VAE).

In machine learning. What appears in variational inference and in the ELBO of a VAE is the reverse KL \(D_{\mathrm{KL}}(q_\phi(z\mid x)\,\|\,p(z))\) — because that is the computable side (the expectation is over \(q\), so it can be estimated by sampling from \(q\)). By contrast, what is used when a student follows a teacher in knowledge distillation is usually the forward KL. Expectation Propagation uses forward, and mean-field approximation uses reverse.

\[\begin{aligned} \text{forward: } &\min_q D_{\mathrm{KL}}(p\,\|\,q) \;\Rightarrow\; \text{mode-covering} \\ \text{reverse: } &\min_q D_{\mathrm{KL}}(q\,\|\,p) \;\Rightarrow\; \text{mode-seeking} \\ p>0, q\approx 0 &\Rightarrow \text{forward blows up} \\ q>0, p\approx 0 &\Rightarrow \text{reverse blows up} \end{aligned}\]

The grey true distribution has two peaks, and the orange q is a Gaussian with only one. Press "Find the optimum" and q moves to the optimal location and width for the chosen direction — forward spreads wide trying to cover both peaks, reverse clings narrowly to one. Same p, same model family, different answers.

Definition. Symmetrize by routing through the average of the two distributions, \(m=\frac{1}{2}(p+q)\): \(D_{\mathrm{JS}}(p,q)=\frac{1}{2}D_{\mathrm{KL}}(p\|m)+\frac{1}{2}D_{\mathrm{KL}}(q\|m)\). By construction \(D_{\mathrm{JS}}(p,q)=D_{\mathrm{JS}}(q,p)\), and the value is confined to \([0,\log 2]\).

Why it does not diverge. If \(p(x)>0\) then \(m(x)\ge p(x)/2>0\), so the "zero denominator" situation that sent KL to infinity cannot arise at all. Even in the extreme case where the supports do not overlap at all, the value stops at \(\log 2\). Being bounded and symmetric makes it convenient to handle as an optimization target.

In machine learning: GANs. In the original GAN objective, when the discriminator is optimal, one can show that what the generator minimizes is exactly \(2D_{\mathrm{JS}}(p_{\text{data}}, p_g)-\log 4\). But boundedness has a price — if the two distributions barely overlap, JS saturates at \(\log 2\) and the gradient vanishes. That the data manifold and the generated manifold overlap poorly in high dimensions made GAN training unstable, and switching to the Wasserstein distance in WGAN was the response.

Note: the square root is a distance. \(\sqrt{D_{\mathrm{JS}}}\) is a genuine metric satisfying the triangle inequality (the Jensen–Shannon distance). Which means it can be used to build a distance matrix between distributions, or to cluster them.

\[\begin{aligned} m &= \tfrac{1}{2}(p+q) \\ D_{\mathrm{JS}}(p,q) &= \tfrac{1}{2}D_{\mathrm{KL}}(p\|m) + \tfrac{1}{2}D_{\mathrm{KL}}(q\|m) \\ 0 \le D_{\mathrm{JS}} &\le \log 2 \quad (\text{nat}) \end{aligned}\]

Move q far away from p. KL grows sharply and diverges, but JS stops at log 2 ≈ 0.693. That saturation is precisely what made gradients vanish in GANs — however much further apart they move, the value does not change, so there is no way to know which direction to go.

Definition. \(I(X;Y) = D_{\mathrm{KL}}\big(p(x,y)\,\|\,p(x)p(y)\big) = \sum_{x,y} p(x,y)\log\frac{p(x,y)}{p(x)p(y)}\). That is, the KL measuring how far the joint distribution sits from the independent case. Think of it as quantifying, in information terms, the "residual \(P(x,y)-P(x)P(y)\)" demo of the previous article.

Reading it as a reduction in uncertainty. The same quantity can also be written \(I(X;Y)=H(Y)-H(Y\mid X)\). Here \(H(Y)\) is the uncertainty about \(Y\) when \(X\) is unknown, and \(H(Y\mid X)\) is the uncertainty remaining after learning \(X\). The difference is the information about \(Y\) gained by knowing \(X\). This expression makes it look non-obvious that \(I(X;Y)=I(Y;X)\), yet the definition makes it plain.

Intuition: the overlap of two circles. The last form above, \(I(X;Y)=H(X)+H(Y)-H(X,Y)\), has literally the same shape as the inclusion–exclusion principle for sets (\(|A\cap B|=|A|+|B|-|A\cup B|\)). So a picture drawing entropies as areas works well — let \(H(X)\) and \(H(Y)\) be the areas of two circles and the joint entropy \(H(X,Y)\) the area of their union, and the mutual information is the overlap. The non-overlapping crescents are the conditional entropies \(H(X\mid Y)\) and \(H(Y\mid X)\), the uncertainty peculiar to each even after knowing the other. The symmetry \(I(X;Y)=I(Y;X)\) also looks obvious in this picture — an overlapping area has no order. Two circles entirely apart is independence (\(I=0\)); two entirely coincident is the case where one completely determines the other.

But the analogy has limits. This picture is safe only for two variables. With three or more, the quantity corresponding to "the area where three circles overlap" (the three-variable extension of mutual information) can be negative and so makes no sense as an area. Use it as a device for holding on to the two-variable case, and return to the definition beyond that.

Independence gives exactly 0. If \(p(x,y)=p(x)p(y)\) then the inside of the log is 1 and \(I=0\). And thanks to the non-negativity of KL the converse holds too — if \(I(X;Y)=0\) the variables must be independent. This is exactly the property the correlation coefficient lacked in the previous article. Even a relationship like \(Y=X^2\), with zero correlation yet complete dependence, has nonzero mutual information, because it is not restricted to linearity.

Why it is hard to compute directly. Mutual information needs the ratio of the joint density \(p(x,y)\) to the two marginal densities \(p(x),p(y)\). For low-dimensional discrete variables it can be counted from a frequency table, but in high-dimensional continuous spaces such as images or embeddings there are so many possible locations that a finite sample barely fills the space. Estimating the three densities even slightly wrongly makes the error in the log-ratio large, and because a neural representation keeps changing during training, a density estimated once soon goes stale. So in real high-dimensional problems one often uses a computable variational lower bound or another surrogate objective rather than computing the MI value directly.

In machine learning. Formalizing the goal of representation learning as "carry as much information about the input as possible while discarding noise" gives the Information Bottleneck. In feature selection, mutual information with the label is used as a criterion, but in high dimensions the estimation difficulties described above mean the theoretical relationship between the actual objective and MI must be interpreted with care.

\[\begin{aligned} I(X;Y) &= D_{\mathrm{KL}}\big(p(x,y)\,\|\,p(x)p(y)\big) \\ &= H(Y) - H(Y\mid X) = H(X) - H(X\mid Y) \\ &= H(X) + H(Y) - H(X,Y) \\ I(X;Y) &\ge 0, \qquad = 0 \iff X \perp Y \end{aligned}\]

Drag the grid cells to change the joint distribution. Piling mass on the diagonal ties X and Y strongly together and I grows; pressing "Make independent" drops I to exactly 0. H(X), H(Y) and H(X,Y) are displayed alongside so the relationship among the three terms can be checked too.

The problem. A latent-variable model assumes we first draw an unseen cause \(z\) and then produce the data \(x\) from that cause: \(p_\theta(x,z)=p(z)p_\theta(x\mid z)\). What we want is \(p_\theta(x)=\int p_\theta(x,z)\,dz\), but for a complex decoder this integral over all possible \(z\), and the true posterior \(p_\theta(z\mid x)\), are often intractable to compute exactly.

The key idea: substitute an easy distribution for a hard one. Introduce an encoder \(q_\phi(z\mid x)\) that assigns probability to "the \(z\) that plausibly produced this \(x\)." We do not use \(q\) because we already know the true posterior. On the contrary, precisely because the true posterior cannot be computed, we approximate it with a family such as Gaussians from which samples are easy to draw and densities easy to evaluate, and learn it jointly.

How the bound arises. Multiplying and dividing by \(q_\phi(z\mid x)\) inside the integral gives \[\log p_\theta(x)=\log\mathbb{E}_{q_\phi(z\mid x)}\!\left[\frac{p_\theta(x,z)}{q_\phi(z\mid x)}\right].\] The log is concave, so Jensen's inequality \(\log\mathbb{E}[A]\ge\mathbb{E}[\log A]\) applies. The result is \(\mathcal L_{\mathrm{ELBO}}=\mathbb{E}_{q_\phi}\!\left[\log\frac{p_\theta(x,z)}{q_\phi(z\mid x)}\right]\le\log p_\theta(x)\). In other words we have laid a computable floor beneath the intractable ceiling \(\log p(x)\).

Why Jensen's inequality holds. This single step is the whole derivation, so it is worth attaching an intuition. The log is a concave function — convex upward — so a chord joining any two points always lies below the curve. Now \(\mathbb{E}[\log A]\) is a weighted average of points on the curve and so lies on those chords, while \(\log\mathbb{E}[A]\) is the point reached by first averaging the \(x\)-coordinates and then climbing onto the curve. Hence always \(\log\mathbb{E}[A]\ge\mathbb{E}[\log A]\). Concrete numbers make it plain — if \(A\) takes the values \(1\) and \(100\) with probability one half each, then \(\log\mathbb{E}[A]=\log 50.5\approx 3.92\) while \(\mathbb{E}[\log A]=\tfrac12(\log 1+\log 100)\approx 2.30\). "Average then take the log" is always at least "take the log then average," and the difference is exactly the gap we have given up. The less \(A\) fluctuates (the closer all its values are), the closer the two become — which is the other face of the statement below that the gap shrinks as \(q_\phi\) approaches the true posterior.

The exact meaning of the gap. This inequality is in fact the following identity: \[\log p_\theta(x)=\mathcal L_{\mathrm{ELBO}}+D_{\mathrm{KL}}\big(q_\phi(z\mid x)\|p_\theta(z\mid x)\big).\] The second term is non-negative, which is why the ELBO is a lower bound, and that KL is precisely the gap between ceiling and floor. As \(q_\phi\) approaches the true posterior the gap shrinks. If the true posterior cannot be represented by the chosen family for \(q\), the gap may never close entirely.

Why it is actually easy to compute. The expectation in the ELBO is with respect to the \(q_\phi\) we designed, so it can be estimated by Monte Carlo with \(z\sim q_\phi(z\mid x)\). In a VAE, the reparameterization \(z=\mu_\phi(x)+\sigma_\phi(x)\odot\varepsilon\), \(\varepsilon\sim\mathcal N(0,I)\), lets gradients reach the encoder parameters \(\phi\) even through a random sample.

The two demands of a VAE. Rearranging the ELBO gives \(\mathbb{E}_{q_\phi}[\log p_\theta(x\mid z)]-D_{\mathrm{KL}}(q_\phi(z\mid x)\|p(z))\). The first term is the reconstruction term, demanding that the chosen \(z\) explain and rebuild \(x\) well; the second is the regularization term, keeping each input's latent distribution from straying too far from the shared prior. Intuitively the first says "write it down in enough detail to tell each datum apart," and the second says "gather those notes into one tidy latent space." The balance between them produces both reconstruction quality and a latent space you can generate from.

Maximization and the sign of the loss. In theory we maximize the ELBO, but implementations usually minimize \(-\mathcal L_{\mathrm{ELBO}}\), that is the sum of the reconstruction NLL and the KL. Note that the regularization term \(D_{\mathrm{KL}}(q_\phi(z\mid x)\|p(z))\) and the bound's gap \(D_{\mathrm{KL}}(q_\phi(z\mid x)\|p_\theta(z\mid x))\) are different KLs and must not be confused. The former is a computable part of the VAE loss; the latter is the size of an approximation error that generally cannot be computed directly.

Relation to diffusion models. The original variational diffusion derivation treats the several noise states as latent variables and builds a variational lower bound on the data log-likelihood. Thanks to the Gaussian transitions the per-step KLs are computable, and with a particular weighting and parameterization it simplifies into the widely used noise-prediction MSE. That said, not every training objective of every diffusion model is the same ELBO itself, and in practice a simplified objective with altered weighting is often used.

\[\begin{aligned} \log p_\theta(x) &= \mathcal{L}_{\text{ELBO}}(\theta,\phi;x) + D_{\mathrm{KL}}\big(q_\phi(z|x)\,\|\,p_\theta(z|x)\big) \\ \mathcal{L}_{\text{ELBO}} &= \mathbb{E}_{q_\phi}[\log p_\theta(x\mid z)] - D_{\mathrm{KL}}\big(q_\phi(z|x)\,\|\,p(z)\big) \\ &\le \log p_\theta(x) \quad \text{(always)} \end{aligned}\]

Match the mean and width of the approximate posterior q to the true posterior (grey). In the bar figure the ELBO (blue) rises and the KL gap (orange) shrinks, while their sum log p(x) does not change whatever q is — that is the key point. Match them exactly and the gap becomes 0 and the bound is tight.