A Statistical View of Entropy — Question 9

Question 9

Correlations reduce joint entropy while leaving individual entropies unchanged

Two binary variables A and B have joint probabilities $P(00)=P(11)=1/2$ and $P(01)=P(10)=0$. Use $S=-k_B\sum p\ln p$. Compare this ensemble with two independent fair binary variables.

1. Find the marginal probabilities and entropy of each variable, then the joint entropy.

2. Calculate the correlation entropy difference $S_A+S_B-S_{AB}$. Explain why knowledge of A determines B despite each variable being individually uncertain.

Question 9 - Solution

Correlations reduce joint entropy while leaving individual entropies unchanged

(a) Method and calculation

$$P_A(0)=P_A(1)=P_B(0)=P_B(1)=1/2.$$$$S_A=S_B=k_B\ln2,\quad S_{AB}=-2k_B(\tfrac12\ln\tfrac12)=k_B\ln2.$$

(b) Interpretation and further calculation

$$S_A+S_B-S_{AB}=k_B\ln2.$$$$S_{AB,\mathrm{independent}}=k_B\ln4=2k_B\ln2,\quad P(B=A\mid A)=1.$$Only the joint probabilities change between the ensembles. Correlations remove one binary choice without changing either marginal distribution.

Physical check

Entropy additivity applies to independent variables. Physically creating correlations requires accounting for the preparation process and environment.