A Statistical View of Entropy — Question 8

Question 8

Unequal state probabilities require a weighted entropy

A system occupies four possible states with probabilities $(1/2,1/4,1/4,0)$. For this exercise use the ensemble entropy $S=-k_B\sum_i p_i\ln p_i$, with the continuous convention $0\ln0=0$. Compare with a uniform distribution over the same four states.

1. Evaluate the entropy of the stated distribution and its deficit below the uniform value.

2. Explain why $k_B\ln3$, obtained by counting nonzero-probability states, is incorrect here. Derive the reduction to $k_B\ln\Omega$ when all $\Omega$ probabilities are equal.

Question 8 - Solution

Unequal state probabilities require a weighted entropy

(a) Method and calculation

$$S/k_B=-\tfrac12\ln\tfrac12-2(\tfrac14\ln\tfrac14)=\tfrac32\ln2\approx1.03972.$$$$S_{\mathrm{uniform}}/k_B=\ln4=2\ln2,\quad(S_{\mathrm{uniform}}-S)/k_B=\tfrac12\ln2\approx.346574.$$

(b) Interpretation and further calculation

$$p_i=1/\Omega:\quad S=-k_B\Omega(1/\Omega)\ln(1/\Omega)=k_B\ln\Omega.$$$$\ln3\approx1.09861\ne\tfrac32\ln2.$$Counting occupied states alone would assign equal weight to three states, contradicting the specified probabilities.

Physical check

The zero-probability term vanishes by its limit; taking $\ln0$ alone would be an invalid intermediate step.