信息测度实例（一）：信息熵、联合熵、条件熵、互信息、条件互信息

例1：均匀分布

注意，本文所举实例中对数运算均采用自然对数底数ee，并且约定：0×log⁡0=00\times \log 0=0

假设随机变量XX，YY，ZZ：

XX	11	11	11	00	00	00
YY	00	11	11	11	00	00
ZZ	00	00	11	11	11	00

例1.1 信息熵

由信息熵的定义，有：
H(X)=−∑x∈Xp(x)⋅log⁡p(x)=−12log⁡12−12log⁡12=log⁡2=0.693147H(Y)=−∑y∈Yp(y)⋅log⁡p(y)=−12log⁡12−12log⁡12=log⁡2=0.693147H(Z)=−∑z∈Zp(z)⋅log⁡p(z)=−12log⁡12−12log⁡12=log⁡2=0.693147

例1.2 联合熵

由概率可知：
p(x=0,y=0)=13，p(x=0,y=1)=16 p(x=1,y=0)=16，p(x=1,y=1)=13 p\left( x=0,y=0 \right) =\frac{1}{3}\text{，}p\left( x=0,y=1 \right) =\frac{1}{6} \ \ \ \ \ p\left( x=1,y=0 \right) =\frac{1}{6}\text{，}p\left( x=1,y=1 \right) =\frac{1}{3} 则：
H(X,Y)=−∑x∈X∑y∈Yp(x,y)log⁡p(x,y)=−16log⁡16−13log⁡13−13log⁡13−16log⁡16=13×(log⁡6+log⁡9)=1.32966

例1.3 条件熵

计算可得
p(x=0∣y=0)=p(x=0,y=0)p(y=0)=1312=23 p(x=1∣y=0)=p(x=1,y=0)p(y=0)=1612=13 p(x=0∣y=1)=p(x=0,y=1)p(y=1)=1612=13 p(x=1∣y=1)=p(x=1,y=1)p(y=1)=1312=23 p\left( x=0\left| y=0 \right. \right) =\frac{p\left( x=0,y=0 \right)}{p\left( y=0 \right)}=\frac{\frac{1}{3}}{\frac{1}{2}}=\frac{2}{3} \ \ \ \ p\left( x=1\left| y=0 \right. \right) =\frac{p\left( x=1,y=0 \right)}{p\left( y=0 \right)}=\frac{\frac{1}{6}}{\frac{1}{2}}=\frac{1}{3} \ \ \ \ p\left( x=0\left| y=1 \right. \right) =\frac{p\left( x=0,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{1}{6}}{\frac{1}{2}}=\frac{1}{3} \ \ \ \ p\left( x=1\left| y=1 \right. \right) =\frac{p\left( x=1,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{1}{3}}{\frac{1}{2}}=\frac{2}{3} 则
H(X∣Y=0)=−p(x=0∣y=0)⋅log⁡p(x=0∣y=0)−p(x=1∣y=0)⋅log⁡p(x=1∣y=0) =−23log⁡23−13log⁡13 =log⁡3−23log⁡2
H(X∣Y=1)=−p(x=0∣y=1)⋅log⁡p(x=0∣y=1)−p(x=1∣y=1)⋅log⁡p(x=1∣y=1) =−13log⁡13−23log⁡23 =log⁡3−23log⁡2
通过以上计算可到条件熵为：
H(X∣Y)=p(y=0)⋅H(X∣Y=0)+p(y=1)⋅H(X∣Y=1) =12×(log⁡3−23log⁡2)+12×(log⁡3−23log⁡2) =log⁡3−23log⁡2=0.636514 同理可计算出
H(Y∣X)=log⁡3−23log⁡2=0.636514 H\left( Y\left| X \right. \right) =\log 3-\frac{2}{3}\log 2=0.636514

例1.4 互信息

p(x=0,y=0)=13，p(x=0,y=1)=16，p(x=1,y=0)=16，p(x=1,y=1)=13 p\left( x=0,y=0 \right) =\frac{1}{3}\text{，}p\left( x=0,y=1 \right) =\frac{1}{6}\text{，}p\left( x=1,y=0 \right) =\frac{1}{6}\text{，}p\left( x=1,y=1 \right) =\frac{1}{3}

则XX和YY的互信息为：
I(X;Y)=p(x=1,y=0)⋅log⁡p(x=1,y=0)p(x=1)⋅p(y=0)+p(x=1,y=1)⋅log⁡p(x=1,y=1)p(x=1)⋅p(y=1) +p(x=0,y=0)⋅log⁡p(x=0,y=0)p(x=0)⋅p(y=0)+p(x=0,y=1)⋅log⁡p(x=0,y=1)p(x=0)⋅p(y=1)=16×log⁡1612×12+13×log⁡1312×12+13×log⁡1312×12+16×log⁡1612×12=13×(5log⁡2−3log⁡3)=0.056633

例1.5 条件互信息

p(z=0)=12，p(z=1)=12 p(x=1∣z=0)=23，p(x=0∣z=0)=13，p(x=1∣z=1)=13，p(x=0∣z=1)=23 p(y=1∣z=0)=13，p(y=0∣z=0)=23，p(y=1∣z=1)=13，p(y=0∣z=1)=23 p\left( z=0 \right) =\frac{1}{2}，p\left( z=1 \right) =\frac{1}{2} \ \ \ \ p\left( \left. x=1 \right|z=0 \right) =\frac{2}{3}\text{，}p\left( \left. x=0 \right|z=0 \right) =\frac{1}{3}\text{，}p\left( \left. x=1 \right|z=1 \right) =\frac{1}{3}\text{，}p\left( \left. x=0 \right|z=1 \right) =\frac{2}{3} \ \ \ \ p\left( \left. y=1 \right|z=0 \right) =\frac{1}{3}\text{，}p\left( \left. y=0 \right|z=0 \right) =\frac{2}{3}\text{，}p\left( \left. y=1 \right|z=1 \right) =\frac{1}{3}\text{，}p\left( \left. y=0 \right|z=1 \right) =\frac{2}{3} \ 记
p11=p(x=1,y=0∣z=0)=13，p21=p(x=1,y=1∣z=0)=13，p31=p(x=0,y=0∣z=0)=13，p41=p(x=0,y=1∣z=0)=0p12=p(x=1,y=0∣z=1)=0，p22=p(x=1,y=1∣z=1)=13，p32=p(x=0,y=0∣z=1)=13，p42=p(x=0,y=1∣z=1)=13 p_{11}=p\left( \left. x=1,y=0 \right|z=0 \right) =\frac{1}{3}\text{，}p_{21}=p\left( \left. x=1,y=1 \right|z=0 \right) =\frac{1}{3}\text{，}p_{31}=p\left( \left. x=0,y=0 \right|z=0 \right) =\frac{1}{3}\text{，}p_{41}=p\left( \left. x=0,y=1 \right|z=0 \right) =0 \ \mathrm{ } \ p_{12}=p\left( \left. x=1,y=0 \right|z=1 \right) =0\text{，}p_{22}=p\left( \left. x=1,y=1 \right|z=1 \right) =\frac{1}{3}\text{，}p_{32}=p\left( \left. x=0,y=0 \right|z=1 \right) =\frac{1}{3}\text{，}p_{42}=p\left( \left. x=0,y=1 \right|z=1 \right) =\frac{1}{3}

P11=p(x=1,y=0∣z=0)p(x=1∣z=0)⋅p(y=0∣z=0)=1323×23=34，P12=p(x=1,y=0∣z=1)p(x=1∣z=1)⋅p(y=0∣z=1)=0 P21=p(x=1,y=1∣z=0)p(x=1∣z=0)⋅p(y=1∣z=0)=1323×13=32，P22=p(x=1,y=1∣z=1)p(x=1∣z=1)⋅p(y=1∣z=1)=1313×23=32 P31=p(x=0,y=0∣z=0)p(x=0∣z=0)⋅p(y=0∣z=0)=1313×23=32，P32=p(x=0,y=0∣z=1)p(x=0∣z=1)⋅p(y=0∣z=1)=1323×13=32， P41=p(x=0,y=1∣z=0)p(x=0∣z=0)⋅p(y=1∣z=0)=0，P42=p(x=0,y=1∣z=1)p(x=0∣z=1)⋅p(y=1∣z=1)=1323×23=34 故条件互信息为：
I(X;Y∣Z)=∑z∈Zp(z)∑x∈X∑y∈Yp(x,y∣z)⋅log⁡p(x,y∣z)p(x∣z)p(y∣z) =p(z=0)×[p11⋅log⁡P11+p21⋅log⁡P21+p31⋅log⁡P31+p41⋅log⁡P41] +p(z=1)×[p12⋅log⁡P12+p22⋅log⁡P22+p32⋅log⁡P32+p42⋅log⁡P42] =12×[13log⁡34+13log⁡32+13log⁡32+0]+12×[0+13log⁡32+13log⁡32+13log⁡34] =log⁡3−43log⁡2=0.174416