信息测度实例（二）：信息熵、联合熵、条件熵、互信息、条件互信息

例二：非均匀分布

假设随机变量X有两个分布律：

X	A	B	C	D
P	\frac{1}{4}	\frac{1}{4}	\frac{1}{4}	\frac{1}{4}

X	A	B	C	D
P	\frac{1}{2}	\frac{1}{4}	\frac{1}{8}	\frac{1}{8}

Y表示选择两种分布律的概率：

Y	1	2
P	\frac{1}{2}	\frac{1}{2}

则X与Y的联合分布律为：

X	A	B	C	D	A	B	C	D
Y	1	1	1	1	2	2	2	2
P	\frac{1}{8}	\frac{1}{8}	\frac{1}{8}	\frac{1}{8}	\frac{1}{4}	\frac{1}{8}	\frac{1}{16}	\frac{1}{16}

Z的分布率分别为：

Z	0	1
P	\frac{1}{2}	\frac{1}{2}

X、Y与Z的联合分布律为：

X	A	B	C	D	A	B	C	D	A	B
Y	1	1	1	1	2	2	2	2	1	1
Z	0	0	0	0	0	0	0	0	1	1
P	\frac{1}{16}	\frac{1}{16}	\frac{1}{16}	\frac{1}{16}	\frac{1}{8}	\frac{1}{16}	\frac{1}{32}	\frac{1}{32}	\frac{1}{4}	\frac{1}{4}

例2.1 信息熵

由信息熵的定义，有：
\begin{aligned} H\left( X \right) &=-\sum_{x\in X}{p\left( x \right)}\cdot \log p\left( x \right) \\ &=-\frac{7}{16}\log \frac{7}{16}-\frac{3}{8}\log \frac{3}{8}-\frac{3}{32}\log \frac{3}{32}-\frac{3}{32}\log \frac{3}{32} \\ &=\frac{1}{16}\times \left( 61\log 2-7\log 7-9\log 3 \right) =1.17332 \\ \\ H\left( Y \right) &=-\sum_{y\in Y}{p\left( y \right)}\cdot \log p\left( y \right) =-\frac{3}{4}\log \frac{3}{4}-\frac{1}{4}\log \frac{1}{4} \\ &=2\log 2-\frac{3}{4}\log 3=0.562335 \\ H\left( Z \right) &=-\sum_{z\in Z}{p\left( z \right)}\cdot \log p\left( z \right) =-\frac{1}{2}\log \frac{1}{2}-\frac{1}{2}\log \frac{1}{2} \\ &=\log 2=0.693147 \end{aligned}

例2.2 联合熵

由概率可知：
p\left( x=A,y=1 \right) =\frac{5}{16}\text{，}p\left( x=A,y=2 \right) =\frac{1}{8} \\ \ \ \\ \\ p\left( x=B,y=1 \right) =\frac{5}{16}\text{，}p\left( x=B,y=2 \right) =\frac{1}{16} \\ \ \ \\ \\ p\left( x=C,y=1 \right) =\frac{1}{16}\text{，}p\left( x=C,y=2 \right) =\frac{1}{32} \\ \ \ \\ \\ p\left( x=D,y=1 \right) =\frac{1}{16}\text{，}p\left( x=D,y=2 \right) =\frac{1}{32}则：
\begin{aligned} H\left( X,Y \right) &=-\sum_{x\in X}{\sum_{y\in Y}{p\left( x,y \right)}\log}p\left( x,y \right) \\ \\ &=-\frac{5}{16}\log \frac{5}{16}-\frac{5}{16}\log \frac{5}{16}-\frac{1}{16}\log \frac{1}{16}-\frac{1}{16}\log \frac{1}{16}-\frac{1}{8}\log \frac{1}{8}-\frac{1}{16}\log \frac{1}{16}-\frac{1}{32}\log \frac{1}{32}-\frac{1}{32}\log \frac{1}{32} \\ \\ &=\frac{1}{16}\times \left( 63\log 2-10\log 5 \right) =1.72337 \end{aligned}

例2.3 条件熵

计算可得
p\left( x=A\left| y=1 \right. \right) =\frac{p\left( x=A,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{5}{16}}{\frac{3}{4}}=\frac{5}{12}\text{，}p\left( x=A\left| y=2 \right. \right) =\frac{p\left( x=A,y=2 \right)}{p\left( y=2 \right)}=\frac{\frac{1}{8}}{\frac{1}{4}}=\frac{1}{2} \\ \,\,\mathrm{ } \\ p\left( x=B\left| y=1 \right. \right) =\frac{p\left( x=B,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{5}{16}}{\frac{3}{4}}=\frac{5}{12}\text{，}p\left( x=B\left| y=2 \right. \right) =\frac{p\left( x=B,y=2 \right)}{p\left( y=2 \right)}=\frac{\frac{1}{16}}{\frac{1}{4}}=\frac{1}{4} \\ \,\,\mathrm{ } \\ p\left( x=C\left| y=1 \right. \right) =\frac{p\left( x=C,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{1}{16}}{\frac{3}{4}}=\frac{1}{12}\text{，}p\left( x=C\left| y=2 \right. \right) =\frac{p\left( x=C,y=2 \right)}{p\left( y=2 \right)}=\frac{\frac{1}{32}}{\frac{1}{4}}=\frac{1}{8} \\ \,\,\mathrm{ } \\ p\left( x=D\left| y=1 \right. \right) =\frac{p\left( x=D,y=1 \right)}{p\left( y=1 \right)}=\frac{\frac{1}{16}}{\frac{3}{4}}=\frac{1}{12}\text{，}p\left( x=D\left| y=2 \right. \right) =\frac{p\left( x=D,y=2 \right)}{p\left( y=2 \right)}=\frac{\frac{1}{32}}{\frac{1}{4}}=\frac{1}{8}
则
\begin{aligned} H\left( X\left| Y \right. =1 \right) &=-p\left( x=A\left| y=1 \right. \right) \cdot \log p\left( x=A\left| y=1 \right. \right) -p\left( x=B\left| y=1 \right. \right) \cdot \log p\left( x=B\left| y=1 \right. \right) \\ \,\, & \,\,\,\,\,\,-p\left( x=C\left| y=1 \right. \right) \cdot \log p\left( x=C\left| y=1 \right. \right) -p\left( x=D\left| y=1 \right. \right) \cdot \log p\left( x=D\left| y=1 \right. \right) \\ \,\,\mathrm{ } \\ &=-\frac{5}{12}\log \frac{5}{12}-\frac{5}{12}\log \frac{5}{12}-\frac{1}{12}\log \frac{1}{12}-\frac{1}{12}\log \frac{1}{12} \\ \,\,\mathrm{ } \\ &=\log 12-\frac{5}{6}\log 5=1.14371 \end{aligned}
\begin{aligned} H\left( X\left| Y \right. =2 \right) &=-p\left( x=A\left| y=2 \right. \right) \cdot \log p\left( x=A\left| y=2 \right. \right) -p\left( x=B\left| y=2 \right. \right) \cdot \log p\left( x=B\left| y=2 \right. \right) \\ \,\, & \,\,\,\,\,\,-p\left( x=C\left| y=2 \right. \right) \cdot \log p\left( x=C\left| y=2 \right. \right) -p\left( x=D\left| y=2 \right. \right) \cdot \log p\left( x=D\left| y=2 \right. \right) \\ \,\,\mathrm{ } \\ &=-\frac{1}{2}\log \frac{1}{2}-\frac{1}{4}\log \frac{1}{4}-\frac{1}{8}\log \frac{1}{8}-\frac{1}{8}\log \frac{1}{8} \\ \,\,\mathrm{ } \\ &=\frac{7}{4}\log 2=1.21301 \end{aligned}
通过以上计算可到条件熵为：
\begin{aligned} H\left( X\left| Y \right. \right) &=p\left( y=1 \right) \cdot H\left( X\left| Y=1 \right. \right) +p\left( y=2 \right) \cdot H\left( X\left| Y=2 \right. \right) \\ \ \ \\ &=\frac{3}{4}\times \left( \log 12-\frac{5}{6}\log 5 \right) +\frac{1}{4}\times \left( \frac{7}{4}\log 2 \right) \\ \ \ \\ &=\frac{3}{4}\log 12-\frac{5}{8}\log 5+\frac{7}{16}\log 2=1.16103 \end{aligned}同理可计算出
p\left( y=1\left| x=A \right. \right) =\frac{p\left( y=1,x=A \right)}{p\left( x=A \right)}=\frac{\frac{5}{16}}{\frac{7}{16}}=\frac{5}{7}\text{，}p\left( y=2\left| x=A \right. \right) =\frac{p\left( y=2,x=A \right)}{p\left( x=A \right)}=\frac{\frac{1}{8}}{\frac{7}{16}}=\frac{2}{7} \\ \,\, \\ p\left( y=1\left| x=B \right. \right) =\frac{p\left( y=1,x=B \right)}{p\left( x=B \right)}=\frac{\frac{5}{16}}{\frac{3}{8}}=\frac{5}{6}\text{，}p\left( y=2\left| x=B \right. \right) =\frac{p\left( y=2,x=B \right)}{p\left( x=B \right)}=\frac{\frac{1}{16}}{\frac{3}{8}}=\frac{1}{6} \\ \,\, \\ p\left( y=1\left| x=C \right. \right) =\frac{p\left( y=1,x=C \right)}{p\left( x=C \right)}=\frac{\frac{1}{16}}{\frac{3}{32}}=\frac{2}{3}\text{，}p\left( y=2\left| x=C \right. \right) =\frac{p\left( y=2,x=C \right)}{p\left( x=C \right)}=\frac{\frac{1}{32}}{\frac{3}{32}}=\frac{1}{3} \\ \,\, \\ p\left( y=1\left| x=D \right. \right) =\frac{p\left( y=1,x=D \right)}{p\left( x=D \right)}=\frac{\frac{1}{16}}{\frac{3}{32}}=\frac{2}{3}\text{，}p\left( y=2\left| x=D \right. \right) =\frac{p\left( y=2,x=D \right)}{p\left( x=D \right)}=\frac{\frac{1}{32}}{\frac{3}{32}}=\frac{1}{3}则
\begin{aligned} H\left( Y\left| X \right. =A \right) &=-p\left( y=1\left| x=A \right. \right) \cdot \log p\left( y=1\left| x=A \right. \right) -p\left( y=2\left| x=A \right. \right) \cdot \log p\left( y=2\left| x=A \right. \right)\\ \\ &=-\frac{5}{7}\log \frac{5}{7}-\frac{2}{7}\log \frac{2}{7}=\log 7-\frac{5}{7}\log 5-\frac{2}{7}\log 2\\ \\ &=0.59827 \\ H\left( Y\left| X \right. =B \right) &=-p\left( y=1\left| x=B \right. \right) \cdot \log p\left( y=1\left| x=B \right. \right) -p\left( y=2\left| x=B \right. \right) \cdot \log p\left( y=2\left| x=B \right. \right)\\ \\ &=-\frac{5}{6}\log \frac{5}{6}-\frac{1}{6}\log \frac{1}{6}=\log 6-\frac{5}{6}\log 5\\ \\ &=0.450561\\ H\left( Y\left| X \right. =C \right) &=-p\left( y=1\left| x=C \right. \right) \cdot \log p\left( y=1\left| x=C \right. \right) -p\left( y=2\left| x=C \right. \right) \cdot \log p\left( y=2\left| x=C \right. \right)\\ \,\,\\ &=-\frac{2}{3}\log \frac{2}{3}-\frac{1}{3}\log \frac{1}{3}=\log 3-\frac{2}{3}\log 2\\ \\ &=0.636514\\ H\left( Y\left| X \right. =D \right) &=-p\left( y=1\left| x=D \right. \right) \cdot \log p\left( y=1\left| x=D \right. \right) -p\left( y=2\left| x=D \right. \right) \cdot \log p\left( y=2\left| x=D \right. \right)\\ \,\,\\ &=-\frac{2}{3}\log \frac{2}{3}-\frac{1}{3}\log \frac{1}{3}=\log 3-\frac{2}{3}\log 2\\ \\ &=0.636514\\ \end{aligned}于是

\begin{aligned} H\left( Y\left| X \right. \right) &=p\left( x=A \right) \cdot H\left( Y\left| X=A \right. \right) +p\left( x=B \right) \cdot H\left( Y\left| X=B \right. \right) +p\left( x=C \right) \cdot H\left( Y\left| X=C \right. \right) +p\left( x=D \right) \cdot H\left( Y\left| X=D \right. \right) \\ \,\,\mathrm{ } \\ &=\frac{7}{16}\times \left( \log 7-\frac{5}{7}\log 5-\frac{2}{7}\log 2 \right) +\frac{3}{8}\times \left( \log 6-\frac{5}{6}\log 5 \right) +\frac{3}{32}\times \left( \log 3-\frac{2}{3}\log 2 \right) +\frac{3}{32}\times \left( \log 3-\frac{2}{3}\log 2 \right) \\ \,\,\mathrm{ } \\ &=\frac{1}{16}\times \left( 7\log 7+6\log 6-10\log 5+3\log 3-4\log 2 \right) \\ &=0.55005 \end{aligned}

例2.4 互信息

p\left( x=0,y=0 \right) =\frac{1}{3}\text{，}p\left( x=0,y=1 \right) =\frac{1}{6}\text{，}p\left( x=1,y=0 \right) =\frac{1}{6}\text{，}p\left( x=1,y=1 \right) =\frac{1}{3}

则X和Y的互信息为：
\begin{aligned} I\left( X;Y \right) &=p\left( x=A,y=1 \right) \cdot \log \frac{p\left( x=A,y=1 \right)}{p\left( x=A \right) \cdot p\left( y=1 \right)}+p\left( x=A,y=2 \right) \cdot \log \frac{p\left( x=A,y=2 \right)}{p\left( x=A \right) \cdot p\left( y=2 \right)} \\ \\ &+p\left( x=B,y=1 \right) \cdot \log \frac{p\left( x=B,y=1 \right)}{p\left( x=B \right) \cdot p\left( y=1 \right)}+p\left( x=B,y=2 \right) \cdot \log \frac{p\left( x=B,y=2 \right)}{p\left( x=B \right) \cdot p\left( y=2 \right)} \\ \\ &+p\left( x=C,y=1 \right) \cdot \log \frac{p\left( x=C,y=1 \right)}{p\left( x=C \right) \cdot p\left( y=1 \right)}+p\left( x=C,y=2 \right) \cdot \log \frac{p\left( x=C,y=2 \right)}{p\left( x=C \right) \cdot p\left( y=2 \right)} \\ \\ &+p\left( x=D,y=1 \right) \cdot \log \frac{p\left( x=D,y=1 \right)}{p\left( x=D \right) \cdot p\left( y=1 \right)}+p\left( x=D,y=2 \right) \cdot \log \frac{p\left( x=D,y=2 \right)}{p\left( x=D \right) \cdot p\left( y=2 \right)} \\ \\ &=\frac{5}{16}\times \log \frac{\small{\frac{5}{16}}}{\frac{7}{16}\times \frac{3}{4}}+\frac{1}{8}\times \log \frac{\frac{1}{8}}{\frac{7}{16}\times \frac{1}{4}}+\frac{5}{16}\times \log \frac{\frac{5}{16}}{\frac{3}{8}\times \frac{3}{4}}+\frac{1}{16}\times \log \frac{\frac{1}{16}}{\frac{3}{8}\times \frac{1}{4}}+\frac{1}{16}\times \log \frac{\frac{1}{16}}{\frac{3}{32}\times \frac{3}{4}}+\frac{1}{32}\times \log \frac{\frac{1}{32}}{\frac{3}{32}\times \frac{1}{4}}+\frac{1}{16}\times \log \frac{\frac{1}{16}}{\frac{3}{32}\times \frac{3}{4}}+\frac{1}{32}\times \log \frac{\frac{1}{32}}{\frac{3}{32}\times \frac{1}{4}} \\ \\ &=\frac{1}{16}\times \left( 30\log 2-21\log 3+10\log 5-7\log 7 \right) \\ &=0.0122853 \end{aligned}

例2.5 条件互信息

p\left( z=0 \right) =\frac{1}{2}，p\left( z=1 \right) =\frac{1}{2} \\ \ \ \\ p\left( \left. x=A \right|z=0 \right) =\frac{3}{8}\text{，}p\left( \left. x=B \right|z=0 \right) =\frac{1}{4}\text{，}p\left( \left. x=C \right|z=0 \right) =\frac{3}{16}\text{，}p\left( \left. x=D \right|z=0 \right) =\frac{3}{16} \\ \ \ \\ p\left( \left. x=A \right|z=1 \right) =\frac{1}{2}\text{，}p\left( \left. x=B \right|z=1 \right) =\frac{1}{2}\text{，}p\left( \left. x=C \right|z=1 \right) =0\text{，}p\left( \left. x=D \right|z=1 \right) =0 \\ \ \ \\ p\left( \left. y=1 \right|z=0 \right) =\frac{1}{2}\text{，}p\left( \left. y=2 \right|z=0 \right) =\frac{1}{2}\text{，}p\left( \left. y=1 \right|z=1 \right) =1\text{，}p\left( \left. y=2 \right|z=1 \right) =0
记
p_{01}=p\left( \left. x=A,y=1 \right|z=0 \right) =\frac{1}{8}\text{，}p_{02}=p\left( \left. x=B,y=1 \right|z=0 \right) =\frac{1}{8}\text{，}p_{03}=p\left( \left. x=C,y=1 \right|z=0 \right) =\frac{1}{8}\text{，}p_{04}=p\left( \left. x=D,y=1 \right|z=0 \right) =\frac{1}{8} \\ \ \ \\ p_{05}=p\left( \left. x=A,y=2 \right|z=0 \right) =\frac{1}{4}\text{，}p_{06}=p\left( \left. x=B,y=2 \right|z=0 \right) =\frac{1}{8}\text{，}p_{07}=p\left( \left. x=C,y=2 \right|z=0 \right) =\frac{1}{16}\text{，}p_{08}=p\left( \left. x=D,y=2 \right|z=0 \right) =\frac{1}{16} \\ \ \ \\ p_{11}=p\left( \left. x=A,y=1 \right|z=1 \right) =\frac{1}{2}\text{，}p_{12}=p\left( \left. x=B,y=1 \right|z=1 \right) =\frac{1}{2}

\begin{aligned} P_{01}&=\frac{p\left( x=A,y=1\left| z=0 \right. \right)}{p\left( x=A\left| z=0 \right. \right) \cdot p\left( y=1\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{3}{8}\times \frac{1}{2}}=\frac{2}{3}\text{，}P_{02}=\frac{p\left( x=B,y=1\left| z=0 \right. \right)}{p\left( x=B\left| z=0 \right. \right) \cdot p\left( y=1\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{1}{4}\times \frac{1}{2}}=1 \\ \,\,\mathrm{ } \\ P_{03}&=\frac{p\left( x=C,y=1\left| z=0 \right. \right)}{p\left( x=C\left| z=0 \right. \right) \cdot p\left( y=1\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{3}{16}\times \frac{1}{2}}=\frac{4}{3}\text{，}P_{04}=\frac{p\left( x=D,y=1\left| z=0 \right. \right)}{p\left( x=D\left| z=0 \right. \right) \cdot p\left( y=1\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{3}{16}\times \frac{1}{2}}=\frac{4}{3} \\ \,\,\mathrm{ } \\ P_{05}&=\frac{p\left( x=A,y=2\left| z=0 \right. \right)}{p\left( x=A\left| z=0 \right. \right) \cdot p\left( y=2\left| z=0 \right. \right)}=\frac{\frac{1}{4}}{\frac{3}{8}\times \frac{1}{2}}=\frac{4}{3}\text{，}P_{06}=\frac{p\left( x=B,y=2\left| z=0 \right. \right)}{p\left( x=B\left| z=0 \right. \right) \cdot p\left( y=2\left| z=0 \right. \right)}=\frac{\frac{1}{8}}{\frac{1}{4}\times \frac{1}{2}}=1 \\ \,\,\mathrm{ } \\ P_{07}&=\frac{p\left( x=C,y=2\left| z=0 \right. \right)}{p\left( x=C\left| z=0 \right. \right) \cdot p\left( y=2\left| z=0 \right. \right)}=\frac{\frac{1}{16}}{\frac{3}{16}\times \frac{1}{2}}=\frac{2}{3}\text{，}P_{08}=\frac{p\left( x=D,y=2\left| z=0 \right. \right)}{p\left( x=D\left| z=0 \right. \right) \cdot p\left( y=2\left| z=0 \right. \right)}=\frac{\frac{1}{16}}{\frac{3}{16}\times \frac{1}{2}}=\frac{2}{3} \\ \ \ \\ P_{11}&=\frac{p\left( x=A,y=1\left| z=1 \right. \right)}{p\left( x=A\left| z=1 \right. \right) \cdot p\left( y=1\left| z=1 \right. \right)}=\frac{\frac{1}{2}}{\frac{1}{2}\times 1}=1\text{，}P_{12}=\frac{p\left( x=B,y=1\left| z=1 \right. \right)}{p\left( x=B\left| z=1 \right. \right) \cdot p\left( y=1\left| z=1 \right. \right)}=\frac{\frac{1}{2}}{\frac{1}{2}\times 1}=1 \end{aligned}
故条件互信息为：
\begin{aligned} I\left( \left. X;Y \right|Z \right) &=\sum_{z\in Z}{p\left( z \right)}\sum_{x\in X}{\sum_{y\in Y}{p\left( \left. x,y \right|z \right)}}·\log \frac{p\left( \left. x,y \right|z \right)}{p\left( \left. x \right|z \right) p\left( \left. y \right|z \right)} \\ \,\,\mathrm{ } \\ &=p\left( z=0 \right) \times \left[ p_{01}\cdot \log P_{01}+p_{02}\cdot \log P_{02}+p_{03}\cdot \log P_{03}+p_{04}\cdot \log P_{04}+p_{05}\cdot \log P_{05}+p_{06}\cdot \log P_{06}+p_{07}\cdot \log P_{07}+p_{08}\cdot \log P_{08} \right] \\ \,\,\mathrm{ } \\ &+p\left( z=1 \right) \times \left[ p_{11}\cdot \log P_{11}+p_{12}\cdot \log P_{12} \right] \\ \,\,\mathrm{ } \\ &=\frac{1}{2}\times \left[ \frac{1}{8}\log \frac{2}{3}+0+\frac{1}{8}\log \frac{4}{3}+\frac{1}{8}\log \frac{4}{3}+\frac{1}{4}\log \frac{4}{3}+0+\frac{1}{16}\log \frac{2}{3}+\frac{1}{16}\log \frac{2}{3} \right] +\frac{1}{2}\times \left[ 0+0 \right] \\ \,\,\mathrm{ } \\ &=\frac{1}{8}\times \left( 5\log 2-3\log 3 \right) =0.0212374 \end{aligned}