矩阵混乱度（熵值）代码计算

1、先回顾下熵值的数据公式：

2、jax.numpy代码

注意的点：熵值计算的输入的必须是归一化的正值
import jax.numpy as jnp
import jax

def _entroy(probs):
  log_probs = jnp.log2(jnp.maximum(1.0e-30, probs))
  mean_sum_plogp = jnp.mean(- jnp.sum(log_probs * probs, axis=-1))
  return mean_sum_plogp

随机

key = jax.random.PRNGKey(123)
inputs = jax.random.normal(key, shape=(3, 4))
print(f'inputs:\n{inputs}')
probs1 = jax.nn.softmax(inputs)
print(f'probs1:\n{probs1}')
entroy_value1 = _entroy(probs1)
print(f'entroy_value1: {entroy_value1}\n\n')

输出：
inputs:
[[-0.31682462 -1.5700184   0.6431673  -0.11953171]
 [ 0.21440512 -0.886306   -0.0515956  -0.81674606]
 [-1.241783   -0.63905096 -0.65371424  0.88143796]]
probs1:
[[0.19548938 0.05583005 0.5105548  0.23812577]
 [0.40722093 0.13545571 0.31210986 0.14521345]
 [0.07700823 0.140702   0.1386539  0.64363587]]
entroy_value1: 1.6717370748519897

极端均匀

极端均匀，熵值最大。最大值为log₂(dim)，例子的shape为3 * 4，我们计算的为最后一维的熵值情况，因此dim为4，所以log₂(4) = 2。

probs2 = jnp.array([[0.25, 0.25, 0.25, 0.25], [0.25, 0.25, 0.25, 0.25], [0.25, 0.25, 0.25, 0.25]])
print(f'probs2:\n{probs2}')
entroy_value2 = _entroy(probs2)
print(f'entroy_value2: {entroy_value2}\n\n')
输出：
probs2:
[[0.25 0.25 0.25 0.25]
 [0.25 0.25 0.25 0.25]
 [0.25 0.25 0.25 0.25]]
entroy_value2: 2.0

增加混乱程度

增加混乱度，熵值减小

# 修改了矩阵的概率值
probs3 = jnp.array([[0.5, 0, 0.25, 0.25], [0.25, 0.25, 0.25, 0.25], [0.25, 0.25, 0.25, 0.25]])
print(f'probs3:\n{probs3}')
entroy_value3 = _entroy(probs3)
print(f'entroy_value3: {entroy_value3}\n\n')
输出：
probs3:
[[0.5  0.   0.25 0.25]
 [0.25 0.25 0.25 0.25]
 [0.25 0.25 0.25 0.25]]
entroy_value3: 1.8333333730697632

极端混乱

极端混乱度，熵值最小，最小值跟矩阵的维度无关，基本都为0

probs4 = jnp.array([[0, 0, 1, 0],[0, 0, 1, 0], [0, 0, 1, 0]])
print(f'probs4:\n{probs4}')
entroy_value4 = _entroy(probs4)
print(f'entroy_value4: {entroy_value4}\n\n')
输出：
probs4:
[[0 0 1 0]
 [0 0 1 0]
 [0 0 1 0]]
entroy_value4: 0.0

3、numpy代码：

import numpy as np
剩下代码把随机输jnp换成np即可。然后就是生成随机输入和Softmax也有点不一样。

4、torch代码

import torch


def _entroy(probs):
  log_probs = torch.log2(torch.maximum(torch.tensor(1.0e-30), probs))
  mean_sum_plogp = torch.mean(- torch.sum(log_probs * probs, dim=-1))
  return mean_sum_plogp
    
torch.manual_seed(123)

随机

inputs = torch.rand(3, 4)
print(f'inputs:\n{inputs}')
probs1 = torch.nn.functional.softmax(inputs)
print(f'probs1:\n{probs1}')
entroy_value1 = _entroy(probs1)
print(f'entroy_value1: {entroy_value1}\n\n')
输出：
inputs:
tensor([[0.2961, 0.5166, 0.2517, 0.6886],
        [0.0740, 0.8665, 0.1366, 0.1025],
        [0.1841, 0.7264, 0.3153, 0.6871]])
probs1:
tensor([[0.2135, 0.2662, 0.2042, 0.3161],
        [0.1886, 0.4166, 0.2008, 0.1940],
        [0.1814, 0.3120, 0.2068, 0.2999]])
entroy_value1: 1.947859764099121

极端均匀

probs2 = torch.tensor([[0.25, 0.25, 0.25, 0.25], [0.25, 0.25, 0.25, 0.25], [0.25, 0.25, 0.25, 0.25]])
print(f'probs2:\n{probs2}')
entroy_value2 = _entroy(probs2)
print(f'entroy_value2: {entroy_value2}\n\n')
输出：
probs2:
tensor([[0.2500, 0.2500, 0.2500, 0.2500],
        [0.2500, 0.2500, 0.2500, 0.2500],
        [0.2500, 0.2500, 0.2500, 0.2500]])
entroy_value2: 2.0

改变矩阵的混乱程度

probs3 = torch.tensor([[0.5, 0, 0.25, 0.25], [0.25, 0.25, 0.25, 0.25], [0.25, 0.25, 0.25, 0.25]])
print(f'probs3:\n{probs3}')
entroy_value3 = _entroy(probs3)
print(f'entroy_value3: {entroy_value3}\n\n')
输出：
probs3:
tensor([[0.5000, 0.0000, 0.2500, 0.2500],
        [0.2500, 0.2500, 0.2500, 0.2500],
        [0.2500, 0.2500, 0.2500, 0.2500]])
entroy_value3: 1.8333333730697632

极端混乱

probs4 = torch.tensor([[0, 0, 1, 0],[0, 0, 1, 0], [0, 0, 1, 0]])
print(f'probs4:\n{probs4}')
entroy_value4 = _entroy(probs4)
print(f'entroy_value4: {entroy_value4}\n\n')
输出：
tensor([[0, 0, 1, 0],
        [0, 0, 1, 0],
        [0, 0, 1, 0]])
entroy_value4: 0.0