https://haleuan.github.io/en/tags/kl-divergence/ Recent content in KL Divergence on Haleuan's Blog Hugo en Sat, 03 May 2025 15:59:21 +0800 https://haleuan.github.io/en/post/kl%E6%95%A3%E5%BA%A6%E5%92%8C%E4%BA%A4%E5%8F%89%E7%86%B5%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0/ Sat, 03 May 2025 15:59:21 +0800 https://haleuan.github.io/en/post/kl%E6%95%A3%E5%BA%A6%E5%92%8C%E4%BA%A4%E5%8F%89%E7%86%B5%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0/ <h2 id="kl-divergence-kullback-leibler-divergence-kld">KL Divergence (Kullback-Leibler Divergence, KLD)</h2> <p>Suppose we have two coins:</p> <ul> <li>A fair coin with the following probability distribution:<br> $$P=\begin{cases}&0.5,&\text{Heads} \\ &0.5, &\text{Tails}\end{cases}$$</li> <li>A biased coin with the following probability distribution:<br> $$Q=\begin{cases}&0.9,&\text{Heads} \\ &0.1, &\text{Tails}\end{cases}$$</li> </ul> <p>How can we describe the difference between these two distributions? More importantly, how can we <strong>quantify</strong> this difference?</p> <p>KL divergence allows us to quantify the difference (or similarity) between two distributions. It measures how one probability distribution $Q(x)$ diverges from a reference distribution $P(x)$, and is defined as:<br> </p>