[{"content":"KL Divergence (Kullback-Leibler Divergence, KLD) Suppose we have two coins:\nA fair coin with the following probability distribution:\n$$P=\\begin{cases}\u00260.5,\u0026\\text{Heads} \\\\ \u00260.5, \u0026\\text{Tails}\\end{cases}$$ A biased coin with the following probability distribution:\n$$Q=\\begin{cases}\u00260.9,\u0026\\text{Heads} \\\\ \u00260.1, \u0026\\text{Tails}\\end{cases}$$ How can we describe the difference between these two distributions? More importantly, how can we quantify this difference?\nKL divergence allows us to quantify the difference (or similarity) between two distributions. It measures how one probability distribution $Q(x)$ diverges from a reference distribution $P(x)$, and is defined as:\n$$D_{KL}(Q|| P)=\\begin{cases} \\sum\\limits_i Q(i)\\log\\frac{Q(i)}{P(i)},\\quad \\text{discrete} \\\\ \\int Q(i)\\log\\frac{Q(i)}{P(i)}\\ di,\\quad \\text{continuous}\\end{cases}$$Note: KL divergence is asymmetric, meaning that $D_{KL}(Q||P) \\ne D_{KL}(P||Q)$.\n$D_{KL}(Q||P)$ represents the information loss when using distribution $P$ to approximate $Q$. $D_{KL}(P||Q)$ represents the information loss when using distribution $Q$ to approximate $P$. Entropy of a Distribution Entropy is a concept often introduced in high school physics as a measure of disorder in a system. In statistics, entropy measures the uncertainty of a probability distribution.\nFor a distribution $P$, its entropy is defined as:\n$$H(P) = -\\sum\\limits_i P(x_i)\\log\\big(P(x_i)\\big)$$From KL Divergence to Cross Entropy In real-world machine learning tasks, we often use a predicted distribution to approximate the true data distribution. Let $P$ be the true distribution and $Q$ be the predicted distribution. The cross entropy is defined as:\n$$H(P, Q) = -\\sum\\limits_i P(x_i)\\log\\big(Q(x_i)\\big)$$We can derive this from KL divergence as follows:\n$$\\begin{aligned} D_{KL}(P||Q) \u0026= \\sum\\limits_i P(x_i)\\log\\left(\\frac{P(x_i)}{Q(x_i)}\\right) \\\\ \u0026= \\sum\\limits_i P(x_i)\\left(\\log(P_i) - \\log(Q_i)\\right) \\\\ \u0026= \\sum_i P(x_i)\\log(P_{x_i}) - \\sum_i P(x_i)\\log(Q_{x_i}) \\end{aligned}$$We can observe that $\\sum_i P(x_i)\\log(P_{x_i}) = -H(P)$, which is the entropy of $P$, and $\\sum_i P(x_i)\\log(Q_{x_i})$ is the cross entropy $H(P, Q)$. Therefore, we can rewrite the equation as:\n$$D_{KL}(P||Q) = -H(P) + H(P, Q) \\quad \\Longrightarrow \\quad H(P, Q) = D_{KL}(P||Q) + H(P)$$This shows that\nCross Entropy = Entropy of the true distribution + KL Divergence (from true to predicted distribution).\nSo, when we minimize the cross entropy during training, we are effectively minimizing the KL divergence $D_{KL}(P||Q)$, since the true distribution’s entropy $H(P)$ is constant and independent of the model.\nUsing Cross Entropy as the Loss Function in Classification Tasks In classification problems with one-hot encoded labels, there is exactly one correct class for each prediction. The cross entropy is:\n$$H(P, Q) = -\\sum\\limits_i P(x_i)\\log\\big(Q(x_i)\\big)$$\nwhere $P(i)$ is the true probability for class $i$ and $Q(i)$ is the predicted probability for class $i$.\nExpanding the sum:\n$$\\begin{aligned} H(P, Q) \u0026= -\\sum\\limits_i P(x_i)\\log(Q(x_i)) \\\\ \u0026= -\\big(0\\cdot\\log(Q_1) + 0\\cdot\\log(Q_2) + \\ldots + 1\\cdot\\log(Q_C) + \\ldots + 0\\cdot\\log(Q_N)\\big) \\\\ H(P, Q) \u0026= -\\log\\big(Q(i = C)\\big) \\end{aligned}$$Here, $C$ is the correct class.\nHowever, note that in practice, we don’t compute the loss for a single sample, but for a batch of samples. Therefore, the loss function is typically calculated as the average cross entropy over the batch:\n$$\\text{batch\\_loss} = -\\frac{1}{N}\\sum\\limits_{i=1}^N\\log\\big(Q(k_i)\\big)$$","permalink":"https://haleuan.github.io/en/post/kl%E6%95%A3%E5%BA%A6%E5%92%8C%E4%BA%A4%E5%8F%89%E7%86%B5%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0/","summary":"\u003ch2 id=\"kl-divergence-kullback-leibler-divergence-kld\"\u003eKL Divergence (Kullback-Leibler Divergence, KLD)\u003c/h2\u003e\n\u003cp\u003eSuppose we have two coins:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eA fair coin with the following probability distribution:\u003cbr\u003e\n\n$$P=\\begin{cases}\u00260.5,\u0026\\text{Heads} \\\\ \u00260.5, \u0026\\text{Tails}\\end{cases}$$\u003c/li\u003e\n\u003cli\u003eA biased coin with the following probability distribution:\u003cbr\u003e\n\n$$Q=\\begin{cases}\u00260.9,\u0026\\text{Heads} \\\\ \u00260.1, \u0026\\text{Tails}\\end{cases}$$\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eHow can we describe the difference between these two distributions? More importantly, how can we \u003cstrong\u003equantify\u003c/strong\u003e this difference?\u003c/p\u003e\n\u003cp\u003eKL divergence allows us to quantify the difference (or similarity) between two distributions. It measures how one probability distribution $Q(x)$ diverges from a reference distribution $P(x)$, and is defined as:\u003cbr\u003e\n\u003c/p\u003e","title":"KL Divergence and Cross Entropy"}]